Overhaul Cilium manifests to match the newer versions (#8717)

* [cilium] Separate templates for cilium, cilium-operator, and hubble installations

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Update cilium-operator templates

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Allow using custom args and mounting extra volumes for the Cilium Operator

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Update the cilium configmap to filter out the deprecated variables, and add the new variables

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Add an option to use Wireguard encryption on Cilium 1.10 and up

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Update cilium-agent templates

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Bump Cilium version to 1.11.3

Signed-off-by: necatican <necaticanyildirim@gmail.com>

docs/cilium.md

@@ -12,10 +12,51 @@ the external loadbalancer (even from a node in the control plane)
 and if there is no external load balancer It will ignore any local load
 balancer deployed by Kubespray and **only contacts the first master**.
 
+## Cilium Operator
+
+Unlike some operators, Cilium Operator does not exist for installation purposes.
+> The Cilium Operator is responsible for managing duties in the cluster which should logically be handled once for the entire cluster, rather than once for each node in the cluster.
+
+### Adding custom flags to the Cilium Operator
+
+You can set additional cilium-operator container arguments using `cilium_operator_custom_args`.
+This is an advanced option, and you should only use it if you know what you are doing.
+
+Accepts an array or a string.
+
+```yml
+cilium_operator_custom_args: ["--foo=bar", "--baz=qux"]
+```
+
+or
+
+```yml
+cilium_operator_custom_args: "--foo=bar"
+```
+
+You do not need to add a custom flag to enable debugging. Instead, feel free to use the `CILIUM_DEBUG` variable.
+
+### Adding extra volumes and mounting them
+
+You can use `cilium_operator_extra_volumes` to add extra volumes to the Cilium Operator, and use `cilium_operator_extra_volume_mounts` to mount those volumes.
+This is an advanced option, and you should only use it if you know what you are doing.
+
+```yml
+cilium_operator_extra_volumes:
+  - configMap:
+      name: foo
+    name: foo-mount-path
+
+cilium_operator_extra_volume_mounts:
+  - mountPath: /tmp/foo/bar
+    name: foo-mount-path
+    readOnly: true
+```
+
 ## Choose Cilium version
 
 ```yml
-cilium_version: v1.11.0
+cilium_version: v1.11.3
 ```
 
 ## Add variable to config
@@ -39,6 +80,47 @@ Cilium currently supports two different identity allocation modes:
   - These can be queried with `kubectl get ciliumid`
 - "kvstore" stores identities in an etcd kvstore.
 
+## Enable Transparent Encryption
+
+Cilium supports the transparent encryption of Cilium-managed host traffic and
+traffic between Cilium-managed endpoints either using IPsec or Wireguard.
+
+Wireguard option is only available in Cilium 1.10.0 and newer.
+
+### IPsec Encryption
+
+For further information, make sure to check the official [Cilium documentation.](https://docs.cilium.io/en/stable/gettingstarted/encryption-ipsec/)
+
+To enable IPsec encryption, you just need to set three variables.
+
+```yml
+cilium_encryption_enabled: true
+cilium_encryption_type: "ipsec"
+```
+
+The third variable is `cilium_ipsec_key.` You need to create a secret key string for this variable.
+Kubespray does not automate this process.
+Cilium documentation currently recommends creating a key using the following command:
+
+```shell
+echo "3 rfc4106(gcm(aes)) $(echo $(dd if=/dev/urandom count=20 bs=1 2> /dev/null | xxd -p -c 64)) 128"
+```
+
+Note that Kubespray handles secret creation. So you only need to pass the key as the `cilium_ipsec_key` variable.
+
+### Wireguard Encryption
+
+For further information, make sure to check the official [Cilium documentation.](https://docs.cilium.io/en/stable/gettingstarted/encryption-wireguard/)
+
+To enable Wireguard encryption, you just need to set two variables.
+
+```yml
+cilium_encryption_enabled: true
+cilium_encryption_type: "wireguard"
+```
+
+Kubespray currently supports Linux distributions with Wireguard Kernel mode on Linux 5.6 and newer.
+
 ## Install Cilium Hubble
 
 k8s-net-cilium.yml:
@@ -68,6 +150,6 @@ cilium_hubble_metrics:
   - flow
   - icmp
   - http
-```  
+```
 
 [More](https://docs.cilium.io/en/v1.9/operations/metrics/#hubble-exported-metrics)

inventory/sample/group_vars/k8s_cluster/k8s-net-cilium.yml

@@ -1,4 +1,47 @@
 # see roles/network_plugin/cilium/defaults/main.yml
 
-# cilium_version: "v1.11.0"
+# cilium_version: "v1.11.3"
 # cilium_identity_allocation_mode: kvstore # kvstore or crd
+
+# For adding and mounting extra volumes to the cilium operator
+# cilium_operator_extra_volumes: []
+# cilium_operator_extra_volume_mounts: []
+
+# Name of the cluster. Only relevant when building a mesh of clusters.
+# cilium_cluster_name: default
+
+# Unique ID of the cluster. Must be unique across all conneted clusters and
+# in the range of 1 and 255. Only relevant when building a mesh of clusters.
+# This value is not defined by default
+# cluster-id:
+
+# Allows to explicitly specify the IPv4 CIDR for native routing.
+# When specified, Cilium assumes networking for this CIDR is preconfigured and
+# hands traffic destined for that range to the Linux network stack without
+# applying any SNAT.
+# Generally speaking, specifying a native routing CIDR implies that Cilium can
+# depend on the underlying networking stack to route packets to their
+# destination. To offer a concrete example, if Cilium is configured to use
+# direct routing and the Kubernetes CIDR is included in the native routing CIDR,
+# the user must configure the routes to reach pods, either manually or by
+# setting the auto-direct-node-routes flag.
+# cilium_native_routing_cidr: ""
+
+# Allows to explicitly specify the IPv6 CIDR for native routing.
+# cilium_native_routing_cidr_ipv6: ""
+
+# Encryption
+# Enable transparent network encryption.
+# cilium_encryption_enabled: false
+
+# Encryption method. Can be either ipsec or wireguard.
+# Only effective when `cilium_encryption_enabled` is set to true.
+# cilium_encryption_type: "ipsec"
+
+# Enable encryption for pure node to node traffic.
+# This option is only effective when `cilium_encryption_type` is set to `ipsec`.
+# cilium_ipsec_node_encryption: "false"
+
+# Enables the fallback to the user-space implementation.
+# This option is only effective when `cilium_encryption_type` is set to `wireguard`.
+# cilium_wireguard_userspace_fallback: "false"

roles/download/defaults/main.yml

@@ -110,7 +110,7 @@ flannel_cni_version: "v1.0.1"
 cni_version: "v1.0.1"
 weave_version: 2.8.1
 pod_infra_version: "3.3"
-cilium_version: "v1.11.1"
+cilium_version: "v1.11.3"
 kube_ovn_version: "v1.8.1"
 kube_router_version: "v1.4.0"
 multus_version: "v3.8"

roles/network_plugin/cilium/defaults/main.yml

@@ -48,7 +48,11 @@ cilium_kube_proxy_replacement: probe
 # to prevent service disruptions. See also:
 # http://docs.cilium.io/en/stable/install/upgrade/#changes-that-may-require-action
 cilium_preallocate_bpf_maps: false
+
+# `cilium_tofqdns_enable_poller` is deprecated in 1.8, removed in 1.9
 cilium_tofqdns_enable_poller: false
+
+# `cilium_enable_legacy_services` is deprecated in 1.6, removed in 1.9
 cilium_enable_legacy_services: false
 
 # Deploy cilium even if kube_network_plugin is not cilium.
@@ -62,10 +66,38 @@ cilium_deploy_additionally: false
 # make this work. Please refer to the cilium documentation for more
 # information about this kind of setups.
 cilium_auto_direct_node_routes: false
+
+# Allows to explicitly specify the IPv4 CIDR for native routing.
+# When specified, Cilium assumes networking for this CIDR is preconfigured and
+# hands traffic destined for that range to the Linux network stack without
+# applying any SNAT.
+# Generally speaking, specifying a native routing CIDR implies that Cilium can
+# depend on the underlying networking stack to route packets to their
+# destination. To offer a concrete example, if Cilium is configured to use
+# direct routing and the Kubernetes CIDR is included in the native routing CIDR,
+# the user must configure the routes to reach pods, either manually or by
+# setting the auto-direct-node-routes flag.
 cilium_native_routing_cidr: ""
 
-# IPsec based transparent encryption between nodes
+# Allows to explicitly specify the IPv6 CIDR for native routing.
-cilium_ipsec_enabled: false
+cilium_native_routing_cidr_ipv6: ""
+
+# Enable transparent network encryption.
+cilium_encryption_enabled: false
+
+# Encryption method. Can be either ipsec or wireguard.
+# Only effective when `cilium_encryption_enabled` is set to true.
+cilium_encryption_type: "ipsec"
+
+# Enable encryption for pure node to node traffic.
+# This option is only effective when `cilium_encryption_type` is set to `ipsec`.
+cilium_ipsec_node_encryption: "false"
+
+# If your kernel or distribution does not support WireGuard, Cilium agent can be configured to fall back on the user-space implementation.
+# When this flag is enabled and Cilium detects that the kernel has no native support for WireGuard,
+# it will fallback on the wireguard-go user-space implementation of WireGuard.
+# This option is only effective when `cilium_encryption_type` is set to `wireguard`.
+cilium_wireguard_userspace_fallback: "false"
 
 # Hubble
 ### Enable Hubble without install
@@ -89,6 +121,15 @@ cilium_hubble_tls_generate: false
 # https://docs.cilium.io/en/v1.9/concepts/networking/ipam/
 cilium_ipam_mode: kubernetes
 
+# Extra arguments for the Cilium agent
+cilium_agent_custom_args: []
+
+# For adding and mounting extra volumes to the cilium agent
+cilium_agent_extra_volumes: []
+cilium_agent_extra_volume_mounts: []
+
+cilium_agent_extra_env_vars: []
+
 # The address at which the cillium operator bind health check api
 cilium_operator_api_serve_addr: "127.0.0.1:9234"
 
@@ -97,3 +138,23 @@ cilium_operator_api_serve_addr: "127.0.0.1:9234"
 ##    var1: "value1"
 ##    var2: "value2"
 cilium_config_extra_vars: {}
+
+# For adding and mounting extra volumes to the cilium operator
+cilium_operator_extra_volumes: []
+cilium_operator_extra_volume_mounts: []
+
+# Extra arguments for the Cilium Operator
+cilium_operator_custom_args: []
+
+# Name of the cluster. Only relevant when building a mesh of clusters.
+cilium_cluster_name: default
+
+# Make Cilium take ownership over the `/etc/cni/net.d` directory on the node, renaming all non-Cilium CNI configurations to `*.cilium_bak`.
+# This ensures no Pods can be scheduled using other CNI plugins during Cilium agent downtime.
+# Available for Cilium v1.10 and up.
+cilium_cni_exclusive: "true"
+
+# Configure the log file for CNI logging with retention policy of 7 days.
+# Disable CNI file logging by setting this field to empty explicitly.
+# Available for Cilium v1.12 and up.
+cilium_cni_log_file: "/var/run/cilium/cilium-cni.log"

roles/network_plugin/cilium/tasks/apply.yml

@@ -5,7 +5,7 @@
     namespace: "kube-system"
     kubectl: "{{ bin_dir }}/kubectl"
     resource: "{{ item.item.type }}"
-    filename: "{{ kube_config_dir }}/{{ item.item.file }}"
+    filename: "{{ kube_config_dir }}/{{ item.item.name }}-{{ item.item.file }}"
     state: "latest"
   loop: "{{ cilium_node_manifests.results }}"
   when: inventory_hostname == groups['kube_control_plane'][0] and not item is skipped
@@ -25,7 +25,7 @@
     namespace: "kube-system"
     kubectl: "{{ bin_dir }}/kubectl"
     resource: "{{ item.item.type }}"
-    filename: "{{ kube_config_dir }}/addons/hubble/{{ item.item.file }}"
+    filename: "{{ kube_config_dir }}/addons/hubble/{{ item.item.name }}-{{ item.item.file }}"
     state: "latest"
   loop: "{{ cilium_hubble_manifests.results }}"
   when:

roles/network_plugin/cilium/tasks/check.yml

@@ -1,14 +1,66 @@
 ---
-- name: Cilium | Check cilium_ipsec_enabled variables
+- name: Cilium | Check Cilium encryption `cilium_ipsec_key` for ipsec
   assert:
     that:
       - "cilium_ipsec_key is defined"
-    msg: "cilium_ipsec_key should be defined to use cilium_ipsec_enabled"
+    msg: "cilium_ipsec_key should be defined to enable encryption using ipsec"
   when:
-    - cilium_ipsec_enabled
+    - cilium_encryption_enabled
+    - cilium_encryption_type == "ipsec"
     - cilium_tunnel_mode in ['vxlan']
 
+# TODO: Clean this task up when we drop backward compatibility support for `cilium_ipsec_enabled`
+- name: Stop if `cilium_ipsec_enabled` is defined and `cilium_encryption_type` is not `ipsec`
+  assert:
+    that: cilium_encryption_type == 'ipsec'
+    msg: >
+      It is not possible to use `cilium_ipsec_enabled` when `cilium_encryption_type` is set to {{ cilium_encryption_type }}.
+  when:
+    - cilium_ipsec_enabled is defined
+    - cilium_ipsec_enabled
+    - kube_network_plugin == 'cilium' or cilium_deploy_additionally | default(false) | bool
+
+- name: Stop if kernel version is too low for Cilium Wireguard encryption
+  assert:
+    that: ansible_kernel.split('-')[0] is version('5.6.0', '>=')
+  when:
+    - kube_network_plugin == 'cilium' or cilium_deploy_additionally | default(false) | bool
+    - cilium_encryption_enabled
+    - cilium_encryption_type == "wireguard"
+    - not ignore_assert_errors
+
 - name: Stop if bad Cilium identity allocation mode
   assert:
     that: cilium_identity_allocation_mode in ['crd', 'kvstore']
     msg: "cilium_identity_allocation_mode must be either 'crd' or 'kvstore'"
+
+- name: Stop if bad Cilium Cluster ID
+  assert:
+    that:
+      - cilium_cluster_id <= 255
+      - cilium_cluster_id >= 0
+    msg: "'cilium_cluster_id' must be between 1 and 255"
+  when: cilium_cluster_id is defined
+
+- name: Stop if bad encryption type
+  assert:
+    that: cilium_encryption_type in ['ipsec', 'wireguard']
+    msg: "cilium_encryption_type must be either 'ipsec' or 'wireguard'"
+  when: cilium_encryption_enabled
+
+- name: Stop if `cilium_encryption_type` is set to "wireguard" and cilium_version is < v1.10.0
+  assert:
+    that: cilium_version | regex_replace('v') is version('1.10', '>')
+    msg: "cilium_encryption_type is set to 'wireguard' but cilium_version is < v1.10.0"
+  when:
+    - cilium_encryption_enabled
+    - cilium_encryption_type == "wireguard"
+
+# TODO: Clean this task up when we drop backward compatibility support for `cilium_ipsec_enabled`
+- name: Set `cilium_encryption_type` to "ipsec" and  if `cilium_ipsec_enabled` is true
+  set_fact:
+    cilium_encryption_type: ipsec
+    cilium_encryption_enabled: true
+  when:
+    - cilium_ipsec_enabled is defined
+    - cilium_ipsec_enabled

roles/network_plugin/cilium/tasks/install.yml

@@ -43,17 +43,20 @@
 
 - name: Cilium | Create Cilium node manifests
   template:
-    src: "{{ item.file }}.j2"
+    src: "{{ item.name }}/{{ item.file }}.j2"
-    dest: "{{ kube_config_dir }}/{{ item.file }}"
+    dest: "{{ kube_config_dir }}/{{ item.name }}-{{ item.file }}"
     mode: 0644
   loop:
-    - {name: cilium, file: cilium-config.yml, type: cm}
+    - {name: cilium, file: config.yml, type: cm}
-    - {name: cilium, file: cilium-crb.yml, type: clusterrolebinding}
+    - {name: cilium-operator, file: crb.yml, type: clusterrolebinding}
-    - {name: cilium, file: cilium-cr.yml, type: clusterrole}
+    - {name: cilium-operator, file: cr.yml, type: clusterrole}
-    - {name: cilium, file: cilium-secret.yml, type: secret, when: "{{ cilium_ipsec_enabled }}"}
+    - {name: cilium, file: crb.yml, type: clusterrolebinding}
-    - {name: cilium, file: cilium-ds.yml, type: ds}
+    - {name: cilium, file: cr.yml, type: clusterrole}
-    - {name: cilium, file: cilium-deploy.yml, type: deploy}
+    - {name: cilium, file: secret.yml, type: secret, when: "{{ cilium_encryption_enabled and cilium_encryption_type == 'ipsec' }}"}
-    - {name: cilium, file: cilium-sa.yml, type: sa}
+    - {name: cilium, file: ds.yml, type: ds}
+    - {name: cilium-operator, file: deploy.yml, type: deploy}
+    - {name: cilium-operator, file: sa.yml, type: sa}
+    - {name: cilium, file: sa.yml, type: sa}
   register: cilium_node_manifests
   when:
     - inventory_hostname in groups['kube_control_plane']
@@ -61,18 +64,18 @@
 
 - name: Cilium | Create Cilium Hubble manifests
   template:
-    src: "{{ item.file }}.j2"
+    src: "{{ item.name }}/{{ item.file }}.j2"
-    dest: "{{ kube_config_dir }}/addons/hubble/{{ item.file }}"
+    dest: "{{ kube_config_dir }}/addons/hubble/{{ item.name }}-{{ item.file }}"
     mode: 0644
   loop:
-    - {name: hubble, file: hubble-config.yml, type: cm}
+    - {name: hubble, file: config.yml, type: cm}
-    - {name: hubble, file: hubble-crb.yml, type: clusterrolebinding}
+    - {name: hubble, file: crb.yml, type: clusterrolebinding}
-    - {name: hubble, file: hubble-cr.yml, type: clusterrole}
+    - {name: hubble, file: cr.yml, type: clusterrole}
-    - {name: hubble, file: hubble-cronjob.yml, type: cronjob, when: "{{ cilium_hubble_tls_generate }}"}
+    - {name: hubble, file: cronjob.yml, type: cronjob, when: "{{ cilium_hubble_tls_generate }}"}
-    - {name: hubble, file: hubble-deploy.yml, type: deploy}
+    - {name: hubble, file: deploy.yml, type: deploy}
-    - {name: hubble, file: hubble-job.yml, type: job, when: "{{ cilium_hubble_tls_generate }}"}
+    - {name: hubble, file: job.yml, type: job, when: "{{ cilium_hubble_tls_generate }}"}
-    - {name: hubble, file: hubble-sa.yml, type: sa}
+    - {name: hubble, file: sa.yml, type: sa}
-    - {name: hubble, file: hubble-service.yml, type: service}
+    - {name: hubble, file: service.yml, type: service}
   register: cilium_hubble_manifests
   when:
     - inventory_hostname == groups['kube_control_plane'][0]

roles/network_plugin/cilium/templates/cilium-operator/cr.yml.j2

@@ -23,6 +23,15 @@ rules:
   - get
   - list
   - watch
+{% if cilium_version | regex_replace('v') is version('1.10', '>=') %}
+- apiGroups:
+  - ""
+  resources:
+  # to perform LB IP allocation for BGP
+  - services/status
+  verbs:
+  - update
+{% endif %}
 - apiGroups:
   - ""
   resources:
@@ -68,9 +77,22 @@ rules:
   - ciliumlocalredirectpolicies
   - ciliumlocalredirectpolicies/status
   - ciliumlocalredirectpolicies/finalizers
+{% endif %}
+{% if cilium_version | regex_replace('v') is version('1.11', '>=') %}
+  - ciliumendpointslices
+{% endif %}
+{% if cilium_version | regex_replace('v') is version('1.12', '>=') %}
+  - ciliumbgploadbalancerippools
+  - ciliumbgppeeringpolicies
+  - ciliumenvoyconfigs
 {% endif %}
   verbs:
-  - '*'
+  - get
+  - list
+  - watch
+  - create
+  - update
+  - delete
 - apiGroups:
   - apiextensions.k8s.io
   resources:
@@ -100,109 +122,3 @@ rules:
   - get
   - update
 {% endif %}
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: cilium
-rules:
-- apiGroups:
-  - networking.k8s.io
-  resources:
-{% if cilium_version | regex_replace('v') is version('1.7', '<') %}
-  - ingresses
-{% endif %}
-  - networkpolicies
-  verbs:
-  - get
-  - list
-  - watch
-- apiGroups:
-  - discovery.k8s.io
-  resources:
-  - endpointslices
-  verbs:
-  - get
-  - list
-  - watch
-- apiGroups:
-  - ""
-  resources:
-  - namespaces
-  - services
-  - nodes
-  - endpoints
-{% if cilium_version | regex_replace('v') is version('1.7', '<') %}
-  - componentstatuses
-{% endif %}
-  verbs:
-  - get
-  - list
-  - watch
-{% if cilium_version | regex_replace('v') is version('1.7', '<') %}
-- apiGroups:
-  - extensions
-  resources:
-  - ingresses
-  verbs:
-  - create
-  - get
-  - list
-  - watch
-{% endif %}
-- apiGroups:
-  - ""
-  resources:
-  - pods
-  - nodes
-  verbs:
-  - get
-  - list
-  - watch
-  - update
-- apiGroups:
-  - ""
-  resources:
-  - nodes
-  - nodes/status
-  verbs:
-  - patch
-- apiGroups:
-  - apiextensions.k8s.io
-  resources:
-  - customresourcedefinitions
-  verbs:
-  - create
-  - get
-  - list
-  - watch
-  - update
-- apiGroups:
-  - cilium.io
-  resources:
-  - ciliumnetworkpolicies
-  - ciliumnetworkpolicies/status
-{% if cilium_version | regex_replace('v') is version('1.7', '>=') %}
-  - ciliumclusterwidenetworkpolicies
-  - ciliumclusterwidenetworkpolicies/status
-{% endif %}
-  - ciliumendpoints
-  - ciliumendpoints/status
-{% if cilium_version | regex_replace('v') is version('1.6', '>=') %}
-  - ciliumnodes
-  - ciliumnodes/status
-  - ciliumidentities
-  - ciliumidentities/status
-{% endif %}
-{% if cilium_version | regex_replace('v') is version('1.9', '>=') %}
-  - ciliumnetworkpolicies/finalizers
-  - ciliumclusterwidenetworkpolicies/finalizers
-  - ciliumendpoints/finalizers
-  - ciliumnodes/finalizers
-  - ciliumidentities/finalizers
-  - ciliumlocalredirectpolicies
-  - ciliumlocalredirectpolicies/status
-  - ciliumlocalredirectpolicies/finalizers
-{% endif %}
-  verbs:
-  - '*'

roles/network_plugin/cilium/templates/cilium-operator/crb.yml.j2

@@ -0,0 +1,13 @@
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: cilium-operator
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: cilium-operator
+subjects:
+- kind: ServiceAccount
+  name: cilium-operator
+  namespace: kube-system

roles/network_plugin/cilium/templates/cilium-operator/deploy.yml.j2

@@ -45,6 +45,13 @@ spec:
         - args:
             - --debug=$(CILIUM_DEBUG)
             - --config-dir=/tmp/cilium/config-map
+{% if cilium_operator_custom_args is string %}
+            - {{ cilium_operator_custom_args }}
+{% else %}
+{% for flag in cilium_operator_custom_args %}
+            - {{ flag }}
+{% endfor %}
+{% endif %}
           command:
             - cilium-operator
           env:
@@ -69,6 +76,9 @@ spec:
                   key: debug
                   name: cilium-config
                   optional: true
+# We are already mounting the whole ConfigMap as a directory.
+# https://github.com/cilium/cilium/pull/10347
+{% if cilium_version | regex_replace('v') is version('1.8', '<') %}
             - name: CILIUM_CLUSTER_NAME
               valueFrom:
                 configMapKeyRef:
@@ -87,6 +97,7 @@ spec:
                   key: disable-endpoint-crd
                   name: cilium-config
                   optional: true
+{% endif %}
             - name: AWS_ACCESS_KEY_ID
               valueFrom:
                 secretKeyRef:
@@ -146,6 +157,9 @@ spec:
             - mountPath: /tmp/cilium/config-map
               name: cilium-config-path
               readOnly: true
+{% for volume_mount in cilium_operator_extra_volume_mounts %}
+            - {{ volume_mount | to_nice_yaml(indent=2) | indent(14) }}
+{% endfor %}
       dnsPolicy: ClusterFirst
       priorityClassName: system-node-critical
       restartPolicy: Always
@@ -172,3 +186,6 @@ spec:
         - configMap:
             name: cilium-config
           name: cilium-config-path
+{% for volume in cilium_operator_extra_volumes %}
+        - {{ volume | to_nice_yaml(indent=2) | indent(10) }}
+{% endfor %}

roles/network_plugin/cilium/templates/cilium-operator/sa.yml.j2

@@ -4,9 +4,3 @@ kind: ServiceAccount
 metadata:
   name: cilium-operator
   namespace: kube-system
----
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: cilium
-  namespace: kube-system

roles/network_plugin/cilium/templates/cilium/config.yml.j2

@@ -118,12 +118,18 @@ data:
   tunnel: "{{ cilium_tunnel_mode }}"
 
   # Name of the cluster. Only relevant when building a mesh of clusters.
-  cluster-name: default
+  cluster-name: "{{ cilium_cluster_name }}"
 
   # Unique ID of the cluster. Must be unique across all conneted clusters and
   # in the range of 1 and 255. Only relevant when building a mesh of clusters.
   #cluster-id: 1
+{% if cilium_cluster_id is defined %}
+  cluster-id: "{{ cilium_cluster_id }}"
+{% endif %}
 
+# `tofqdns-enable-poller` is deprecated in 1.8, removed in 1.9
+# https://github.com/cilium/cilium/issues/8604
+{% if cilium_version | regex_replace('v') is version('1.9', '<') %}
   # DNS Polling periodically issues a DNS lookup for each `matchName` from
   # cilium-agent. The result is used to regenerate endpoint policy.
   # DNS lookups are repeated with an interval of 5 seconds, and are made for
@@ -144,17 +150,39 @@ data:
   # [0] http://docs.cilium.io/en/stable/policy/language/#dns-based
   # [1] http://docs.cilium.io/en/stable/install/upgrade/#changes-that-may-require-action
   tofqdns-enable-poller: "{{cilium_tofqdns_enable_poller}}"
+{% endif %}
 
+# `wait-bpf-mount` is removed after v1.10.4
+# https://github.com/cilium/cilium/commit/d2217045cb3726a7f823174e086913b69b8090da
+{% if cilium_version | regex_replace('v') is version('1.10.4', '<') %}
   # wait-bpf-mount makes init container wait until bpf filesystem is mounted
   wait-bpf-mount: "false"
+{% endif %}
 
+# `enable-legacy-services` is deprecated in 1.6, removed in 1.9
+# https://github.com/cilium/cilium/pull/10255
+{% if cilium_version | regex_replace('v') is version('1.9', '<') %}
   # Enable legacy services (prior v1.5) to prevent from terminating existing
   # connections with services when upgrading Cilium from < v1.5 to v1.5.
   enable-legacy-services: "{{cilium_enable_legacy_services}}"
+{% endif %}
 
   kube-proxy-replacement: "{{ cilium_kube_proxy_replacement }}"
 
+# `native-routing-cidr` is deprecated in 1.10, removed in 1.12.
+# Replaced by `ipv4-native-routing-cidr`
+# https://github.com/cilium/cilium/pull/16695
+{% if cilium_version | regex_replace('v') is version('1.12', '<') %}
   native-routing-cidr: "{{ cilium_native_routing_cidr }}"
+{% else %}
+  {% if cilium_native_routing_cidr | length %}
+  ipv4-native-routing-cidr: "{{ cilium_native_routing_cidr }}"
+  {% endif %}
+  {% if cilium_native_routing_cidr_ipv6 | length %}
+  ipv6-native-routing-cidr: "{{ cilium_native_routing_cidr_ipv6 }}"
+  {% endif %}
+{% endif %}
+
   auto-direct-node-routes: "{{ cilium_auto_direct_node_routes }}"
 
   operator-api-serve-addr: "{{ cilium_operator_api_serve_addr }}"
@@ -182,11 +210,18 @@ data:
   {{ key }}: "{{ value }}"
 {% endfor %}
 
-  # IPsec based transparent encryption between nodes
+  # Enable transparent network encryption
-{% if cilium_ipsec_enabled %}
+{% if cilium_encryption_enabled %}
+  {% if cilium_encryption_type == "ipsec" %}
   enable-ipsec: "true"
   ipsec-key-file: /etc/ipsec/keys
-  encrypt-node: "false"
+  encrypt-node: "{{ cilium_ipsec_node_encryption }}"
+  {% endif %}
+
+  {% if cilium_encryption_type == "wireguard" %}
+  enable-wireguard: "true"
+  enable-wireguard-userspace-fallback: "{{ cilium_wireguard_userspace_fallback }}"
+  {% endif %}
 {% endif %}
 
   # IPAM settings

roles/network_plugin/cilium/templates/cilium/cr.yml.j2

@@ -0,0 +1,134 @@
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: cilium
+rules:
+- apiGroups:
+  - networking.k8s.io
+  resources:
+{% if cilium_version | regex_replace('v') is version('1.7', '<') %}
+  - ingresses
+{% endif %}
+  - networkpolicies
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - discovery.k8s.io
+  resources:
+  - endpointslices
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - ""
+  resources:
+  - namespaces
+  - services
+  - nodes
+  - endpoints
+{% if cilium_version | regex_replace('v') is version('1.7', '<') %}
+  - componentstatuses
+{% endif %}
+  verbs:
+  - get
+  - list
+  - watch
+{% if cilium_version | regex_replace('v') is version('1.7', '<') %}
+- apiGroups:
+  - extensions
+  resources:
+  - ingresses
+  verbs:
+  - create
+  - get
+  - list
+  - watch
+{% endif %}
+{% if cilium_version | regex_replace('v') is version('1.7', '>') %}
+- apiGroups:
+  - ""
+  resources:
+  - pods/finalizers
+  verbs:
+  - update
+{% endif %}
+- apiGroups:
+  - ""
+  resources:
+  - pods
+  - nodes
+  verbs:
+  - get
+  - list
+  - watch
+  - update
+- apiGroups:
+  - ""
+  resources:
+  - nodes
+  - nodes/status
+  verbs:
+  - patch
+- apiGroups:
+  - apiextensions.k8s.io
+  resources:
+  - customresourcedefinitions
+  verbs:
+  - create
+  - get
+  - list
+  - watch
+  - update
+- apiGroups:
+  - cilium.io
+  resources:
+  - ciliumnetworkpolicies
+  - ciliumnetworkpolicies/status
+{% if cilium_version | regex_replace('v') is version('1.7', '>=') %}
+  - ciliumclusterwidenetworkpolicies
+  - ciliumclusterwidenetworkpolicies/status
+{% endif %}
+  - ciliumendpoints
+  - ciliumendpoints/status
+{% if cilium_version | regex_replace('v') is version('1.6', '>=') %}
+  - ciliumnodes
+  - ciliumnodes/status
+  - ciliumidentities
+  - ciliumidentities/status
+{% endif %}
+{% if cilium_version | regex_replace('v') is version('1.9', '>=') %}
+  - ciliumnetworkpolicies/finalizers
+  - ciliumclusterwidenetworkpolicies/finalizers
+  - ciliumendpoints/finalizers
+  - ciliumnodes/finalizers
+  - ciliumidentities/finalizers
+  - ciliumlocalredirectpolicies
+  - ciliumlocalredirectpolicies/status
+  - ciliumlocalredirectpolicies/finalizers
+{% endif %}
+{% if cilium_version | regex_replace('v') is version('1.10', '>=') %}
+  - ciliumegressnatpolicies
+{% endif %}
+{% if cilium_version | regex_replace('v') is version('1.11', '>=') %}
+  - ciliumendpointslices
+{% endif %}
+{% if cilium_version | regex_replace('v') is version('1.12', '>=') %}
+  - ciliumbgploadbalancerippools
+  - ciliumbgppeeringpolicies
+{% endif %}
+  verbs:
+  - '*'
+{% if cilium_version | regex_replace('v') is version('1.12', '>=') %}
+- apiGroups:
+  - cilium.io
+  resources:
+  - ciliumclusterwideenvoyconfigs
+  - ciliumenvoyconfigs
+  verbs:
+  - list
+  - watch
+{% endif %}

roles/network_plugin/cilium/templates/cilium/crb.yml.j2

@@ -1,19 +1,6 @@
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding
-metadata:
-  name: cilium-operator
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: cilium-operator
-subjects:
-- kind: ServiceAccount
-  name: cilium-operator
-  namespace: kube-system
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
 metadata:
   name: cilium
 roleRef:

roles/network_plugin/cilium/templates/cilium/ds.yml.j2

@@ -1,3 +1,4 @@
+---
 apiVersion: apps/v1
 kind: DaemonSet
 metadata:
@@ -35,6 +36,13 @@ spec:
         - --config-dir=/tmp/cilium/config-map
 {% if cilium_mtu != "" %}
         - --mtu={{ cilium_mtu }}
+{% endif %}
+{% if cilium_agent_custom_args is string %}
+        - {{ cilium_agent_custom_args }}
+{% else %}
+{% for flag in cilium_agent_custom_args %}
+        - {{ flag }}
+{% endfor %}
 {% endif %}
         command:
         - cilium-agent
@@ -57,6 +65,9 @@ spec:
         - name: KUBERNETES_SERVICE_PORT
           value: "{{ kube_apiserver_global_endpoint | urlsplit('port') }}"
 {% endif %}
+{% for env_var in cilium_agent_extra_env_vars %}
+        - {{ env_var | to_nice_yaml(indent=2) | indent(10) }}
+{% endfor %}
         image: "{{cilium_image_repo}}:{{cilium_image_tag}}"
         imagePullPolicy: {{ k8s_image_pull_policy }}
         resources:
@@ -70,11 +81,29 @@ spec:
           postStart:
             exec:
               command:
-              - /cni-install.sh
+              - "/cni-install.sh"
+{% if cilium_version | regex_replace('v') is version('1.10', '>=') %}
+              - "--cni-exclusive={{ cilium_cni_exclusive }}"
+{% endif %}
+{% if cilium_version | regex_replace('v') is version('1.12', '>=') %}
+              - "--log-file={{ cilium_cni_log_file }}"
+{% endif %}
           preStop:
             exec:
               command:
               - /cni-uninstall.sh
+        startupProbe:
+          httpGet:
+            host: '127.0.0.1'
+            path: /healthz
+            port: 9876
+            scheme: HTTP
+            httpHeaders:
+            - name: "brief"
+              value: "true"
+          failureThreshold: 105
+          periodSeconds: 2
+          successThreshold: 1
         livenessProbe:
           httpGet:
             host: '127.0.0.1'
@@ -85,13 +114,23 @@ spec:
             - name: "brief"
               value: "true"
           failureThreshold: 10
-          # The initial delay for the liveness probe is intentionally large to
-          # avoid an endless kill & restart cycle if in the event that the initial
-          # bootstrapping takes longer than expected.
-          initialDelaySeconds: 120
           periodSeconds: 30
           successThreshold: 1
           timeoutSeconds: 5
+        readinessProbe:
+          httpGet:
+            host: 127.0.0.1
+            path: /healthz
+            port: 9876
+            scheme: HTTP
+            httpHeaders:
+            - name: "brief"
+              value: "true"
+          initialDelaySeconds: 5
+          periodSeconds: 30
+          successThreshold: 1
+          failureThreshold: 3
+          timeoutSeconds: 5
         name: cilium-agent
 {% if cilium_enable_prometheus or cilium_enable_hubble_metrics %}
         ports:
@@ -108,35 +147,21 @@ spec:
           name: hubble-metrics
           protocol: TCP
 {% endif %}
-        readinessProbe:
-          httpGet:
-            host: '127.0.0.1'
-            path: /healthz
-            port: 9876
-            scheme: HTTP
-            httpHeaders:
-            - name: "brief"
-              value: "true"
-          failureThreshold: 3
-          initialDelaySeconds: 5
-          periodSeconds: 30
-          successThreshold: 1
-          timeoutSeconds: 5
         securityContext:
-          capabilities:
-            add:
-            - NET_ADMIN
-            - SYS_MODULE
           privileged: true
         volumeMounts:
         - mountPath: /sys/fs/bpf
           name: bpf-maps
+          mountPropagation: Bidirectional
         - mountPath: /var/run/cilium
           name: cilium-run
         - mountPath: /host/opt/cni/bin
           name: cni-path
         - mountPath: /host/etc/cni/net.d
           name: etc-cni-netd
+# pkg/workloads was depreca, removed in 1.7
+# https://github.com/cilium/cilium/pull/9447
+{% if cilium_version | regex_replace('v') is version('1.7', '<') %}
 {% if container_manager == 'docker' %}
         - mountPath: /var/run/docker.sock
           name: docker-socket
@@ -146,6 +171,7 @@ spec:
           mountPath: {{ cri_socket }}
           readOnly: true
 {% endif %}
+{% endif %}
 {% if cilium_identity_allocation_mode == "kvstore" %}
         - mountPath: /var/lib/etcd-config
           name: etcd-config-path
@@ -166,7 +192,7 @@ spec:
           readOnly: true
         - mountPath: /run/xtables.lock
           name: xtables-lock
-{% if cilium_ipsec_enabled %}
+{% if cilium_encryption_enabled and cilium_encryption_type == "ipsec" %}
         - mountPath: /etc/ipsec
           name: cilium-ipsec-secrets
           readOnly: true
@@ -176,9 +202,14 @@ spec:
           name: hubble-tls
           readOnly: true
 {% endif %}
+{% for volume_mount in cilium_agent_extra_volume_mounts %}
+        - {{ volume_mount | to_nice_yaml(indent=2) | indent(10) }}
+{% endfor %}
+# In managed etcd mode, Cilium must be able to resolve the DNS name of the etcd service
+{% if cilium_identity_allocation_mode == "kvstore" %}
       dnsPolicy: ClusterFirstWithHostNet
+{% endif %}
       hostNetwork: true
-      hostPID: false
       initContainers:
       - command:
         - /init-container.sh
@@ -189,18 +220,30 @@ spec:
               key: clean-cilium-state
               name: cilium-config
               optional: true
-        - name: CLEAN_CILIUM_BPF_STATE
+        # CLEAN_CILIUM_BPF_STATE is deprecated in 1.6.
+        # https://github.com/cilium/cilium/pull/7478
+        - name: "{{ cilium_version | regex_replace('v') is version('1.6', '<')| ternary('CLEAN_CILIUM_BPF_STATE','CILIUM_BPF_STATE') }}"
           valueFrom:
             configMapKeyRef:
               key: clean-cilium-bpf-state
               name: cilium-config
               optional: true
+# Removed in 1.11 and up.
+# https://github.com/cilium/cilium/commit/f7a3f59fd74983c600bfce9cac364b76d20849d9
+{% if cilium_version | regex_replace('v') is version('1.11', '<') %}
         - name: CILIUM_WAIT_BPF_MOUNT
           valueFrom:
             configMapKeyRef:
               key: wait-bpf-mount
               name: cilium-config
               optional: true
+{% endif %}
+{% if cilium_kube_proxy_replacement == 'strict' %}
+        - name: KUBERNETES_SERVICE_HOST
+          value: "{{ kube_apiserver_global_endpoint | urlsplit('hostname') }}"
+        - name: KUBERNETES_SERVICE_PORT
+          value: "{{ kube_apiserver_global_endpoint | urlsplit('port') }}"
+{% endif %}
 {% if cilium_version | regex_replace('v') is version('1.9', '<') %}
         image: "{{cilium_init_image_repo}}:{{cilium_init_image_tag}}"
 {% else %}
@@ -209,13 +252,16 @@ spec:
         imagePullPolicy: {{ k8s_image_pull_policy }}
         name: clean-cilium-state
         securityContext:
-          capabilities:
-            add:
-            - NET_ADMIN
           privileged: true
         volumeMounts:
         - mountPath: /sys/fs/bpf
           name: bpf-maps
+{% if cilium_version | regex_replace('v') is version('1.11', '>=') %}
+          # Required to mount cgroup filesystem from the host to cilium agent pod
+        - name: cilium-cgroup
+          mountPath: /run/cilium/cgroupv2
+          mountPropagation: HostToContainer
+{% endif %}
         - mountPath: /var/run/cilium
           name: cilium-run
         resources:
@@ -227,6 +273,11 @@ spec:
       serviceAccount: cilium
       serviceAccountName: cilium
       terminationGracePeriodSeconds: 1
+      hostNetwork: true
+# In managed etcd mode, Cilium must be able to resolve the DNS name of the etcd service
+{% if cilium_identity_allocation_mode == "kvstore" %}
+      dnsPolicy: ClusterFirstWithHostNet
+{% endif %}
       tolerations:
       - operator: Exists
       volumes:
@@ -240,6 +291,9 @@ spec:
           path: /sys/fs/bpf
           type: DirectoryOrCreate
         name: bpf-maps
+# pkg/workloads was deprecated in 1.6, removed in 1.7
+# https://github.com/cilium/cilium/pull/9447
+{% if cilium_version | regex_replace('v') is version('1.7', '<') %}
 {% if container_manager == 'docker' %}
         # To read docker events from the node
       - hostPath:
@@ -252,6 +306,19 @@ spec:
           path: {{ cri_socket }}
           type: Socket
         name: {{ container_manager }}-socket
+{% endif %}
+{% endif %}
+{% if cilium_version | regex_replace('v') is version('1.11', '>=') %}
+      # To mount cgroup2 filesystem on the host
+      - name: hostproc
+        hostPath:
+          path: /proc
+          type: Directory
+      # To keep state between restarts / upgrades for cgroup2 filesystem
+      - name: cilium-cgroup
+        hostPath:
+          path: /run/cilium/cgroupv2
+          type: DirectoryOrCreate
 {% endif %}
         # To install cilium cni plugin in the host
       - hostPath:
@@ -275,7 +342,8 @@ spec:
 {% if cilium_identity_allocation_mode == "kvstore" %}
         # To read the etcd config stored in config maps
       - configMap:
-          defaultMode: 420
+          # note: the leading zero means this number is in octal representation: do not remove it
+          defaultMode: 0400
           items:
           - key: etcd-config
             path: etcd.config
@@ -289,14 +357,15 @@ spec:
         # To read the clustermesh configuration
       - name: clustermesh-secrets
         secret:
-          defaultMode: 420
+          # note: the leading zero means this number is in octal representation: do not remove it
+          defaultMode: 0400
           optional: true
           secretName: cilium-clustermesh
         # To read the configuration from the config map
       - configMap:
           name: cilium-config
         name: cilium-config-path
-{% if cilium_ipsec_enabled %}
+{% if cilium_encryption_enabled and cilium_encryption_type == "ipsec" %}
       - name: cilium-ipsec-secrets
         secret:
           secretName: cilium-ipsec-keys
@@ -304,6 +373,8 @@ spec:
 {% if cilium_hubble_install %}
       - name: hubble-tls
         projected:
+          # note: the leading zero means this number is in octal representation: do not remove it
+          defaultMode: 0400
           sources:
           - secret:
               name: hubble-server-certs

roles/network_plugin/cilium/templates/cilium/sa.yml.j2

@@ -0,0 +1,6 @@
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: cilium
+  namespace: kube-system