diff --git a/docs/cilium.md b/docs/cilium.md index 4ce441e0b..665e319ae 100644 --- a/docs/cilium.md +++ b/docs/cilium.md @@ -12,10 +12,51 @@ the external loadbalancer (even from a node in the control plane) and if there is no external load balancer It will ignore any local load balancer deployed by Kubespray and **only contacts the first master**. +## Cilium Operator + +Unlike some operators, Cilium Operator does not exist for installation purposes. +> The Cilium Operator is responsible for managing duties in the cluster which should logically be handled once for the entire cluster, rather than once for each node in the cluster. + +### Adding custom flags to the Cilium Operator + +You can set additional cilium-operator container arguments using `cilium_operator_custom_args`. +This is an advanced option, and you should only use it if you know what you are doing. + +Accepts an array or a string. + +```yml +cilium_operator_custom_args: ["--foo=bar", "--baz=qux"] +``` + +or + +```yml +cilium_operator_custom_args: "--foo=bar" +``` + +You do not need to add a custom flag to enable debugging. Instead, feel free to use the `CILIUM_DEBUG` variable. + +### Adding extra volumes and mounting them + +You can use `cilium_operator_extra_volumes` to add extra volumes to the Cilium Operator, and use `cilium_operator_extra_volume_mounts` to mount those volumes. +This is an advanced option, and you should only use it if you know what you are doing. + +```yml +cilium_operator_extra_volumes: + - configMap: + name: foo + name: foo-mount-path + +cilium_operator_extra_volume_mounts: + - mountPath: /tmp/foo/bar + name: foo-mount-path + readOnly: true +``` + ## Choose Cilium version ```yml -cilium_version: v1.11.0 +cilium_version: v1.11.3 ``` ## Add variable to config @@ -39,6 +80,47 @@ Cilium currently supports two different identity allocation modes: - These can be queried with `kubectl get ciliumid` - "kvstore" stores identities in an etcd kvstore. +## Enable Transparent Encryption + +Cilium supports the transparent encryption of Cilium-managed host traffic and +traffic between Cilium-managed endpoints either using IPsec or Wireguard. + +Wireguard option is only available in Cilium 1.10.0 and newer. + +### IPsec Encryption + +For further information, make sure to check the official [Cilium documentation.](https://docs.cilium.io/en/stable/gettingstarted/encryption-ipsec/) + +To enable IPsec encryption, you just need to set three variables. + +```yml +cilium_encryption_enabled: true +cilium_encryption_type: "ipsec" +``` + +The third variable is `cilium_ipsec_key.` You need to create a secret key string for this variable. +Kubespray does not automate this process. +Cilium documentation currently recommends creating a key using the following command: + +```shell +echo "3 rfc4106(gcm(aes)) $(echo $(dd if=/dev/urandom count=20 bs=1 2> /dev/null | xxd -p -c 64)) 128" +``` + +Note that Kubespray handles secret creation. So you only need to pass the key as the `cilium_ipsec_key` variable. + +### Wireguard Encryption + +For further information, make sure to check the official [Cilium documentation.](https://docs.cilium.io/en/stable/gettingstarted/encryption-wireguard/) + +To enable Wireguard encryption, you just need to set two variables. + +```yml +cilium_encryption_enabled: true +cilium_encryption_type: "wireguard" +``` + +Kubespray currently supports Linux distributions with Wireguard Kernel mode on Linux 5.6 and newer. + ## Install Cilium Hubble k8s-net-cilium.yml: @@ -68,6 +150,6 @@ cilium_hubble_metrics: - flow - icmp - http -``` +``` [More](https://docs.cilium.io/en/v1.9/operations/metrics/#hubble-exported-metrics) diff --git a/inventory/sample/group_vars/k8s_cluster/k8s-net-cilium.yml b/inventory/sample/group_vars/k8s_cluster/k8s-net-cilium.yml index c35b95182..922f42881 100644 --- a/inventory/sample/group_vars/k8s_cluster/k8s-net-cilium.yml +++ b/inventory/sample/group_vars/k8s_cluster/k8s-net-cilium.yml @@ -1,4 +1,47 @@ # see roles/network_plugin/cilium/defaults/main.yml -# cilium_version: "v1.11.0" +# cilium_version: "v1.11.3" # cilium_identity_allocation_mode: kvstore # kvstore or crd + +# For adding and mounting extra volumes to the cilium operator +# cilium_operator_extra_volumes: [] +# cilium_operator_extra_volume_mounts: [] + +# Name of the cluster. Only relevant when building a mesh of clusters. +# cilium_cluster_name: default + +# Unique ID of the cluster. Must be unique across all conneted clusters and +# in the range of 1 and 255. Only relevant when building a mesh of clusters. +# This value is not defined by default +# cluster-id: + +# Allows to explicitly specify the IPv4 CIDR for native routing. +# When specified, Cilium assumes networking for this CIDR is preconfigured and +# hands traffic destined for that range to the Linux network stack without +# applying any SNAT. +# Generally speaking, specifying a native routing CIDR implies that Cilium can +# depend on the underlying networking stack to route packets to their +# destination. To offer a concrete example, if Cilium is configured to use +# direct routing and the Kubernetes CIDR is included in the native routing CIDR, +# the user must configure the routes to reach pods, either manually or by +# setting the auto-direct-node-routes flag. +# cilium_native_routing_cidr: "" + +# Allows to explicitly specify the IPv6 CIDR for native routing. +# cilium_native_routing_cidr_ipv6: "" + +# Encryption +# Enable transparent network encryption. +# cilium_encryption_enabled: false + +# Encryption method. Can be either ipsec or wireguard. +# Only effective when `cilium_encryption_enabled` is set to true. +# cilium_encryption_type: "ipsec" + +# Enable encryption for pure node to node traffic. +# This option is only effective when `cilium_encryption_type` is set to `ipsec`. +# cilium_ipsec_node_encryption: "false" + +# Enables the fallback to the user-space implementation. +# This option is only effective when `cilium_encryption_type` is set to `wireguard`. +# cilium_wireguard_userspace_fallback: "false" diff --git a/roles/download/defaults/main.yml b/roles/download/defaults/main.yml index 658d9bb70..6ba48bb3c 100644 --- a/roles/download/defaults/main.yml +++ b/roles/download/defaults/main.yml @@ -110,7 +110,7 @@ flannel_cni_version: "v1.0.1" cni_version: "v1.0.1" weave_version: 2.8.1 pod_infra_version: "3.3" -cilium_version: "v1.11.1" +cilium_version: "v1.11.3" kube_ovn_version: "v1.8.1" kube_router_version: "v1.4.0" multus_version: "v3.8" diff --git a/roles/network_plugin/cilium/defaults/main.yml b/roles/network_plugin/cilium/defaults/main.yml index e36a19906..867cd9a87 100644 --- a/roles/network_plugin/cilium/defaults/main.yml +++ b/roles/network_plugin/cilium/defaults/main.yml @@ -48,7 +48,11 @@ cilium_kube_proxy_replacement: probe # to prevent service disruptions. See also: # http://docs.cilium.io/en/stable/install/upgrade/#changes-that-may-require-action cilium_preallocate_bpf_maps: false + +# `cilium_tofqdns_enable_poller` is deprecated in 1.8, removed in 1.9 cilium_tofqdns_enable_poller: false + +# `cilium_enable_legacy_services` is deprecated in 1.6, removed in 1.9 cilium_enable_legacy_services: false # Deploy cilium even if kube_network_plugin is not cilium. @@ -62,10 +66,38 @@ cilium_deploy_additionally: false # make this work. Please refer to the cilium documentation for more # information about this kind of setups. cilium_auto_direct_node_routes: false + +# Allows to explicitly specify the IPv4 CIDR for native routing. +# When specified, Cilium assumes networking for this CIDR is preconfigured and +# hands traffic destined for that range to the Linux network stack without +# applying any SNAT. +# Generally speaking, specifying a native routing CIDR implies that Cilium can +# depend on the underlying networking stack to route packets to their +# destination. To offer a concrete example, if Cilium is configured to use +# direct routing and the Kubernetes CIDR is included in the native routing CIDR, +# the user must configure the routes to reach pods, either manually or by +# setting the auto-direct-node-routes flag. cilium_native_routing_cidr: "" -# IPsec based transparent encryption between nodes -cilium_ipsec_enabled: false +# Allows to explicitly specify the IPv6 CIDR for native routing. +cilium_native_routing_cidr_ipv6: "" + +# Enable transparent network encryption. +cilium_encryption_enabled: false + +# Encryption method. Can be either ipsec or wireguard. +# Only effective when `cilium_encryption_enabled` is set to true. +cilium_encryption_type: "ipsec" + +# Enable encryption for pure node to node traffic. +# This option is only effective when `cilium_encryption_type` is set to `ipsec`. +cilium_ipsec_node_encryption: "false" + +# If your kernel or distribution does not support WireGuard, Cilium agent can be configured to fall back on the user-space implementation. +# When this flag is enabled and Cilium detects that the kernel has no native support for WireGuard, +# it will fallback on the wireguard-go user-space implementation of WireGuard. +# This option is only effective when `cilium_encryption_type` is set to `wireguard`. +cilium_wireguard_userspace_fallback: "false" # Hubble ### Enable Hubble without install @@ -89,6 +121,15 @@ cilium_hubble_tls_generate: false # https://docs.cilium.io/en/v1.9/concepts/networking/ipam/ cilium_ipam_mode: kubernetes +# Extra arguments for the Cilium agent +cilium_agent_custom_args: [] + +# For adding and mounting extra volumes to the cilium agent +cilium_agent_extra_volumes: [] +cilium_agent_extra_volume_mounts: [] + +cilium_agent_extra_env_vars: [] + # The address at which the cillium operator bind health check api cilium_operator_api_serve_addr: "127.0.0.1:9234" @@ -97,3 +138,23 @@ cilium_operator_api_serve_addr: "127.0.0.1:9234" ## var1: "value1" ## var2: "value2" cilium_config_extra_vars: {} + +# For adding and mounting extra volumes to the cilium operator +cilium_operator_extra_volumes: [] +cilium_operator_extra_volume_mounts: [] + +# Extra arguments for the Cilium Operator +cilium_operator_custom_args: [] + +# Name of the cluster. Only relevant when building a mesh of clusters. +cilium_cluster_name: default + +# Make Cilium take ownership over the `/etc/cni/net.d` directory on the node, renaming all non-Cilium CNI configurations to `*.cilium_bak`. +# This ensures no Pods can be scheduled using other CNI plugins during Cilium agent downtime. +# Available for Cilium v1.10 and up. +cilium_cni_exclusive: "true" + +# Configure the log file for CNI logging with retention policy of 7 days. +# Disable CNI file logging by setting this field to empty explicitly. +# Available for Cilium v1.12 and up. +cilium_cni_log_file: "/var/run/cilium/cilium-cni.log" diff --git a/roles/network_plugin/cilium/tasks/apply.yml b/roles/network_plugin/cilium/tasks/apply.yml index 89ccb1e56..ac323a4aa 100644 --- a/roles/network_plugin/cilium/tasks/apply.yml +++ b/roles/network_plugin/cilium/tasks/apply.yml @@ -5,7 +5,7 @@ namespace: "kube-system" kubectl: "{{ bin_dir }}/kubectl" resource: "{{ item.item.type }}" - filename: "{{ kube_config_dir }}/{{ item.item.file }}" + filename: "{{ kube_config_dir }}/{{ item.item.name }}-{{ item.item.file }}" state: "latest" loop: "{{ cilium_node_manifests.results }}" when: inventory_hostname == groups['kube_control_plane'][0] and not item is skipped @@ -25,7 +25,7 @@ namespace: "kube-system" kubectl: "{{ bin_dir }}/kubectl" resource: "{{ item.item.type }}" - filename: "{{ kube_config_dir }}/addons/hubble/{{ item.item.file }}" + filename: "{{ kube_config_dir }}/addons/hubble/{{ item.item.name }}-{{ item.item.file }}" state: "latest" loop: "{{ cilium_hubble_manifests.results }}" when: diff --git a/roles/network_plugin/cilium/tasks/check.yml b/roles/network_plugin/cilium/tasks/check.yml index 086eb91b4..fffa1b53a 100644 --- a/roles/network_plugin/cilium/tasks/check.yml +++ b/roles/network_plugin/cilium/tasks/check.yml @@ -1,14 +1,66 @@ --- -- name: Cilium | Check cilium_ipsec_enabled variables +- name: Cilium | Check Cilium encryption `cilium_ipsec_key` for ipsec assert: that: - "cilium_ipsec_key is defined" - msg: "cilium_ipsec_key should be defined to use cilium_ipsec_enabled" + msg: "cilium_ipsec_key should be defined to enable encryption using ipsec" when: - - cilium_ipsec_enabled + - cilium_encryption_enabled + - cilium_encryption_type == "ipsec" - cilium_tunnel_mode in ['vxlan'] +# TODO: Clean this task up when we drop backward compatibility support for `cilium_ipsec_enabled` +- name: Stop if `cilium_ipsec_enabled` is defined and `cilium_encryption_type` is not `ipsec` + assert: + that: cilium_encryption_type == 'ipsec' + msg: > + It is not possible to use `cilium_ipsec_enabled` when `cilium_encryption_type` is set to {{ cilium_encryption_type }}. + when: + - cilium_ipsec_enabled is defined + - cilium_ipsec_enabled + - kube_network_plugin == 'cilium' or cilium_deploy_additionally | default(false) | bool + +- name: Stop if kernel version is too low for Cilium Wireguard encryption + assert: + that: ansible_kernel.split('-')[0] is version('5.6.0', '>=') + when: + - kube_network_plugin == 'cilium' or cilium_deploy_additionally | default(false) | bool + - cilium_encryption_enabled + - cilium_encryption_type == "wireguard" + - not ignore_assert_errors + - name: Stop if bad Cilium identity allocation mode assert: that: cilium_identity_allocation_mode in ['crd', 'kvstore'] msg: "cilium_identity_allocation_mode must be either 'crd' or 'kvstore'" + +- name: Stop if bad Cilium Cluster ID + assert: + that: + - cilium_cluster_id <= 255 + - cilium_cluster_id >= 0 + msg: "'cilium_cluster_id' must be between 1 and 255" + when: cilium_cluster_id is defined + +- name: Stop if bad encryption type + assert: + that: cilium_encryption_type in ['ipsec', 'wireguard'] + msg: "cilium_encryption_type must be either 'ipsec' or 'wireguard'" + when: cilium_encryption_enabled + +- name: Stop if `cilium_encryption_type` is set to "wireguard" and cilium_version is < v1.10.0 + assert: + that: cilium_version | regex_replace('v') is version('1.10', '>') + msg: "cilium_encryption_type is set to 'wireguard' but cilium_version is < v1.10.0" + when: + - cilium_encryption_enabled + - cilium_encryption_type == "wireguard" + +# TODO: Clean this task up when we drop backward compatibility support for `cilium_ipsec_enabled` +- name: Set `cilium_encryption_type` to "ipsec" and if `cilium_ipsec_enabled` is true + set_fact: + cilium_encryption_type: ipsec + cilium_encryption_enabled: true + when: + - cilium_ipsec_enabled is defined + - cilium_ipsec_enabled diff --git a/roles/network_plugin/cilium/tasks/install.yml b/roles/network_plugin/cilium/tasks/install.yml index c394b6119..a948f7ec7 100644 --- a/roles/network_plugin/cilium/tasks/install.yml +++ b/roles/network_plugin/cilium/tasks/install.yml @@ -43,17 +43,20 @@ - name: Cilium | Create Cilium node manifests template: - src: "{{ item.file }}.j2" - dest: "{{ kube_config_dir }}/{{ item.file }}" + src: "{{ item.name }}/{{ item.file }}.j2" + dest: "{{ kube_config_dir }}/{{ item.name }}-{{ item.file }}" mode: 0644 loop: - - {name: cilium, file: cilium-config.yml, type: cm} - - {name: cilium, file: cilium-crb.yml, type: clusterrolebinding} - - {name: cilium, file: cilium-cr.yml, type: clusterrole} - - {name: cilium, file: cilium-secret.yml, type: secret, when: "{{ cilium_ipsec_enabled }}"} - - {name: cilium, file: cilium-ds.yml, type: ds} - - {name: cilium, file: cilium-deploy.yml, type: deploy} - - {name: cilium, file: cilium-sa.yml, type: sa} + - {name: cilium, file: config.yml, type: cm} + - {name: cilium-operator, file: crb.yml, type: clusterrolebinding} + - {name: cilium-operator, file: cr.yml, type: clusterrole} + - {name: cilium, file: crb.yml, type: clusterrolebinding} + - {name: cilium, file: cr.yml, type: clusterrole} + - {name: cilium, file: secret.yml, type: secret, when: "{{ cilium_encryption_enabled and cilium_encryption_type == 'ipsec' }}"} + - {name: cilium, file: ds.yml, type: ds} + - {name: cilium-operator, file: deploy.yml, type: deploy} + - {name: cilium-operator, file: sa.yml, type: sa} + - {name: cilium, file: sa.yml, type: sa} register: cilium_node_manifests when: - inventory_hostname in groups['kube_control_plane'] @@ -61,18 +64,18 @@ - name: Cilium | Create Cilium Hubble manifests template: - src: "{{ item.file }}.j2" - dest: "{{ kube_config_dir }}/addons/hubble/{{ item.file }}" + src: "{{ item.name }}/{{ item.file }}.j2" + dest: "{{ kube_config_dir }}/addons/hubble/{{ item.name }}-{{ item.file }}" mode: 0644 loop: - - {name: hubble, file: hubble-config.yml, type: cm} - - {name: hubble, file: hubble-crb.yml, type: clusterrolebinding} - - {name: hubble, file: hubble-cr.yml, type: clusterrole} - - {name: hubble, file: hubble-cronjob.yml, type: cronjob, when: "{{ cilium_hubble_tls_generate }}"} - - {name: hubble, file: hubble-deploy.yml, type: deploy} - - {name: hubble, file: hubble-job.yml, type: job, when: "{{ cilium_hubble_tls_generate }}"} - - {name: hubble, file: hubble-sa.yml, type: sa} - - {name: hubble, file: hubble-service.yml, type: service} + - {name: hubble, file: config.yml, type: cm} + - {name: hubble, file: crb.yml, type: clusterrolebinding} + - {name: hubble, file: cr.yml, type: clusterrole} + - {name: hubble, file: cronjob.yml, type: cronjob, when: "{{ cilium_hubble_tls_generate }}"} + - {name: hubble, file: deploy.yml, type: deploy} + - {name: hubble, file: job.yml, type: job, when: "{{ cilium_hubble_tls_generate }}"} + - {name: hubble, file: sa.yml, type: sa} + - {name: hubble, file: service.yml, type: service} register: cilium_hubble_manifests when: - inventory_hostname == groups['kube_control_plane'][0] diff --git a/roles/network_plugin/cilium/templates/cilium-cr.yml.j2 b/roles/network_plugin/cilium/templates/cilium-operator/cr.yml.j2 similarity index 60% rename from roles/network_plugin/cilium/templates/cilium-cr.yml.j2 rename to roles/network_plugin/cilium/templates/cilium-operator/cr.yml.j2 index 4fbbf173a..d23346caa 100644 --- a/roles/network_plugin/cilium/templates/cilium-cr.yml.j2 +++ b/roles/network_plugin/cilium/templates/cilium-operator/cr.yml.j2 @@ -23,6 +23,15 @@ rules: - get - list - watch +{% if cilium_version | regex_replace('v') is version('1.10', '>=') %} +- apiGroups: + - "" + resources: + # to perform LB IP allocation for BGP + - services/status + verbs: + - update +{% endif %} - apiGroups: - "" resources: @@ -68,9 +77,22 @@ rules: - ciliumlocalredirectpolicies - ciliumlocalredirectpolicies/status - ciliumlocalredirectpolicies/finalizers +{% endif %} +{% if cilium_version | regex_replace('v') is version('1.11', '>=') %} + - ciliumendpointslices +{% endif %} +{% if cilium_version | regex_replace('v') is version('1.12', '>=') %} + - ciliumbgploadbalancerippools + - ciliumbgppeeringpolicies + - ciliumenvoyconfigs {% endif %} verbs: - - '*' + - get + - list + - watch + - create + - update + - delete - apiGroups: - apiextensions.k8s.io resources: @@ -100,109 +122,3 @@ rules: - get - update {% endif %} ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: cilium -rules: -- apiGroups: - - networking.k8s.io - resources: -{% if cilium_version | regex_replace('v') is version('1.7', '<') %} - - ingresses -{% endif %} - - networkpolicies - verbs: - - get - - list - - watch -- apiGroups: - - discovery.k8s.io - resources: - - endpointslices - verbs: - - get - - list - - watch -- apiGroups: - - "" - resources: - - namespaces - - services - - nodes - - endpoints -{% if cilium_version | regex_replace('v') is version('1.7', '<') %} - - componentstatuses -{% endif %} - verbs: - - get - - list - - watch -{% if cilium_version | regex_replace('v') is version('1.7', '<') %} -- apiGroups: - - extensions - resources: - - ingresses - verbs: - - create - - get - - list - - watch -{% endif %} -- apiGroups: - - "" - resources: - - pods - - nodes - verbs: - - get - - list - - watch - - update -- apiGroups: - - "" - resources: - - nodes - - nodes/status - verbs: - - patch -- apiGroups: - - apiextensions.k8s.io - resources: - - customresourcedefinitions - verbs: - - create - - get - - list - - watch - - update -- apiGroups: - - cilium.io - resources: - - ciliumnetworkpolicies - - ciliumnetworkpolicies/status -{% if cilium_version | regex_replace('v') is version('1.7', '>=') %} - - ciliumclusterwidenetworkpolicies - - ciliumclusterwidenetworkpolicies/status -{% endif %} - - ciliumendpoints - - ciliumendpoints/status -{% if cilium_version | regex_replace('v') is version('1.6', '>=') %} - - ciliumnodes - - ciliumnodes/status - - ciliumidentities - - ciliumidentities/status -{% endif %} -{% if cilium_version | regex_replace('v') is version('1.9', '>=') %} - - ciliumnetworkpolicies/finalizers - - ciliumclusterwidenetworkpolicies/finalizers - - ciliumendpoints/finalizers - - ciliumnodes/finalizers - - ciliumidentities/finalizers - - ciliumlocalredirectpolicies - - ciliumlocalredirectpolicies/status - - ciliumlocalredirectpolicies/finalizers -{% endif %} - verbs: - - '*' diff --git a/roles/network_plugin/cilium/templates/cilium-operator/crb.yml.j2 b/roles/network_plugin/cilium/templates/cilium-operator/crb.yml.j2 new file mode 100644 index 000000000..00f083535 --- /dev/null +++ b/roles/network_plugin/cilium/templates/cilium-operator/crb.yml.j2 @@ -0,0 +1,13 @@ +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: cilium-operator +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cilium-operator +subjects: +- kind: ServiceAccount + name: cilium-operator + namespace: kube-system diff --git a/roles/network_plugin/cilium/templates/cilium-deploy.yml.j2 b/roles/network_plugin/cilium/templates/cilium-operator/deploy.yml.j2 similarity index 89% rename from roles/network_plugin/cilium/templates/cilium-deploy.yml.j2 rename to roles/network_plugin/cilium/templates/cilium-operator/deploy.yml.j2 index 44dcea2b6..a5813d3d4 100644 --- a/roles/network_plugin/cilium/templates/cilium-deploy.yml.j2 +++ b/roles/network_plugin/cilium/templates/cilium-operator/deploy.yml.j2 @@ -45,6 +45,13 @@ spec: - args: - --debug=$(CILIUM_DEBUG) - --config-dir=/tmp/cilium/config-map +{% if cilium_operator_custom_args is string %} + - {{ cilium_operator_custom_args }} +{% else %} +{% for flag in cilium_operator_custom_args %} + - {{ flag }} +{% endfor %} +{% endif %} command: - cilium-operator env: @@ -69,6 +76,9 @@ spec: key: debug name: cilium-config optional: true +# We are already mounting the whole ConfigMap as a directory. +# https://github.com/cilium/cilium/pull/10347 +{% if cilium_version | regex_replace('v') is version('1.8', '<') %} - name: CILIUM_CLUSTER_NAME valueFrom: configMapKeyRef: @@ -87,6 +97,7 @@ spec: key: disable-endpoint-crd name: cilium-config optional: true +{% endif %} - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: @@ -146,6 +157,9 @@ spec: - mountPath: /tmp/cilium/config-map name: cilium-config-path readOnly: true +{% for volume_mount in cilium_operator_extra_volume_mounts %} + - {{ volume_mount | to_nice_yaml(indent=2) | indent(14) }} +{% endfor %} dnsPolicy: ClusterFirst priorityClassName: system-node-critical restartPolicy: Always @@ -172,3 +186,6 @@ spec: - configMap: name: cilium-config name: cilium-config-path +{% for volume in cilium_operator_extra_volumes %} + - {{ volume | to_nice_yaml(indent=2) | indent(10) }} +{% endfor %} diff --git a/roles/network_plugin/cilium/templates/cilium-sa.yml.j2 b/roles/network_plugin/cilium/templates/cilium-operator/sa.yml.j2 similarity index 52% rename from roles/network_plugin/cilium/templates/cilium-sa.yml.j2 rename to roles/network_plugin/cilium/templates/cilium-operator/sa.yml.j2 index 2c5c0cb19..c5d189364 100644 --- a/roles/network_plugin/cilium/templates/cilium-sa.yml.j2 +++ b/roles/network_plugin/cilium/templates/cilium-operator/sa.yml.j2 @@ -4,9 +4,3 @@ kind: ServiceAccount metadata: name: cilium-operator namespace: kube-system ---- -apiVersion: v1 -kind: ServiceAccount -metadata: - name: cilium - namespace: kube-system diff --git a/roles/network_plugin/cilium/templates/cilium-config.yml.j2 b/roles/network_plugin/cilium/templates/cilium/config.yml.j2 similarity index 83% rename from roles/network_plugin/cilium/templates/cilium-config.yml.j2 rename to roles/network_plugin/cilium/templates/cilium/config.yml.j2 index 820d214c7..09d3dbfbc 100644 --- a/roles/network_plugin/cilium/templates/cilium-config.yml.j2 +++ b/roles/network_plugin/cilium/templates/cilium/config.yml.j2 @@ -118,12 +118,18 @@ data: tunnel: "{{ cilium_tunnel_mode }}" # Name of the cluster. Only relevant when building a mesh of clusters. - cluster-name: default + cluster-name: "{{ cilium_cluster_name }}" # Unique ID of the cluster. Must be unique across all conneted clusters and # in the range of 1 and 255. Only relevant when building a mesh of clusters. #cluster-id: 1 +{% if cilium_cluster_id is defined %} + cluster-id: "{{ cilium_cluster_id }}" +{% endif %} +# `tofqdns-enable-poller` is deprecated in 1.8, removed in 1.9 +# https://github.com/cilium/cilium/issues/8604 +{% if cilium_version | regex_replace('v') is version('1.9', '<') %} # DNS Polling periodically issues a DNS lookup for each `matchName` from # cilium-agent. The result is used to regenerate endpoint policy. # DNS lookups are repeated with an interval of 5 seconds, and are made for @@ -144,17 +150,39 @@ data: # [0] http://docs.cilium.io/en/stable/policy/language/#dns-based # [1] http://docs.cilium.io/en/stable/install/upgrade/#changes-that-may-require-action tofqdns-enable-poller: "{{cilium_tofqdns_enable_poller}}" +{% endif %} +# `wait-bpf-mount` is removed after v1.10.4 +# https://github.com/cilium/cilium/commit/d2217045cb3726a7f823174e086913b69b8090da +{% if cilium_version | regex_replace('v') is version('1.10.4', '<') %} # wait-bpf-mount makes init container wait until bpf filesystem is mounted wait-bpf-mount: "false" +{% endif %} +# `enable-legacy-services` is deprecated in 1.6, removed in 1.9 +# https://github.com/cilium/cilium/pull/10255 +{% if cilium_version | regex_replace('v') is version('1.9', '<') %} # Enable legacy services (prior v1.5) to prevent from terminating existing # connections with services when upgrading Cilium from < v1.5 to v1.5. enable-legacy-services: "{{cilium_enable_legacy_services}}" +{% endif %} kube-proxy-replacement: "{{ cilium_kube_proxy_replacement }}" +# `native-routing-cidr` is deprecated in 1.10, removed in 1.12. +# Replaced by `ipv4-native-routing-cidr` +# https://github.com/cilium/cilium/pull/16695 +{% if cilium_version | regex_replace('v') is version('1.12', '<') %} native-routing-cidr: "{{ cilium_native_routing_cidr }}" +{% else %} + {% if cilium_native_routing_cidr | length %} + ipv4-native-routing-cidr: "{{ cilium_native_routing_cidr }}" + {% endif %} + {% if cilium_native_routing_cidr_ipv6 | length %} + ipv6-native-routing-cidr: "{{ cilium_native_routing_cidr_ipv6 }}" + {% endif %} +{% endif %} + auto-direct-node-routes: "{{ cilium_auto_direct_node_routes }}" operator-api-serve-addr: "{{ cilium_operator_api_serve_addr }}" @@ -182,11 +210,18 @@ data: {{ key }}: "{{ value }}" {% endfor %} - # IPsec based transparent encryption between nodes -{% if cilium_ipsec_enabled %} + # Enable transparent network encryption +{% if cilium_encryption_enabled %} + {% if cilium_encryption_type == "ipsec" %} enable-ipsec: "true" ipsec-key-file: /etc/ipsec/keys - encrypt-node: "false" + encrypt-node: "{{ cilium_ipsec_node_encryption }}" + {% endif %} + + {% if cilium_encryption_type == "wireguard" %} + enable-wireguard: "true" + enable-wireguard-userspace-fallback: "{{ cilium_wireguard_userspace_fallback }}" + {% endif %} {% endif %} # IPAM settings diff --git a/roles/network_plugin/cilium/templates/cilium/cr.yml.j2 b/roles/network_plugin/cilium/templates/cilium/cr.yml.j2 new file mode 100644 index 000000000..02d8e6bf7 --- /dev/null +++ b/roles/network_plugin/cilium/templates/cilium/cr.yml.j2 @@ -0,0 +1,134 @@ +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: cilium +rules: +- apiGroups: + - networking.k8s.io + resources: +{% if cilium_version | regex_replace('v') is version('1.7', '<') %} + - ingresses +{% endif %} + - networkpolicies + verbs: + - get + - list + - watch +- apiGroups: + - discovery.k8s.io + resources: + - endpointslices + verbs: + - get + - list + - watch +- apiGroups: + - "" + resources: + - namespaces + - services + - nodes + - endpoints +{% if cilium_version | regex_replace('v') is version('1.7', '<') %} + - componentstatuses +{% endif %} + verbs: + - get + - list + - watch +{% if cilium_version | regex_replace('v') is version('1.7', '<') %} +- apiGroups: + - extensions + resources: + - ingresses + verbs: + - create + - get + - list + - watch +{% endif %} +{% if cilium_version | regex_replace('v') is version('1.7', '>') %} +- apiGroups: + - "" + resources: + - pods/finalizers + verbs: + - update +{% endif %} +- apiGroups: + - "" + resources: + - pods + - nodes + verbs: + - get + - list + - watch + - update +- apiGroups: + - "" + resources: + - nodes + - nodes/status + verbs: + - patch +- apiGroups: + - apiextensions.k8s.io + resources: + - customresourcedefinitions + verbs: + - create + - get + - list + - watch + - update +- apiGroups: + - cilium.io + resources: + - ciliumnetworkpolicies + - ciliumnetworkpolicies/status +{% if cilium_version | regex_replace('v') is version('1.7', '>=') %} + - ciliumclusterwidenetworkpolicies + - ciliumclusterwidenetworkpolicies/status +{% endif %} + - ciliumendpoints + - ciliumendpoints/status +{% if cilium_version | regex_replace('v') is version('1.6', '>=') %} + - ciliumnodes + - ciliumnodes/status + - ciliumidentities + - ciliumidentities/status +{% endif %} +{% if cilium_version | regex_replace('v') is version('1.9', '>=') %} + - ciliumnetworkpolicies/finalizers + - ciliumclusterwidenetworkpolicies/finalizers + - ciliumendpoints/finalizers + - ciliumnodes/finalizers + - ciliumidentities/finalizers + - ciliumlocalredirectpolicies + - ciliumlocalredirectpolicies/status + - ciliumlocalredirectpolicies/finalizers +{% endif %} +{% if cilium_version | regex_replace('v') is version('1.10', '>=') %} + - ciliumegressnatpolicies +{% endif %} +{% if cilium_version | regex_replace('v') is version('1.11', '>=') %} + - ciliumendpointslices +{% endif %} +{% if cilium_version | regex_replace('v') is version('1.12', '>=') %} + - ciliumbgploadbalancerippools + - ciliumbgppeeringpolicies +{% endif %} + verbs: + - '*' +{% if cilium_version | regex_replace('v') is version('1.12', '>=') %} +- apiGroups: + - cilium.io + resources: + - ciliumclusterwideenvoyconfigs + - ciliumenvoyconfigs + verbs: + - list + - watch +{% endif %} diff --git a/roles/network_plugin/cilium/templates/cilium-crb.yml.j2 b/roles/network_plugin/cilium/templates/cilium/crb.yml.j2 similarity index 59% rename from roles/network_plugin/cilium/templates/cilium-crb.yml.j2 rename to roles/network_plugin/cilium/templates/cilium/crb.yml.j2 index 5bc7b2779..f7516d707 100644 --- a/roles/network_plugin/cilium/templates/cilium-crb.yml.j2 +++ b/roles/network_plugin/cilium/templates/cilium/crb.yml.j2 @@ -1,19 +1,6 @@ --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding -metadata: - name: cilium-operator -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cilium-operator -subjects: -- kind: ServiceAccount - name: cilium-operator - namespace: kube-system ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding metadata: name: cilium roleRef: diff --git a/roles/network_plugin/cilium/templates/cilium-ds.yml.j2 b/roles/network_plugin/cilium/templates/cilium/ds.yml.j2 similarity index 71% rename from roles/network_plugin/cilium/templates/cilium-ds.yml.j2 rename to roles/network_plugin/cilium/templates/cilium/ds.yml.j2 index 9589465ca..a7ff207a2 100644 --- a/roles/network_plugin/cilium/templates/cilium-ds.yml.j2 +++ b/roles/network_plugin/cilium/templates/cilium/ds.yml.j2 @@ -1,3 +1,4 @@ +--- apiVersion: apps/v1 kind: DaemonSet metadata: @@ -35,6 +36,13 @@ spec: - --config-dir=/tmp/cilium/config-map {% if cilium_mtu != "" %} - --mtu={{ cilium_mtu }} +{% endif %} +{% if cilium_agent_custom_args is string %} + - {{ cilium_agent_custom_args }} +{% else %} +{% for flag in cilium_agent_custom_args %} + - {{ flag }} +{% endfor %} {% endif %} command: - cilium-agent @@ -57,6 +65,9 @@ spec: - name: KUBERNETES_SERVICE_PORT value: "{{ kube_apiserver_global_endpoint | urlsplit('port') }}" {% endif %} +{% for env_var in cilium_agent_extra_env_vars %} + - {{ env_var | to_nice_yaml(indent=2) | indent(10) }} +{% endfor %} image: "{{cilium_image_repo}}:{{cilium_image_tag}}" imagePullPolicy: {{ k8s_image_pull_policy }} resources: @@ -70,11 +81,29 @@ spec: postStart: exec: command: - - /cni-install.sh + - "/cni-install.sh" +{% if cilium_version | regex_replace('v') is version('1.10', '>=') %} + - "--cni-exclusive={{ cilium_cni_exclusive }}" +{% endif %} +{% if cilium_version | regex_replace('v') is version('1.12', '>=') %} + - "--log-file={{ cilium_cni_log_file }}" +{% endif %} preStop: exec: command: - /cni-uninstall.sh + startupProbe: + httpGet: + host: '127.0.0.1' + path: /healthz + port: 9876 + scheme: HTTP + httpHeaders: + - name: "brief" + value: "true" + failureThreshold: 105 + periodSeconds: 2 + successThreshold: 1 livenessProbe: httpGet: host: '127.0.0.1' @@ -85,13 +114,23 @@ spec: - name: "brief" value: "true" failureThreshold: 10 - # The initial delay for the liveness probe is intentionally large to - # avoid an endless kill & restart cycle if in the event that the initial - # bootstrapping takes longer than expected. - initialDelaySeconds: 120 periodSeconds: 30 successThreshold: 1 timeoutSeconds: 5 + readinessProbe: + httpGet: + host: 127.0.0.1 + path: /healthz + port: 9876 + scheme: HTTP + httpHeaders: + - name: "brief" + value: "true" + initialDelaySeconds: 5 + periodSeconds: 30 + successThreshold: 1 + failureThreshold: 3 + timeoutSeconds: 5 name: cilium-agent {% if cilium_enable_prometheus or cilium_enable_hubble_metrics %} ports: @@ -108,35 +147,21 @@ spec: name: hubble-metrics protocol: TCP {% endif %} - readinessProbe: - httpGet: - host: '127.0.0.1' - path: /healthz - port: 9876 - scheme: HTTP - httpHeaders: - - name: "brief" - value: "true" - failureThreshold: 3 - initialDelaySeconds: 5 - periodSeconds: 30 - successThreshold: 1 - timeoutSeconds: 5 securityContext: - capabilities: - add: - - NET_ADMIN - - SYS_MODULE privileged: true volumeMounts: - mountPath: /sys/fs/bpf name: bpf-maps + mountPropagation: Bidirectional - mountPath: /var/run/cilium name: cilium-run - mountPath: /host/opt/cni/bin name: cni-path - mountPath: /host/etc/cni/net.d name: etc-cni-netd +# pkg/workloads was depreca, removed in 1.7 +# https://github.com/cilium/cilium/pull/9447 +{% if cilium_version | regex_replace('v') is version('1.7', '<') %} {% if container_manager == 'docker' %} - mountPath: /var/run/docker.sock name: docker-socket @@ -146,6 +171,7 @@ spec: mountPath: {{ cri_socket }} readOnly: true {% endif %} +{% endif %} {% if cilium_identity_allocation_mode == "kvstore" %} - mountPath: /var/lib/etcd-config name: etcd-config-path @@ -166,7 +192,7 @@ spec: readOnly: true - mountPath: /run/xtables.lock name: xtables-lock -{% if cilium_ipsec_enabled %} +{% if cilium_encryption_enabled and cilium_encryption_type == "ipsec" %} - mountPath: /etc/ipsec name: cilium-ipsec-secrets readOnly: true @@ -176,9 +202,14 @@ spec: name: hubble-tls readOnly: true {% endif %} +{% for volume_mount in cilium_agent_extra_volume_mounts %} + - {{ volume_mount | to_nice_yaml(indent=2) | indent(10) }} +{% endfor %} +# In managed etcd mode, Cilium must be able to resolve the DNS name of the etcd service +{% if cilium_identity_allocation_mode == "kvstore" %} dnsPolicy: ClusterFirstWithHostNet +{% endif %} hostNetwork: true - hostPID: false initContainers: - command: - /init-container.sh @@ -189,18 +220,30 @@ spec: key: clean-cilium-state name: cilium-config optional: true - - name: CLEAN_CILIUM_BPF_STATE + # CLEAN_CILIUM_BPF_STATE is deprecated in 1.6. + # https://github.com/cilium/cilium/pull/7478 + - name: "{{ cilium_version | regex_replace('v') is version('1.6', '<')| ternary('CLEAN_CILIUM_BPF_STATE','CILIUM_BPF_STATE') }}" valueFrom: configMapKeyRef: key: clean-cilium-bpf-state name: cilium-config optional: true +# Removed in 1.11 and up. +# https://github.com/cilium/cilium/commit/f7a3f59fd74983c600bfce9cac364b76d20849d9 +{% if cilium_version | regex_replace('v') is version('1.11', '<') %} - name: CILIUM_WAIT_BPF_MOUNT valueFrom: configMapKeyRef: key: wait-bpf-mount name: cilium-config optional: true +{% endif %} +{% if cilium_kube_proxy_replacement == 'strict' %} + - name: KUBERNETES_SERVICE_HOST + value: "{{ kube_apiserver_global_endpoint | urlsplit('hostname') }}" + - name: KUBERNETES_SERVICE_PORT + value: "{{ kube_apiserver_global_endpoint | urlsplit('port') }}" +{% endif %} {% if cilium_version | regex_replace('v') is version('1.9', '<') %} image: "{{cilium_init_image_repo}}:{{cilium_init_image_tag}}" {% else %} @@ -209,13 +252,16 @@ spec: imagePullPolicy: {{ k8s_image_pull_policy }} name: clean-cilium-state securityContext: - capabilities: - add: - - NET_ADMIN privileged: true volumeMounts: - mountPath: /sys/fs/bpf name: bpf-maps +{% if cilium_version | regex_replace('v') is version('1.11', '>=') %} + # Required to mount cgroup filesystem from the host to cilium agent pod + - name: cilium-cgroup + mountPath: /run/cilium/cgroupv2 + mountPropagation: HostToContainer +{% endif %} - mountPath: /var/run/cilium name: cilium-run resources: @@ -227,6 +273,11 @@ spec: serviceAccount: cilium serviceAccountName: cilium terminationGracePeriodSeconds: 1 + hostNetwork: true +# In managed etcd mode, Cilium must be able to resolve the DNS name of the etcd service +{% if cilium_identity_allocation_mode == "kvstore" %} + dnsPolicy: ClusterFirstWithHostNet +{% endif %} tolerations: - operator: Exists volumes: @@ -240,6 +291,9 @@ spec: path: /sys/fs/bpf type: DirectoryOrCreate name: bpf-maps +# pkg/workloads was deprecated in 1.6, removed in 1.7 +# https://github.com/cilium/cilium/pull/9447 +{% if cilium_version | regex_replace('v') is version('1.7', '<') %} {% if container_manager == 'docker' %} # To read docker events from the node - hostPath: @@ -252,6 +306,19 @@ spec: path: {{ cri_socket }} type: Socket name: {{ container_manager }}-socket +{% endif %} +{% endif %} +{% if cilium_version | regex_replace('v') is version('1.11', '>=') %} + # To mount cgroup2 filesystem on the host + - name: hostproc + hostPath: + path: /proc + type: Directory + # To keep state between restarts / upgrades for cgroup2 filesystem + - name: cilium-cgroup + hostPath: + path: /run/cilium/cgroupv2 + type: DirectoryOrCreate {% endif %} # To install cilium cni plugin in the host - hostPath: @@ -275,7 +342,8 @@ spec: {% if cilium_identity_allocation_mode == "kvstore" %} # To read the etcd config stored in config maps - configMap: - defaultMode: 420 + # note: the leading zero means this number is in octal representation: do not remove it + defaultMode: 0400 items: - key: etcd-config path: etcd.config @@ -289,14 +357,15 @@ spec: # To read the clustermesh configuration - name: clustermesh-secrets secret: - defaultMode: 420 + # note: the leading zero means this number is in octal representation: do not remove it + defaultMode: 0400 optional: true secretName: cilium-clustermesh # To read the configuration from the config map - configMap: name: cilium-config name: cilium-config-path -{% if cilium_ipsec_enabled %} +{% if cilium_encryption_enabled and cilium_encryption_type == "ipsec" %} - name: cilium-ipsec-secrets secret: secretName: cilium-ipsec-keys @@ -304,6 +373,8 @@ spec: {% if cilium_hubble_install %} - name: hubble-tls projected: + # note: the leading zero means this number is in octal representation: do not remove it + defaultMode: 0400 sources: - secret: name: hubble-server-certs diff --git a/roles/network_plugin/cilium/templates/cilium/sa.yml.j2 b/roles/network_plugin/cilium/templates/cilium/sa.yml.j2 new file mode 100644 index 000000000..c03ac59b4 --- /dev/null +++ b/roles/network_plugin/cilium/templates/cilium/sa.yml.j2 @@ -0,0 +1,6 @@ +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: cilium + namespace: kube-system diff --git a/roles/network_plugin/cilium/templates/cilium-secret.yml.j2 b/roles/network_plugin/cilium/templates/cilium/secret.yml.j2 similarity index 100% rename from roles/network_plugin/cilium/templates/cilium-secret.yml.j2 rename to roles/network_plugin/cilium/templates/cilium/secret.yml.j2 diff --git a/roles/network_plugin/cilium/templates/hubble-config.yml.j2 b/roles/network_plugin/cilium/templates/hubble/config.yml.j2 similarity index 100% rename from roles/network_plugin/cilium/templates/hubble-config.yml.j2 rename to roles/network_plugin/cilium/templates/hubble/config.yml.j2 diff --git a/roles/network_plugin/cilium/templates/hubble-cr.yml.j2 b/roles/network_plugin/cilium/templates/hubble/cr.yml.j2 similarity index 100% rename from roles/network_plugin/cilium/templates/hubble-cr.yml.j2 rename to roles/network_plugin/cilium/templates/hubble/cr.yml.j2 diff --git a/roles/network_plugin/cilium/templates/hubble-crb.yml.j2 b/roles/network_plugin/cilium/templates/hubble/crb.yml.j2 similarity index 100% rename from roles/network_plugin/cilium/templates/hubble-crb.yml.j2 rename to roles/network_plugin/cilium/templates/hubble/crb.yml.j2 diff --git a/roles/network_plugin/cilium/templates/hubble-cronjob.yml.j2 b/roles/network_plugin/cilium/templates/hubble/cronjob.yml.j2 similarity index 100% rename from roles/network_plugin/cilium/templates/hubble-cronjob.yml.j2 rename to roles/network_plugin/cilium/templates/hubble/cronjob.yml.j2 diff --git a/roles/network_plugin/cilium/templates/hubble-deploy.yml.j2 b/roles/network_plugin/cilium/templates/hubble/deploy.yml.j2 similarity index 100% rename from roles/network_plugin/cilium/templates/hubble-deploy.yml.j2 rename to roles/network_plugin/cilium/templates/hubble/deploy.yml.j2 diff --git a/roles/network_plugin/cilium/templates/hubble-job.yml.j2 b/roles/network_plugin/cilium/templates/hubble/job.yml.j2 similarity index 100% rename from roles/network_plugin/cilium/templates/hubble-job.yml.j2 rename to roles/network_plugin/cilium/templates/hubble/job.yml.j2 diff --git a/roles/network_plugin/cilium/templates/hubble-sa.yml.j2 b/roles/network_plugin/cilium/templates/hubble/sa.yml.j2 similarity index 100% rename from roles/network_plugin/cilium/templates/hubble-sa.yml.j2 rename to roles/network_plugin/cilium/templates/hubble/sa.yml.j2 diff --git a/roles/network_plugin/cilium/templates/hubble-service.yml.j2 b/roles/network_plugin/cilium/templates/hubble/service.yml.j2 similarity index 100% rename from roles/network_plugin/cilium/templates/hubble-service.yml.j2 rename to roles/network_plugin/cilium/templates/hubble/service.yml.j2