diff --git a/docs/hardening.md b/docs/hardening.md index df757df32..7a5cddb85 100644 --- a/docs/hardening.md +++ b/docs/hardening.md @@ -85,6 +85,13 @@ kubelet_streaming_connection_idle_timeout: "5m" kubelet_make_iptables_util_chains: true kubelet_feature_gates: ["RotateKubeletServerCertificate=true","SeccompDefault=true"] kubelet_seccomp_default: true +kubelet_systemd_hardening: true +# In case you have multiple interfaces in your +# control plane nodes and you want to specify the right +# IP addresses, kubelet_secure_addresses allows you +# to specify the IP from which the kubelet +# will receive the packets. +kubelet_secure_addresses: "192.168.10.110 192.168.10.111 192.168.10.112" # additional configurations kube_owner: root @@ -103,6 +110,8 @@ Let's take a deep look to the resultant **kubernetes** configuration: * The `encryption-provider-config` provide encryption at rest. This means that the `kube-apiserver` encrypt data that is going to be stored before they reach `etcd`. So the data is completely unreadable from `etcd` (in case an attacker is able to exploit this). * The `rotateCertificates` in `KubeletConfiguration` is set to `true` along with `serverTLSBootstrap`. This could be used in alternative to `tlsCertFile` and `tlsPrivateKeyFile` parameters. Additionally it automatically generates certificates by itself, but you need to manually approve them or at least using an operator to do this (for more details, please take a look here: ). * If you are installing **kubernetes** in an AppArmor-based OS (eg. Debian/Ubuntu) you can enable the `AppArmor` feature gate uncommenting the lines with the comment `# AppArmor-based OS` on top. +* The `kubelet_systemd_hardening`, both with `kubelet_secure_addresses` setup a minimal firewall on the system. To better understand how these variables work, here's an explanatory image: + ![kubelet hardening](img/kubelet-hardening.png) Once you have the file properly filled, you can run the **Ansible** command to start the installation: diff --git a/docs/img/kubelet-hardening.png b/docs/img/kubelet-hardening.png new file mode 100644 index 000000000..5546a8ba9 Binary files /dev/null and b/docs/img/kubelet-hardening.png differ diff --git a/docs/vars.md b/docs/vars.md index 7c86ebe1c..1fde812d3 100644 --- a/docs/vars.md +++ b/docs/vars.md @@ -175,25 +175,46 @@ Stack](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/dns-stack.m * *docker_options* - Commonly used to set ``--insecure-registry=myregistry.mydomain:5000`` + * *docker_plugins* - This list can be used to define [Docker plugins](https://docs.docker.com/engine/extend/) to install. + * *containerd_default_runtime* - If defined, changes the default Containerd runtime used by the Kubernetes CRI plugin. + * *containerd_additional_runtimes* - Sets the additional Containerd runtimes used by the Kubernetes CRI plugin. [Default config](https://github.com/kubernetes-sigs/kubespray/blob/master/roles/container-engine/containerd/defaults/main.yml) can be overriden in inventory vars. + * *http_proxy/https_proxy/no_proxy/no_proxy_exclude_workers/additional_no_proxy* - Proxy variables for deploying behind a proxy. Note that no_proxy defaults to all internal cluster IPs and hostnames that correspond to each node. + * *kubelet_cgroup_driver* - Allows manual override of the cgroup-driver option for Kubelet. By default autodetection is used to match container manager configuration. `systemd` is the preferred driver for `containerd` though it can have issues with `cgroups v1` and `kata-containers` in which case you may want to change to `cgroupfs`. + * *kubelet_rotate_certificates* - Auto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. + * *kubelet_rotate_server_certificates* - Auto rotate the kubelet server certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. **Note** that server certificates are **not** approved automatically. Approve them manually (`kubectl get csr`, `kubectl certificate approve`) or implement custom approving controller like [kubelet-rubber-stamp](https://github.com/kontena/kubelet-rubber-stamp). + * *kubelet_streaming_connection_idle_timeout* - Set the maximum time a streaming connection can be idle before the connection is automatically closed. + * *kubelet_make_iptables_util_chains* - If `true`, causes the kubelet ensures a set of `iptables` rules are present on host. + +* *kubelet_systemd_hardening* - If `true`, provides kubelet systemd service with security features for isolation. + + **N.B.** To enable this feature, ensure you are using the **`cgroup v2`** on your system. Check it out with command: `sudo ls -l /sys/fs/cgroup/*.slice`. If directory does not exists, enable this with the following guide: [enable cgroup v2](https://rootlesscontaine.rs/getting-started/common/cgroup2/#enabling-cgroup-v2). + + * *kubelet_secure_addresses* - By default *kubelet_systemd_hardening* set the **control plane** `ansible_host` IPs as the `kubelet_secure_addresses`. In case you have multiple interfaces in your control plane nodes and the `kube-apiserver` is not bound to the default interface, you can override them with this variable. + Example: + + The **control plane** node may have 2 interfaces with the following IP addresses: `eth0:10.0.0.110`, `eth1:192.168.1.110`. + + By default the `kubelet_secure_addresses` is set with the `10.0.0.110` the ansible control host uses `eth0` to connect to the machine. In case you want to use `eth1` as the outgoing interface on which `kube-apiserver` connects to the `kubelet`s, you should override the variable in this way: `kubelet_secure_addresses: "192.168.1.110"`. + * *node_labels* - Labels applied to nodes via kubelet --node-labels parameter. For example, labels can be set in the inventory as variables or more widely in group_vars. *node_labels* can only be defined as a dict: diff --git a/roles/kubernetes/node/defaults/main.yml b/roles/kubernetes/node/defaults/main.yml index f1657c0e6..6cf237ea1 100644 --- a/roles/kubernetes/node/defaults/main.yml +++ b/roles/kubernetes/node/defaults/main.yml @@ -22,6 +22,12 @@ kubelet_kubelet_cgroups_cgroupfs: "/system.slice/kubelet.service" ### fail with swap on (default true) kubelet_fail_swap_on: true +# Set systemd service hardening features +kubelet_systemd_hardening: false + +# List of secure IPs for kubelet +kubelet_secure_addresses: "{{ groups['kube_control_plane'] | map('extract', hostvars, ['ansible_host']) | join(' ') }}" + # Reserve this space for kube resources kube_memory_reserved: 256Mi kube_cpu_reserved: 100m diff --git a/roles/kubernetes/node/templates/kubelet.service.j2 b/roles/kubernetes/node/templates/kubelet.service.j2 index 38360c8df..feb837424 100644 --- a/roles/kubernetes/node/templates/kubelet.service.j2 +++ b/roles/kubernetes/node/templates/kubelet.service.j2 @@ -24,6 +24,11 @@ ExecStart={{ bin_dir }}/kubelet \ $KUBELET_CLOUDPROVIDER Restart=always RestartSec=10s +{% if kubelet_systemd_hardening %} +# Hardening setup +IPAddressDeny=any +IPAddressAllow={{ kubelet_secure_addresses }} +{% endif %} [Install] WantedBy=multi-user.target