Drop linux capabilities and rework users/groups

* Drop linux capabilities for unprivileged containerized
  worlkoads Kargo configures for deployments.
* Configure required securityContext/user/group/groups for kube
  components' static manifests, etcd, calico-rr and k8s apps,
  like dnsmasq daemonset.
* Rework cloud-init (etcd) users creation for CoreOS.
* Fix nologin paths, adjust defaults for addusers role and ensure
  supplementary groups membership added for users.
* Add netplug user for network plugins (yet unused by privileged
  networking containers though).
* Grant the kube and netplug users read access for etcd certs via
  the etcd certs group.
* Grant group read access to kube certs via the kube cert group.
* Remove priveleged mode for calico-rr and run it under its uid/gid
  and supplementary etcd_cert group.
* Adjust docs.
* Align cpu/memory limits and dropped caps with added rkt support
  for control plane.

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
This commit is contained in:
Bogdan Dobrelya 2016-12-28 14:58:37 +01:00
parent 979b01a145
commit 48e77cd8bb
48 changed files with 413 additions and 81 deletions

31
docs/security.md Normal file
View file

@ -0,0 +1,31 @@
Users and groups
================
There are following users and groups defined by the addusers role:
* Kube user, group from the ``kubelet_user`` and ``kubelet_group`` vars.
* Etcd user, group from the ``etcd_user`` and ``etcd_group`` vars.
* Network plugin user, group from the ``netplug_user`` and ``netplug_group`` vars.
There are additional certificate access groups for kube and etcd users defined.
For example, kubelet and network plugins require read access to the
etcd certs and keys. This is defined via the corresponding ``etcd_cert_group``
var. Members of that group (defaults to `kube` and `netplug` users) will read
etcd secret keys and certs. Same applies to the ``kube_cert_group``
(defaults to `kube` user) members. You may want to share kube certs via that
group with bastion proxies or the like.
Linux capabilites
=================
Kargo allows to control dropped Linux capabilities for unprivileged docker
containers it configures for deployments. For examle, etcd or some networking
related systemd units or k8s workloads, like kubedns, dnsmasq or netchecker apps.
Dropped capabilites are represented by the ``apps_drop_cap``, ``dnsmasq_drop_cap``,
``etcd_drop_cap``, ``calico_drop_cap`` vars.
Be carefull changing defaults - different kube components and k8s apps might
expect specific capabilities to be present and can only run as root! Also note
that kublet, kube-proxy and network plugins require privileged mode and ignore
dropped capabilities.

View file

@ -36,10 +36,19 @@ retry_stagger: 5
# Directory where python binary is installed # Directory where python binary is installed
# ansible_python_interpreter: "/opt/bin/python" # ansible_python_interpreter: "/opt/bin/python"
# This is the group that the cert creation scripts chgrp the # This is the users/groups to own files and groups that the cert creation
# cert files to. Not really changable... # scripts chgrp the cert files to
kubelet_user: kube
kubelet_group: kube
kube_cert_group: kube-cert kube_cert_group: kube-cert
netplug_user: netplug
netplug_group: netplug
etcd_user: etcd
etcd_group: etcd
etcd_cert_group: etcd-cert
# Cluster Loglevel configuration # Cluster Loglevel configuration
kube_log_level: 2 kube_log_level: 2

View file

@ -1,24 +1,40 @@
--- ---
addusers: addusers:
etcd: etcd:
name: etcd name: "{{ etcd_user }}"
comment: "Etcd user" comment: "Etcd user"
createhome: yes createhome: >-
{% if ansible_os_family in ["CoreOS", "Container Linux by CoreOS"] %}no{% else %}yes{% endif %}
home: "/var/lib/etcd" home: "/var/lib/etcd"
system: yes system: yes
shell: /bin/nologin shell: /usr/sbin/nologin
group: "{{ etcd_group }}"
groups: "{{ etcd_cert_group }}"
type: >-
{% if ansible_os_family in ["CoreOS", "Container Linux by CoreOS"] %}cloud-init{% endif %}
kube: kube:
name: kube name: "{{ kubelet_user }}"
comment: "Kubernetes user" comment: "Kubernetes user"
shell: /sbin/nologin shell: /usr/sbin/nologin
system: yes system: yes
group: "{{ kube_cert_group }}" group: "{{ kubelet_group }}"
groups: "{{ etcd_cert_group }},{{ kube_cert_group }}"
createhome: no createhome: no
netplug:
name: "{{ netplug_user }}"
comment: "Network plugin user"
createhome: no
system: yes
shell: /usr/sbin/nologin
group: "{{ netplug_group }}"
groups: "{{ etcd_cert_group }}"
adduser: adduser:
name: "{{ user.name }}" name: "{{ user.name }}"
group: "{{ user.name|default(None) }}" group: "{{ user.name|default(None) }}"
groups: "{{ user.groups|default(None) }}"
comment: "{{ user.comment|default(None) }}" comment: "{{ user.comment|default(None) }}"
shell: "{{ user.shell|default(None) }}" shell: "{{ user.shell|default(None) }}"
system: "{{ user.system|default(None) }}" system: "{{ user.system|default(None) }}"
createhome: "{{ user.createhome|default(None) }}" createhome: "{{ user.createhome|default(None) }}"
type: "{{ user.type|default(None) }}"

View file

@ -0,0 +1,4 @@
---
- name: User | update users for cloud-init
command: /usr/bin/coreos-cloudinit --from-file /etc/{{ user.name }}_user_cloud_init_conf
when: ansible_os_family in ["CoreOS", "Container Linux by CoreOS"]

View file

@ -1,13 +1,35 @@
--- ---
- name: User | Create Certificate Access Groups
group: name={{ item }} system=yes
with_items: "{{ user.groups.split(',') }}"
- name: User | Create User Group - name: User | Create User Group
group: name={{user.group|default(user.name)}} system={{user.system|default(omit)}} group: name={{user.group|default(user.name)}} system={{user.system|default(omit)}}
- name: User | Create cloud-init user
template:
dest: /etc/{{ user.name }}_user_cloud_init_conf
src: users.j2
owner: root
mode: 0640
notify: User | update users for cloud-init
when: "{{ user.type|default('standard') == 'cloud-init' }}"
- meta: flush_handlers
- name: User | Hack groups for existing cloud-init users CoreOS
command: /usr/sbin/usermod -aG {{ item }} {{ user.name }}
with_items: "{{ user.groups.split(',') }}"
when: "{{ ansible_os_family in ['CoreOS', 'Container Linux by CoreOS'] and user.type|default('standard') == 'cloud-init' }}"
- name: User | Create User - name: User | Create User
user: user:
comment: "{{user.comment|default(omit)}}" comment: "{{user.comment|default(omit)}}"
createhome: "{{user.create_home|default(omit)}}" createhome: "{{user.createhome|default(omit)}}"
group: "{{user.group|default(user.name)}}" group: "{{user.group|default(user.name)}}"
groups: "{{user.groups|default(omit)}}"
home: "{{user.home|default(omit)}}" home: "{{user.home|default(omit)}}"
shell: "{{user.shell|default(omit)}}" shell: "{{user.shell|default(omit)}}"
name: "{{user.name}}" name: "{{user.name}}"
system: "{{user.system|default(omit)}}" system: "{{user.system|default(omit)}}"
when: "{{ user.type|default('standard') != 'cloud-init' }}"

View file

@ -0,0 +1,15 @@
#cloud-config
users:
- name: {{ user.name }}
gecos: {{ user.comment }}
system: {{ user.system|bool }}
no-log-init: {{ user.system|bool }}
primary-group: {{ user.group }}
no-create-home: {{ not user.createhome|bool }}
homedir: {{ user.home }}
shell: {{ user.shell }}
groups: |
{% for g in user.groups.split(',') %}
- {{ g }}
{% endfor %}
#

View file

@ -1,8 +0,0 @@
---
addusers:
- name: kube
comment: "Kubernetes user"
shell: /sbin/nologin
system: yes
group: "{{ kube_cert_group }}"
createhome: no

View file

@ -1,15 +0,0 @@
---
addusers:
- name: etcd
comment: "Etcd user"
createhome: yes
home: "/var/lib/etcd"
system: yes
shell: /bin/nologin
- name: kube
comment: "Kubernetes user"
shell: /sbin/nologin
system: yes
group: "{{ kube_cert_group }}"
createhome: no

View file

@ -1,15 +0,0 @@
---
addusers:
- name: etcd
comment: "Etcd user"
createhome: yes
home: "/var/lib/etcd"
system: yes
shell: /bin/nologin
- name: kube
comment: "Kubernetes user"
shell: /sbin/nologin
system: yes
group: "{{ kube_cert_group }}"
createhome: no

View file

@ -26,3 +26,16 @@ dns_cpu_limit: 100m
dns_memory_limit: 170Mi dns_memory_limit: 170Mi
dns_cpu_requests: 70m dns_cpu_requests: 70m
dns_memory_requests: 70Mi dns_memory_requests: 70Mi
# Linux capabilities to be dropped for dnsmasq k8s app ran container engines
dnsmasq_drop_cap:
- chown
- dac_override
- fowner
- fsetid
- kill
- setpcap
- sys_chroot
- mknod
- audit_write
- setfcap

View file

@ -26,6 +26,10 @@ spec:
capabilities: capabilities:
add: add:
- NET_ADMIN - NET_ADMIN
drop:
{% for c in dnsmasq_drop_cap %}
- {{ c.upper() }}
{% endfor %}
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
resources: resources:
limits: limits:

View file

@ -3,10 +3,26 @@ etcd_bin_dir: "{{ local_release_dir }}/etcd/etcd-{{ etcd_version }}-linux-amd64/
etcd_config_dir: /etc/ssl/etcd etcd_config_dir: /etc/ssl/etcd
etcd_cert_dir: "{{ etcd_config_dir }}/ssl" etcd_cert_dir: "{{ etcd_config_dir }}/ssl"
etcd_cert_group: root
etcd_script_dir: "{{ bin_dir }}/etcd-scripts" etcd_script_dir: "{{ bin_dir }}/etcd-scripts"
# Linux capabilities to be dropped for container engines
etcd_drop_cap:
- chown
- dac_override
- fowner
- fsetid
- kill
- setgid
- setuid
- setpcap
- net_bind_service
- net_raw
- sys_chroot
- mknod
- audit_write
- setfcap
# Limits # Limits
etcd_memory_limit: 512M etcd_memory_limit: 512M
etcd_cpu_limit: 300m etcd_cpu_limit: 300m

View file

@ -94,5 +94,8 @@ if [ -n "$HOSTS" ]; then
done done
fi fi
# Grant the group read access
chmod g+r *.pem
# Install certs # Install certs
mv *.pem ${SSLDIR}/ mv *.pem ${SSLDIR}/

View file

@ -2,7 +2,7 @@
dependencies: dependencies:
- role: adduser - role: adduser
user: "{{ addusers.etcd }}" user: "{{ addusers.etcd }}"
when: not ansible_os_family in ['CoreOS', 'Container Linux by CoreOS'] tags: bootstrap-os
- role: download - role: download
file: "{{ downloads.etcd }}" file: "{{ downloads.etcd }}"
tags: download tags: download

View file

@ -4,14 +4,15 @@
path={{ etcd_cert_dir }} path={{ etcd_cert_dir }}
group={{ etcd_cert_group }} group={{ etcd_cert_group }}
state=directory state=directory
owner=root mode=0750
owner={{ etcd_user }}
recurse=yes recurse=yes
- name: "Gen_certs | create etcd script dir (on {{groups['etcd'][0]}})" - name: "Gen_certs | create etcd script dir (on {{groups['etcd'][0]}})"
file: file:
path: "{{ etcd_script_dir }}" path: "{{ etcd_script_dir }}"
state: directory state: directory
owner: root owner: "{{ etcd_user }}"
run_once: yes run_once: yes
delegate_to: "{{groups['etcd'][0]}}" delegate_to: "{{groups['etcd'][0]}}"
@ -20,7 +21,8 @@
path={{ etcd_cert_dir }} path={{ etcd_cert_dir }}
group={{ etcd_cert_group }} group={{ etcd_cert_group }}
state=directory state=directory
owner=root mode=0750
owner={{ etcd_user }}
recurse=yes recurse=yes
run_once: yes run_once: yes
delegate_to: "{{groups['etcd'][0]}}" delegate_to: "{{groups['etcd'][0]}}"
@ -124,12 +126,12 @@
path={{ etcd_cert_dir }} path={{ etcd_cert_dir }}
group={{ etcd_cert_group }} group={{ etcd_cert_group }}
state=directory state=directory
owner=kube owner={{ etcd_user }}
recurse=yes recurse=yes
tags: facts tags: facts
- name: Gen_certs | set permissions on keys - name: Gen_certs | set shared group permissions on keys
shell: chmod 0600 {{ etcd_cert_dir}}/*key.pem shell: chmod 0640 {{ etcd_cert_dir}}/*.pem
when: inventory_hostname in groups['etcd'] when: inventory_hostname in groups['etcd']
changed_when: false changed_when: false

View file

@ -1,6 +1,8 @@
--- ---
- include: pre_upgrade.yml - include: pre_upgrade.yml
tags: etcd-pre-upgrade tags: etcd-pre-upgrade
- include: set_facts.yml
tags: [bootstrap-os, facts]
- include: check_certs.yml - include: check_certs.yml
tags: [etcd-secrets, facts] tags: [etcd-secrets, facts]
- include: gen_certs.yml - include: gen_certs.yml

View file

@ -1,3 +1,4 @@
---
- name: "Pre-upgrade | check for etcd-proxy unit file" - name: "Pre-upgrade | check for etcd-proxy unit file"
stat: stat:
path: /etc/systemd/system/etcd-proxy.service path: /etc/systemd/system/etcd-proxy.service
@ -49,3 +50,7 @@
awk -F"[: =]" '{print "{{ bin_dir }}/etcdctl --peers={{ etcd_access_addresses | regex_replace('https','http') }} member update "$1" https:"$7":"$8}' | bash awk -F"[: =]" '{print "{{ bin_dir }}/etcdctl --peers={{ etcd_access_addresses | regex_replace('https','http') }} member update "$1" https:"$7":"$8}' | bash
run_once: true run_once: true
when: 'etcd_member_list.rc == 0 and "http://" in etcd_member_list.stdout' when: 'etcd_member_list.rc == 0 and "http://" in etcd_member_list.stdout'
- name: "Pre-upgrade | share access to etcd certs for its users"
shell: chmod g+r {{ etcd_cert_dir }}/*.pem
failed_when: false

View file

@ -0,0 +1,17 @@
---
- name: Etcd | get etcd user ID
shell: /usr/bin/id -u {{ etcd_user }} || echo 0
register: etcd_uid
- name: Etcd | get etcd group ID
shell: /usr/bin/getent group {{ etcd_group }} | cut -d':' -f3 || echo 0
register: etcd_gid
- name: Etcd | get etcd cert group ID
shell: /usr/bin/getent group {{ etcd_cert_group }} | cut -d':' -f3 || echo 0
register: etcd_cert_gid
- set_fact:
etcd_user_id: "{{ etcd_uid.stdout }}"
etcd_group_id: "{{ etcd_gid.stdout }}"
etcd_cert_group_id: "{{ etcd_cert_gid.stdout }}"

View file

@ -14,8 +14,12 @@ ExecStart={{ docker_bin_dir }}/docker run --restart=on-failure:5 \
-v /etc/ssl/certs:/etc/ssl/certs:ro \ -v /etc/ssl/certs:/etc/ssl/certs:ro \
-v {{ etcd_cert_dir }}:{{ etcd_cert_dir }}:ro \ -v {{ etcd_cert_dir }}:{{ etcd_cert_dir }}:ro \
-v /var/lib/etcd:/var/lib/etcd:rw \ -v /var/lib/etcd:/var/lib/etcd:rw \
{% for c in etcd_drop_cap %}
--cap-drop={{ c }} \
{% endfor %}
--memory={{ etcd_memory_limit|regex_replace('Mi', 'M') }} --cpu-shares={{ etcd_cpu_limit|regex_replace('m', '') }} \ --memory={{ etcd_memory_limit|regex_replace('Mi', 'M') }} --cpu-shares={{ etcd_cpu_limit|regex_replace('m', '') }} \
--name={{ etcd_member_name | default("etcd") }} \ --name={{ etcd_member_name | default("etcd") }} \
-u {{ etcd_user_id }}:{{ etcd_group_id }} --group-add {{ etcd_cert_group_id }} \
{{ etcd_image_repo }}:{{ etcd_image_tag }} \ {{ etcd_image_repo }}:{{ etcd_image_tag }} \
{% if etcd_after_v3 %} {% if etcd_after_v3 %}
{{ etcd_container_bin_dir }}etcd {{ etcd_container_bin_dir }}etcd

View file

@ -8,6 +8,9 @@ Restart=on-failure
RestartSec=10s RestartSec=10s
TimeoutStartSec=0 TimeoutStartSec=0
LimitNOFILE=40000 LimitNOFILE=40000
User=root
Group={{ etcd_group_id }}
SupplementaryGroups={{ etcd_cert_group_id }}
ExecStart=/usr/bin/rkt run \ ExecStart=/usr/bin/rkt run \
--uuid-file-save=/var/run/etcd.uuid \ --uuid-file-save=/var/run/etcd.uuid \
@ -20,6 +23,11 @@ ExecStart=/usr/bin/rkt run \
--set-env-file=/etc/etcd.env \ --set-env-file=/etc/etcd.env \
--stage1-from-dir=stage1-fly.aci \ --stage1-from-dir=stage1-fly.aci \
{{ etcd_image_repo }}:{{ etcd_image_tag }} \ {{ etcd_image_repo }}:{{ etcd_image_tag }} \
{% for c in etcd_drop_cap %}
--caps-remove=CAP_{{ c.upper() }} \
{% endfor %}
--memory={{ etcd_memory_limit }} --cpu={{ etcd_cpu_limit }} \
--user={{ etcd_user_id }} --group={{ etcd_group_id }} \
--name={{ etcd_member_name | default("etcd") }} --name={{ etcd_member_name | default("etcd") }}
ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/run/etcd.uuid ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/run/etcd.uuid

View file

@ -51,3 +51,18 @@ netchecker_kubectl_memory_requests: 64M
etcd_cert_dir: "/etc/ssl/etcd/ssl" etcd_cert_dir: "/etc/ssl/etcd/ssl"
calico_cert_dir: "/etc/calico/certs" calico_cert_dir: "/etc/calico/certs"
canal_cert_dir: "/etc/canal/certs" canal_cert_dir: "/etc/canal/certs"
# Linux capabilities to be dropped for k8s apps ran by container engines
apps_drop_cap:
- chown
- dac_override
- fowner
- fsetid
- kill
- setgid
- setuid
- setpcap
- sys_chroot
- mknod
- audit_write
- setfcap

View file

@ -25,6 +25,12 @@ spec:
- name: calico-policy-controller - name: calico-policy-controller
image: {{ calico_policy_image_repo }}:{{ calico_policy_image_tag }} image: {{ calico_policy_image_repo }}:{{ calico_policy_image_tag }}
imagePullPolicy: {{ k8s_image_pull_policy }} imagePullPolicy: {{ k8s_image_pull_policy }}
securityContext:
capabilities:
drop:
{% for c in apps_drop_cap %}
- {{ c.upper() }}
{% endfor %}
resources: resources:
limits: limits:
cpu: {{ calico_policy_controller_cpu_limit }} cpu: {{ calico_policy_controller_cpu_limit }}

View file

@ -23,6 +23,12 @@ spec:
- name: REPORT_INTERVAL - name: REPORT_INTERVAL
value: '{{ agent_report_interval }}' value: '{{ agent_report_interval }}'
imagePullPolicy: {{ k8s_image_pull_policy }} imagePullPolicy: {{ k8s_image_pull_policy }}
securityContext:
capabilities:
drop:
{% for c in apps_drop_cap %}
- {{ c.upper() }}
{% endfor %}
resources: resources:
limits: limits:
cpu: {{ netchecker_agent_cpu_limit }} cpu: {{ netchecker_agent_cpu_limit }}

View file

@ -24,6 +24,12 @@ spec:
- name: REPORT_INTERVAL - name: REPORT_INTERVAL
value: '{{ agent_report_interval }}' value: '{{ agent_report_interval }}'
imagePullPolicy: {{ k8s_image_pull_policy }} imagePullPolicy: {{ k8s_image_pull_policy }}
securityContext:
capabilities:
drop:
{% for c in apps_drop_cap %}
- {{ c.upper() }}
{% endfor %}
resources: resources:
limits: limits:
cpu: {{ netchecker_agent_cpu_limit }} cpu: {{ netchecker_agent_cpu_limit }}

View file

@ -33,3 +33,9 @@ spec:
memory: {{ netchecker_kubectl_memory_requests }} memory: {{ netchecker_kubectl_memory_requests }}
args: args:
- proxy - proxy
securityContext:
capabilities:
drop:
{% for c in apps_drop_cap %}
- {{ c.upper() }}
{% endfor %}

View file

@ -13,6 +13,21 @@ kube_apiserver_node_port_range: "30000-32767"
etcd_config_dir: /etc/ssl/etcd etcd_config_dir: /etc/ssl/etcd
etcd_cert_dir: "{{ etcd_config_dir }}/ssl" etcd_cert_dir: "{{ etcd_config_dir }}/ssl"
# Linux capabilities to be dropped for k8s apps ran by container engines
apps_drop_cap:
- chown
- dac_override
- fowner
- fsetid
- kill
- setgid
- setuid
- setpcap
- sys_chroot
- mknod
- audit_write
- setfcap
# Limits for kube components # Limits for kube components
kube_controller_memory_limit: 512M kube_controller_memory_limit: 512M
kube_controller_cpu_limit: 250m kube_controller_cpu_limit: 250m

View file

@ -2,6 +2,9 @@
- include: pre-upgrade.yml - include: pre-upgrade.yml
tags: k8s-pre-upgrade tags: k8s-pre-upgrade
- include: set_facts.yml
tags: facts
- name: Copy kubectl from hyperkube container - name: Copy kubectl from hyperkube container
command: "{{ docker_bin_dir }}/docker run --rm -v {{ bin_dir }}:/systembindir {{ hyperkube_image_repo }}:{{ hyperkube_image_tag }} /bin/cp /hyperkube /systembindir/kubectl" command: "{{ docker_bin_dir }}/docker run --rm -v {{ bin_dir }}:/systembindir {{ hyperkube_image_repo }}:{{ hyperkube_image_tag }} /bin/cp /hyperkube /systembindir/kubectl"
register: kube_task_result register: kube_task_result

View file

@ -0,0 +1,22 @@
---
- name: Master | get kube user ID
shell: /usr/bin/id -u {{ kubelet_user }} || echo 0
register: kube_uid
- name: Master | get kube group ID
shell: /usr/bin/getent group {{ kubelet_group }} | cut -d':' -f3 || echo 0
register: kube_gid
- name: Master | get kube cert group ID
shell: /usr/bin/getent group {{ kube_cert_group }} | cut -d':' -f3 || echo 0
register: kube_cert_gid
- name: Master | get etcd cert group ID
shell: /usr/bin/getent group {{ etcd_cert_group }} | cut -d':' -f3 || echo 0
register: etcd_cert_gid
- set_fact:
kubelet_user_id: "{{ kube_uid.stdout }}"
kubelet_group_id: "{{ kube_gid.stdout }}"
kube_cert_group_id: "{{ kube_cert_gid.stdout }}"
etcd_cert_group_id: "{{ etcd_cert_gid.stdout }}"

View file

@ -12,6 +12,14 @@ spec:
- name: kube-apiserver - name: kube-apiserver
image: {{ hyperkube_image_repo }}:{{ hyperkube_image_tag }} image: {{ hyperkube_image_repo }}:{{ hyperkube_image_tag }}
imagePullPolicy: {{ k8s_image_pull_policy }} imagePullPolicy: {{ k8s_image_pull_policy }}
securityContext:
capabilities:
drop:
{% for c in apps_drop_cap %}
- {{ c.upper() }}
{% endfor %}
add:
- DAC_OVERRIDE
resources: resources:
limits: limits:
cpu: {{ kube_apiserver_cpu_limit }} cpu: {{ kube_apiserver_cpu_limit }}

View file

@ -11,6 +11,17 @@ spec:
- name: kube-controller-manager - name: kube-controller-manager
image: {{ hyperkube_image_repo }}:{{ hyperkube_image_tag }} image: {{ hyperkube_image_repo }}:{{ hyperkube_image_tag }}
imagePullPolicy: {{ k8s_image_pull_policy }} imagePullPolicy: {{ k8s_image_pull_policy }}
securityContext:
runAsUser: {{ kubelet_user_id }}
fsGroup: {{ kubelet_group_id }}
supplementalGroups:
- {{ kube_cert_group_id }}
- {{ etcd_cert_group_id }}
capabilities:
drop:
{% for c in apps_drop_cap %}
- {{ c.upper() }}
{% endfor %}
resources: resources:
limits: limits:
cpu: {{ kube_controller_cpu_limit }} cpu: {{ kube_controller_cpu_limit }}

View file

@ -11,6 +11,17 @@ spec:
- name: kube-scheduler - name: kube-scheduler
image: {{ hyperkube_image_repo }}:{{ hyperkube_image_tag }} image: {{ hyperkube_image_repo }}:{{ hyperkube_image_tag }}
imagePullPolicy: {{ k8s_image_pull_policy }} imagePullPolicy: {{ k8s_image_pull_policy }}
securityContext:
runAsUser: {{ kubelet_user_id }}
fsGroup: {{ kubelet_group_id }}
supplementalGroups:
- {{ kube_cert_group_id }}
- {{ etcd_cert_group_id }}
capabilities:
drop:
{% for c in apps_drop_cap %}
- {{ c.upper() }}
{% endfor %}
resources: resources:
limits: limits:
cpu: {{ kube_scheduler_cpu_limit }} cpu: {{ kube_scheduler_cpu_limit }}

View file

@ -29,3 +29,18 @@ nginx_image_repo: nginx
nginx_image_tag: 1.11.4-alpine nginx_image_tag: 1.11.4-alpine
etcd_config_dir: /etc/ssl/etcd etcd_config_dir: /etc/ssl/etcd
# Linux capabilities to be dropped for container engines
apps_drop_cap:
- chown
- dac_override
- fowner
- fsetid
- kill
- setgid
- setuid
- setpcap
- sys_chroot
- mknod
- audit_write
- setfcap

View file

@ -26,6 +26,6 @@
notify: restart kubelet notify: restart kubelet
- name: install | Install kubelet launch script - name: install | Install kubelet launch script
template: src=kubelet-container.j2 dest="{{ bin_dir }}/kubelet" owner=kube mode=0755 backup=yes template: src=kubelet-container.j2 dest="{{ bin_dir }}/kubelet" owner={{ kubelet_user }} mode=0755 backup=yes
notify: restart kubelet notify: restart kubelet
when: kubelet_deployment_type == "docker" when: kubelet_deployment_type == "docker"

View file

@ -4,6 +4,9 @@
{%- if inventory_hostname in groups['kube-master'] and inventory_hostname not in groups['kube-node'] -%}true{%- else -%}false{%- endif -%} {%- if inventory_hostname in groups['kube-master'] and inventory_hostname not in groups['kube-node'] -%}true{%- else -%}false{%- endif -%}
tags: facts tags: facts
- include: pre-upgrade.yml
tags: k8s-pre-upgrade
- include: install.yml - include: install.yml
tags: kubelet tags: kubelet

View file

@ -0,0 +1,4 @@
---
- name: "Pre-upgrade | share access to kube certs for its users"
shell: chmod g+r {{ kube_cert_dir }}/*.pem
failed_when: false

View file

@ -44,6 +44,7 @@ ExecStart=/usr/bin/rkt run \
--mount volume=var-log,target=/var/log \ --mount volume=var-log,target=/var/log \
--stage1-from-dir=stage1-fly.aci \ --stage1-from-dir=stage1-fly.aci \
{{ hyperkube_image_repo }}:{{ hyperkube_image_tag }} \ {{ hyperkube_image_repo }}:{{ hyperkube_image_tag }} \
--memory={{ kubelet_memory_limit }} --cpu={{ kubelet_cpu_limit }} \
--uuid-file-save=/var/run/kubelet.uuid \ --uuid-file-save=/var/run/kubelet.uuid \
--debug --exec=/kubelet -- \ --debug --exec=/kubelet -- \
$KUBE_LOGTOSTDERR \ $KUBE_LOGTOSTDERR \

View file

@ -2,4 +2,7 @@
dependencies: dependencies:
- role: adduser - role: adduser
user: "{{ addusers.kube }}" user: "{{ addusers.kube }}"
tags: kubelet tags: [bootstrap-os, kubelet]
- role: adduser
user: "{{ addusers.netplug }}"
tags: [bootstrap-os, network]

View file

@ -23,6 +23,12 @@
- include: set_facts.yml - include: set_facts.yml
tags: facts tags: facts
- include: set_resolv_facts.yml
tags: [bootstrap-os, resolvconf, facts]
- include: set_uid_facts.yml
tags: [bootstrap-os, facts]
- name: gather os specific variables - name: gather os specific variables
include_vars: "{{ item }}" include_vars: "{{ item }}"
with_first_found: with_first_found:
@ -42,7 +48,7 @@
file: file:
path: "{{ kube_config_dir }}" path: "{{ kube_config_dir }}"
state: directory state: directory
owner: kube owner: "{{ kubelet_user }}"
when: "{{ inventory_hostname in groups['k8s-cluster'] }}" when: "{{ inventory_hostname in groups['k8s-cluster'] }}"
tags: [kubelet, k8s-secrets, kube-controller-manager, kube-apiserver, bootstrap-os, apps, network, master, node] tags: [kubelet, k8s-secrets, kube-controller-manager, kube-apiserver, bootstrap-os, apps, network, master, node]
@ -50,7 +56,7 @@
file: file:
path: "{{ kube_script_dir }}" path: "{{ kube_script_dir }}"
state: directory state: directory
owner: kube owner: "{{ kubelet_user }}"
when: "{{ inventory_hostname in groups['k8s-cluster'] }}" when: "{{ inventory_hostname in groups['k8s-cluster'] }}"
tags: [k8s-secrets, bootstrap-os] tags: [k8s-secrets, bootstrap-os]
@ -58,7 +64,7 @@
file: file:
path: "{{ kube_manifest_dir }}" path: "{{ kube_manifest_dir }}"
state: directory state: directory
owner: kube owner: "{{ kubelet_user }}"
when: "{{ inventory_hostname in groups['k8s-cluster'] }}" when: "{{ inventory_hostname in groups['k8s-cluster'] }}"
tags: [kubelet, bootstrap-os, master, node] tags: [kubelet, bootstrap-os, master, node]
@ -80,7 +86,7 @@
file: file:
path: "{{ item }}" path: "{{ item }}"
state: directory state: directory
owner: kube owner: "{{ kubelet_user }}"
with_items: with_items:
- "/etc/cni/net.d" - "/etc/cni/net.d"
- "/opt/cni/bin" - "/opt/cni/bin"

View file

@ -51,6 +51,3 @@
etcd_container_bin_dir: "{% if etcd_after_v3 %}/usr/local/bin/{% else %}/{% endif %}" etcd_container_bin_dir: "{% if etcd_after_v3 %}/usr/local/bin/{% else %}/{% endif %}"
- set_fact: - set_fact:
peer_with_calico_rr: "{{ 'calico-rr' in groups and groups['calico-rr']|length > 0 }}" peer_with_calico_rr: "{{ 'calico-rr' in groups and groups['calico-rr']|length > 0 }}"
- include: set_resolv_facts.yml
tags: [bootstrap-os, resolvconf, facts]

View file

@ -0,0 +1,32 @@
---
- name: Preinstall | get kube user ID
shell: /usr/bin/id -u {{ kubelet_user }} || echo 0
register: kube_uid
- name: Preinstall | get kube group ID
shell: /usr/bin/id -g {{ kubelet_group }} || echo 0
register: kube_gid
- name: Preinstall | get kube cert group ID
shell: /usr/bin/id -g {{ kube_cert_group }} || echo 0
register: kube_cert_gid
- name: Preinstall | get etcd cert group ID
shell: /usr/bin/id -g {{ etcd_cert_group }} || echo 0
register: etcd_cert_gid
- name: Preinstall | get netplug user ID
shell: /usr/bin/id -u {{ netplug_user }} || echo 0
register: netplug_uid
- name: Preinstall | get netplug group ID
shell: /usr/bin/getent group {{ netplug_group }} | cut -d':' -f3 || echo 0
register: netplug_gid
- set_fact:
kubelet_user_id: "{{ kube_uid.stdout }}"
kubelet_group_id: "{{ kube_gid.stdout }}"
kube_cert_group_id: "{{ kube_cert_gid.stdout }}"
etcd_cert_group_id: "{{ etcd_cert_gid.stdout }}"
netplug_user_id: "{{ netplug_uid.stdout }}"
netplug_group_id: "{{ netplug_gid.stdout }}"

View file

@ -101,5 +101,8 @@ if [ -n "$HOSTS" ]; then
done done
fi fi
# Grant the group read access
chmod g+r *.pem
# Install certs # Install certs
mv *.pem ${SSLDIR}/ mv *.pem ${SSLDIR}/

View file

@ -110,11 +110,11 @@
file: file:
path={{ kube_cert_dir }} path={{ kube_cert_dir }}
group={{ kube_cert_group }} group={{ kube_cert_group }}
owner=kube owner={{ kubelet_user }}
recurse=yes recurse=yes
- name: Gen_certs | set permissions on keys - name: Gen_certs | set shared group permissions on keys
shell: chmod 0600 {{ kube_cert_dir}}/*key.pem shell: chmod 0640 {{ kube_cert_dir}}/*.pem
when: inventory_hostname in groups['kube-master'] when: inventory_hostname in groups['kube-master']
changed_when: false changed_when: false

View file

@ -9,6 +9,7 @@
path={{ kube_cert_dir }} path={{ kube_cert_dir }}
state=directory state=directory
mode=o-rwx mode=o-rwx
owner={{ kubelet_user }}
group={{ kube_cert_group }} group={{ kube_cert_group }}
- name: Make sure the tokens directory exits - name: Make sure the tokens directory exits
@ -16,14 +17,16 @@
path={{ kube_token_dir }} path={{ kube_token_dir }}
state=directory state=directory
mode=o-rwx mode=o-rwx
group={{ kube_cert_group }} owner={{ kubelet_user }}
group={{ kubelet_group }}
- name: Make sure the users directory exits - name: Make sure the users directory exits
file: file:
path={{ kube_users_dir }} path={{ kube_users_dir }}
state=directory state=directory
mode=o-rwx mode=o-rwx
group={{ kube_cert_group }} owner={{ kubelet_user }}
group={{ kubelet_group }}
- name: Populate users for basic auth in API - name: Populate users for basic auth in API
lineinfile: lineinfile:

View file

@ -20,6 +20,23 @@ global_as_num: "64512"
# defaults. The value should be a number, not a string. # defaults. The value should be a number, not a string.
# calico_mtu: 1500 # calico_mtu: 1500
# Linux capabilities to be dropped for container engines
calico_drop_cap:
- chown
- dac_override
- fowner
- fsetid
- kill
- setgid
- setuid
- setpcap
- net_bind_service
- net_raw
- sys_chroot
- mknod
- audit_write
- setfcap
# Limits for apps # Limits for apps
calico_node_memory_limit: 500M calico_node_memory_limit: 500M
calico_node_cpu_limit: 300m calico_node_cpu_limit: 300m

View file

@ -12,8 +12,8 @@
dest: "{{ calico_cert_dir }}" dest: "{{ calico_cert_dir }}"
state: directory state: directory
mode: 0750 mode: 0750
owner: root owner: "{{ netplug_user }}"
group: root group: "{{ netplug_group }}"
- name: Calico-rr | Link etcd certificates for calico-node - name: Calico-rr | Link etcd certificates for calico-node
file: file:
@ -31,8 +31,8 @@
path: /var/log/calico-rr path: /var/log/calico-rr
state: directory state: directory
mode: 0755 mode: 0755
owner: root owner: "{{ netplug_user }}"
group: root group: "{{ netplug_group }}"
- name: Calico-rr | Write calico-rr.env for systemd init file - name: Calico-rr | Write calico-rr.env for systemd init file
template: src=calico-rr.env.j2 dest=/etc/calico/calico-rr.env template: src=calico-rr.env.j2 dest=/etc/calico/calico-rr.env

View file

@ -6,7 +6,7 @@ Requires=docker.service
[Service] [Service]
EnvironmentFile=/etc/calico/calico-rr.env EnvironmentFile=/etc/calico/calico-rr.env
ExecStartPre=-{{ docker_bin_dir }}/docker rm -f calico-rr ExecStartPre=-{{ docker_bin_dir }}/docker rm -f calico-rr
ExecStart={{ docker_bin_dir }}/docker run --net=host --privileged \ ExecStart={{ docker_bin_dir }}/docker run --net=host \
--name=calico-rr \ --name=calico-rr \
-e IP=${IP} \ -e IP=${IP} \
-e IP6=${IP6} \ -e IP6=${IP6} \
@ -16,6 +16,10 @@ ExecStart={{ docker_bin_dir }}/docker run --net=host --privileged \
-e ETCD_KEY_FILE=${ETCD_KEY_FILE} \ -e ETCD_KEY_FILE=${ETCD_KEY_FILE} \
-v /var/log/calico-rr:/var/log/calico \ -v /var/log/calico-rr:/var/log/calico \
-v {{ calico_cert_dir }}:{{ calico_cert_dir }}:ro \ -v {{ calico_cert_dir }}:{{ calico_cert_dir }}:ro \
{% for c in calico_drop_cap %}
--cap-drop={{ c }} \
{% endfor %}
-u {{ netplug_user_id }}:{{ netplug_group_id }} --group-add {{ etcd_cert_group }} \
--memory={{ calico_rr_memory_limit|regex_replace('Mi', 'M') }} --cpu-shares={{ calico_rr_cpu_limit|regex_replace('m', '') }} \ --memory={{ calico_rr_memory_limit|regex_replace('Mi', 'M') }} --cpu-shares={{ calico_rr_cpu_limit|regex_replace('m', '') }} \
{{ calico_rr_image_repo }}:{{ calico_rr_image_tag }} {{ calico_rr_image_repo }}:{{ calico_rr_image_tag }}

View file

@ -9,15 +9,16 @@
template: template:
src: "cni-calico.conf.j2" src: "cni-calico.conf.j2"
dest: "/etc/cni/net.d/10-calico.conf" dest: "/etc/cni/net.d/10-calico.conf"
owner: kube owner: "{{ kubelet_user }}"
group: "{{ kubelet_group }}"
- name: Calico | Create calico certs directory - name: Calico | Create calico certs directory
file: file:
dest: "{{ calico_cert_dir }}" dest: "{{ calico_cert_dir }}"
state: directory state: directory
mode: 0750 mode: 0750
owner: root owner: "{{ netplug_user }}"
group: root group: "{{ netplug_group }}"
- name: Calico | Link etcd certificates for calico-node - name: Calico | Link etcd certificates for calico-node
file: file:

View file

@ -3,15 +3,16 @@
template: template:
src: "cni-canal.conf.j2" src: "cni-canal.conf.j2"
dest: "/etc/cni/net.d/10-canal.conf" dest: "/etc/cni/net.d/10-canal.conf"
owner: kube owner: "{{ kubelet_user }}"
group: "{{ kubelet_group }}"
- name: Canal | Create canal certs directory - name: Canal | Create canal certs directory
file: file:
dest: "{{ canal_cert_dir }}" dest: "{{ canal_cert_dir }}"
state: directory state: directory
mode: 0750 mode: 0750
owner: root owner: "{{ netplug_user }}"
group: root group: "{{ netplug_group }}"
- name: Canal | Link etcd certificates for canal-node - name: Canal | Link etcd certificates for canal-node
file: file: