c12s-kubespray/contrib/dind
JuanJo Ciarlante 4077934519 [jjo] add DIND support to contrib/ (#3468)
* [jjo] add DIND support to contrib/

- add contrib/dind with ansible playbook to
  create "node" containers, and setup them to mimic
  host nodes as much as possible (using Ubuntu images),
  see contrib/dind/README.md

- nodes' /etc/hosts editing via `blockinfile` and
  `lineinfile` need `unsafe_writes: yes` because /etc/hosts
  are mounted by docker, and thus can't be handled atomically
  (modify copy + rename)

* dind-host role: set node container hostname on creation

* add "Resulting deployment" section with some CLI outputs

* typo

* selectable node_distro: debian, ubuntu

* some fixes for node_distro: ubuntu

* cpu optimization: add early `pkill -STOP agetty`

* typo

* add centos dind support ;)

* add kubespray-dind.yaml, support fedora

- add kubespray-dind.yaml (former custom.yaml at README.md)
- rework README.md as per above
- use some YAML power to share distros' commonality
- add fedora support

* create unique /etc/machine-id and other updates

- create unique /etc/machine-id in each docker node,
  used as seed for e.g. weave mac addresses

- with above, now netchecker 100% passes WoHooOO!
  🎉 🎉 🎉

- updated README.md output from (1.12.1, verified
  netcheck)

* minor typos

* fix centos node creation, needs earlier udevadm removal to avoid flaky facts, also verified netcheck Ok \o/

* add Q&D test-distros.sh, back to manual /etc/machine-id hack

* run-test-distros.sh cosmetics and minor fixes

* run-test-distros.sh: $rc fix and minor formatting changes

* run-test-distros.sh output cosmetics
2018-10-15 09:44:02 +02:00
..
group_vars/all [jjo] add DIND support to contrib/ (#3468) 2018-10-15 09:44:02 +02:00
roles [jjo] add DIND support to contrib/ (#3468) 2018-10-15 09:44:02 +02:00
dind-cluster.yaml [jjo] add DIND support to contrib/ (#3468) 2018-10-15 09:44:02 +02:00
hosts [jjo] add DIND support to contrib/ (#3468) 2018-10-15 09:44:02 +02:00
kubespray-dind.yaml [jjo] add DIND support to contrib/ (#3468) 2018-10-15 09:44:02 +02:00
README.md [jjo] add DIND support to contrib/ (#3468) 2018-10-15 09:44:02 +02:00
requirements.txt [jjo] add DIND support to contrib/ (#3468) 2018-10-15 09:44:02 +02:00
run-test-distros.sh [jjo] add DIND support to contrib/ (#3468) 2018-10-15 09:44:02 +02:00

Kubespray DIND experimental setup

This ansible playbook creates local docker containers to serve as Kubernetes "nodes", which in turn will run "normal" Kubernetes docker containers, a mode usually called DIND (Docker-IN-Docker).

The playbook has two roles:

  • dind-host: creates the "nodes" as containers in localhost, with appropiate settings for DIND (privileged, volume mapping for dind storage, etc).
  • dind-cluster: customizes each node container to have required system packages installed, and some utils (swapoff, lsattr) symlinked to /bin/true to ease mimicking a real node.

This playbook has been test with Ubuntu 16.04 as host and ubuntu:16.04 as docker images (note that dind-cluster has specific customization for these images).

The playbook also creates a /tmp/kubespray.dind.inventory_builder.sh helper (wraps up running contrib/inventory_builder/inventory.py with node containers IPs and prefix).

Deploying

See below for a complete successful run:

  1. Create the node containers
# From the kubespray root dir
cd contrib/dind
pip install -r requirements.txt

ansible-playbook -i hosts dind-cluster.yaml

# Back to kubespray root
cd ../..

NOTE: if the playbook run fails with something like below error message, you may need to specifically set ansible_python_interpreter, see ./hosts file for an example expanded localhost entry.

failed: [localhost] (item=kube-node1) => {"changed": false, "item": "kube-node1", "msg": "Failed to import docker or docker-py - No module named requests.exceptions. Try `pip install docker` or `pip install docker-py` (Python 2.6)"}
  1. Customize kubespray-dind.yaml

Note that there's coupling between above created node containers and kubespray-dind.yaml settings, in particular regarding selected node_distro (as set in group_vars/all/all.yaml), and docker settings.

$EDITOR contrib/dind/kubespray-dind.yaml
  1. Prepare the inventory and run the playbook
INVENTORY_DIR=inventory/local-dind
mkdir -p ${INVENTORY_DIR}
rm -f ${INVENTORY_DIR}/hosts.ini
CONFIG_FILE=${INVENTORY_DIR}/hosts.ini /tmp/kubespray.dind.inventory_builder.sh

ansible-playbook --become -e ansible_ssh_user=debian -i ${INVENTORY_DIR}/hosts.ini cluster.yml --extra-vars @contrib/dind/kubespray-dind.yaml

NOTE: You could also test other distros without editing files by passing --extra-vars as per below commandline, replacing DISTRO by either debian, ubuntu, centos, fedora:

cd contrib/dind
ansible-playbook -i hosts dind-cluster.yaml --extra-vars node_distro=DISTRO

cd ../..
CONFIG_FILE=inventory/local-dind/hosts.ini /tmp/kubespray.dind.inventory_builder.sh
ansible-playbook --become -e ansible_ssh_user=DISTRO -i inventory/local-dind/hosts.ini cluster.yml --extra-vars @contrib/dind/kubespray-dind.yaml --extra-vars bootstrap_os=DISTRO

Resulting deployment

See below to get an idea on how a completed deployment looks like, from the host where you ran kubespray playbooks.

node_distro: debian

Running from an Ubuntu Xenial host:

$ uname -a
Linux ip-xx-xx-xx-xx 4.4.0-1069-aws #79-Ubuntu SMP Mon Sep 24
15:01:41 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ docker ps
CONTAINER ID        IMAGE               COMMAND CREATED             STATUS              PORTS               NAMES
1835dd183b75        debian:9.5          "sh -c 'apt-get -qy …"   43 minutes ago      Up 43 minutes                           kube-node5
30b0af8d2924        debian:9.5          "sh -c 'apt-get -qy …"   43 minutes ago      Up 43 minutes                           kube-node4
3e0d1510c62f        debian:9.5          "sh -c 'apt-get -qy …"   43 minutes ago      Up 43 minutes                           kube-node3
738993566f94        debian:9.5          "sh -c 'apt-get -qy …"   44 minutes ago      Up 44 minutes                           kube-node2
c581ef662ed2        debian:9.5          "sh -c 'apt-get -qy …"   44 minutes ago      Up 44 minutes                           kube-node1

$ docker exec kube-node1 kubectl get node
NAME         STATUS   ROLES         AGE   VERSION
kube-node1   Ready    master,node   18m   v1.12.1
kube-node2   Ready    master,node   17m   v1.12.1
kube-node3   Ready    node          17m   v1.12.1
kube-node4   Ready    node          17m   v1.12.1
kube-node5   Ready    node          17m   v1.12.1

$ docker exec kube-node1 kubectl get pod --all-namespaces
NAMESPACE     NAME                                    READY   STATUS    RESTARTS   AGE
default       netchecker-agent-67489                  1/1     Running   0          2m51s
default       netchecker-agent-6qq6s                  1/1     Running   0          2m51s
default       netchecker-agent-fsw92                  1/1     Running   0          2m51s
default       netchecker-agent-fw6tl                  1/1     Running   0          2m51s
default       netchecker-agent-hostnet-8f2zb          1/1     Running   0          3m
default       netchecker-agent-hostnet-gq7ml          1/1     Running   0          3m
default       netchecker-agent-hostnet-jfkgv          1/1     Running   0          3m
default       netchecker-agent-hostnet-kwfwx          1/1     Running   0          3m
default       netchecker-agent-hostnet-r46nm          1/1     Running   0          3m
default       netchecker-agent-lxdrn                  1/1     Running   0          2m51s
default       netchecker-server-864bd4c897-9vstl      1/1     Running   0          2m40s
default       sh-68fcc6db45-qf55h                     1/1     Running   1          12m
kube-system   coredns-7598f59475-6vknq                1/1     Running   0          14m
kube-system   coredns-7598f59475-l5q5x                1/1     Running   0          14m
kube-system   kube-apiserver-kube-node1               1/1     Running   0          17m
kube-system   kube-apiserver-kube-node2               1/1     Running   0          18m
kube-system   kube-controller-manager-kube-node1      1/1     Running   0          18m
kube-system   kube-controller-manager-kube-node2      1/1     Running   0          18m
kube-system   kube-proxy-5xx9d                        1/1     Running   0          17m
kube-system   kube-proxy-cdqq4                        1/1     Running   0          17m
kube-system   kube-proxy-n64ls                        1/1     Running   0          17m
kube-system   kube-proxy-pswmj                        1/1     Running   0          18m
kube-system   kube-proxy-x89qw                        1/1     Running   0          18m
kube-system   kube-scheduler-kube-node1               1/1     Running   4          17m
kube-system   kube-scheduler-kube-node2               1/1     Running   4          18m
kube-system   kubernetes-dashboard-5db4d9f45f-548rl   1/1     Running   0          14m
kube-system   nginx-proxy-kube-node3                  1/1     Running   4          17m
kube-system   nginx-proxy-kube-node4                  1/1     Running   4          17m
kube-system   nginx-proxy-kube-node5                  1/1     Running   4          17m
kube-system   weave-net-42bfr                         2/2     Running   0          16m
kube-system   weave-net-6gt8m                         2/2     Running   0          16m
kube-system   weave-net-88nnc                         2/2     Running   0          16m
kube-system   weave-net-shckr                         2/2     Running   0          16m
kube-system   weave-net-xr46t                         2/2     Running   0          16m

$ docker exec kube-node1 curl -s http://localhost:31081/api/v1/connectivity_check
{"Message":"All 10 pods successfully reported back to the server","Absent":null,"Outdated":null}