4.1 KiB
Adding/replacing a node
Modified from comments in #3471
Adding/replacing a worker node
This should be the easiest.
1) Add new node to the inventory
2) Run scale.yml
You can use --limit=node1 to limit Kubespray to avoid disturbing other nodes in the cluster.
3) Drain the node that will be removed
kubectl drain NODE_NAME
4) Run the remove-node.yml playbook
With the old node still in the inventory, run remove-node.yml. You need to pass -e node=NODE_NAME to the playbook to limit the execution to the node being removed.
5) Remove the node from the inventory
That's it.
Adding/replacing a master node
1) Recreate apiserver certs manually to include the new master node in the cert SAN field
For some reason, Kubespray will not update the apiserver certificate.
Edit /etc/kubernetes/kubeadm-config.yaml, include new host in certSANs list.
Use kubeadm to recreate the certs.
cd /etc/kubernetes/ssl
mv apiserver.crt apiserver.crt.old
mv apiserver.key apiserver.key.old
cd /etc/kubernetes
kubeadm init phase certs apiserver --config kubeadm-config.yaml
Check the certificate, new host needs to be there.
openssl x509 -text -noout -in /etc/kubernetes/ssl/apiserver.crt
2) Run cluster.yml
Add the new host to the inventory and run cluster.yml.
3) Restart kube-system/nginx-proxy
In all hosts, restart nginx-proxy pod. This pod is a local proxy for the apiserver. Kubespray will update its static config, but it needs to be restarted in order to reload.
# run in every host
docker ps | grep k8s_nginx-proxy_nginx-proxy | awk '{print $1}' | xargs docker restart
4) Remove old master nodes
If you are replacing a node, remove the old one from the inventory, and remove from the cluster runtime.
kubectl drain NODE_NAME
kubectl delete node NODE_NAME
After that, the old node can be safely shutdown. Also, make sure to restart nginx-proxy in all remaining nodes (step 3)
From any active master that remains in the cluster, re-upload kubeadm-config.yaml
kubeadm config upload from-file --config /etc/kubernetes/kubeadm-config.yaml
Adding/Replacing an etcd node
You need to make sure there are always an odd number of etcd nodes in the cluster. In such a way, this is always a replace or scale up operation. Either add two new nodes or remove an old one.
1) Add the new node running cluster.yml
Update the inventory and run cluster.yml passing --limit=etcd,kube-master -e ignore_assert_errors=yes.
Run upgrade-cluster.yml also passing --limit=etcd,kube-master -e ignore_assert_errors=yes. This is necessary to update all etcd configuration in the cluster.
At this point, you will have an even number of nodes. Everything should still be working, and you should only have problems if the cluster decides to elect a new etcd leader before you remove a node. Even so, running applications should continue to be available.
2) Remove an old etcd node
With the node still in the inventory, run remove-node.yml passing -e node=NODE_NAME as the name of the node that should be removed.
3) Make sure the remaining etcd members have their config updated
In each etcd host that remains in the cluster:
cat /etc/etcd.env | grep ETCD_INITIAL_CLUSTER
Only active etcd members should be in that list.
4) Remove old etcd members from the cluster runtime
Acquire a shell prompt into one of the etcd containers and use etcdctl to remove the old member.
# list all members
etcdctl member list
# remove old member
etcdctl member remove MEMBER_ID
# careful!!! if you remove a wrong member you will be in trouble
# note: these command lines are actually much bigger, since you need to pass all certificates to etcdctl.
5) Make sure the apiserver config is correctly updated
In every master node, edit /etc/kubernetes/manifests/kube-apiserver.yaml. Make sure only active etcd nodes are still present in the apiserver command line parameter --etcd-servers=....