2019-12-04 15:22:57 +00:00
# Upgrading Kubernetes in Kubespray
2016-12-07 13:33:08 +00:00
2017-06-16 17:25:46 +00:00
Kubespray handles upgrades the same way it handles initial deployment. That is to
2019-10-16 11:35:41 +00:00
say that each component is laid down in a fixed order.
You can also individually control versions of components by explicitly defining their
2016-12-07 13:33:08 +00:00
versions. Here are all version vars for each component:
* docker_version
2021-11-08 23:56:49 +00:00
* docker_containerd_version (relevant when `container_manager` == `docker` )
* containerd_version (relevant when `container_manager` == `containerd` )
2016-12-07 13:33:08 +00:00
* kube_version
* etcd_version
* calico_version
* calico_cni_version
* weave_version
* flannel_version
* kubedns_version
2019-10-16 11:35:41 +00:00
:warning: [Attempting to upgrade from an older release straight to the latest release is unsupported and likely to break something ](https://github.com/kubernetes-sigs/kubespray/issues/3849#issuecomment-451386515 ) :warning:
2021-03-15 10:59:05 +00:00
See [Multiple Upgrades ](#multiple-upgrades ) for how to upgrade from older Kubespray release to the latest release
2019-10-16 11:35:41 +00:00
2019-12-04 15:22:57 +00:00
## Unsafe upgrade example
2016-12-07 13:33:08 +00:00
2021-03-15 10:59:05 +00:00
If you wanted to upgrade just kube_version from v1.18.10 to v1.19.7, you could
2016-12-07 13:33:08 +00:00
deploy the following way:
2019-12-04 15:22:57 +00:00
```ShellSession
2021-03-15 10:59:05 +00:00
ansible-playbook cluster.yml -i inventory/sample/hosts.ini -e kube_version=v1.18.10 -e upgrade_cluster_setup=true
2016-12-07 13:33:08 +00:00
```
2021-03-15 10:59:05 +00:00
And then repeat with v1.19.7 as kube_version:
2016-12-07 13:33:08 +00:00
2019-12-04 15:22:57 +00:00
```ShellSession
2021-03-15 10:59:05 +00:00
ansible-playbook cluster.yml -i inventory/sample/hosts.ini -e kube_version=v1.19.7 -e upgrade_cluster_setup=true
2016-12-07 13:33:08 +00:00
```
2020-02-17 09:45:29 +00:00
The var ```-e upgrade_cluster_setup=true``` is needed to be set in order to migrate the deploys of e.g kube-apiserver inside the cluster immediately which is usually only done in the graceful upgrade. (Refer to [#4139 ](https://github.com/kubernetes-sigs/kubespray/issues/4139 ) and [#4736 ](https://github.com/kubernetes-sigs/kubespray/issues/4736 ))
2019-12-04 15:22:57 +00:00
## Graceful upgrade
2017-02-14 16:08:44 +00:00
2018-09-18 20:13:15 +00:00
Kubespray also supports cordon, drain and uncordoning of nodes when performing
2017-02-14 16:08:44 +00:00
a cluster upgrade. There is a separate playbook used for this purpose. It is
important to note that upgrade-cluster.yml can only be used for upgrading an
2021-03-24 00:26:05 +00:00
existing cluster. That means there must be at least 1 kube_control_plane already
2017-02-14 16:08:44 +00:00
deployed.
2019-12-04 15:22:57 +00:00
```ShellSession
2021-03-15 10:59:05 +00:00
ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=v1.19.7
2017-03-30 02:00:52 +00:00
```
2019-09-26 06:27:08 +00:00
After a successful upgrade, the Server Version should be updated:
2017-03-30 02:00:52 +00:00
2019-12-04 15:22:57 +00:00
```ShellSession
2017-03-30 02:00:52 +00:00
$ kubectl version
2021-03-15 10:59:05 +00:00
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
2017-02-14 16:08:44 +00:00
```
2021-01-18 07:53:43 +00:00
If you want to manually control the upgrade procedure, you can use the variables `upgrade_node_confirm` or `upgrade_node_pause_seconds` :
`upgrade_node_confirm: true` - waiting to confirmation to upgrade next node
`upgrade_node_pause_seconds: 60` - pause 60 seconds before upgrade next node
2021-09-14 06:59:07 +00:00
## Node-based upgrade
If you don't want to upgrade all nodes in one run, you can use `--limit` [patterns ](https://docs.ansible.com/ansible/latest/user_guide/intro_patterns.html#patterns-and-ansible-playbook-flags ).
Before using `--limit` run playbook `facts.yml` without the limit to refresh facts cache for all nodes:
```ShellSession
ansible-playbook facts.yml -b -i inventory/sample/hosts.ini
```
After this upgrade control plane and etcd groups [#5147 ](https://github.com/kubernetes-sigs/kubespray/issues/5147 ):
```ShellSession
ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=v1.20.7 --limit "kube_control_plane:etcd"
```
Now you can upgrade other nodes in any order and quantity:
```ShellSession
ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=v1.20.7 --limit "node4:node6:node7:node12"
ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=v1.20.7 --limit "node5*"
```
2019-12-04 15:22:57 +00:00
## Multiple upgrades
2019-10-16 11:35:41 +00:00
:warning: [Do not skip releases when upgrading--upgrade by one tag at a time. ](https://github.com/kubernetes-sigs/kubespray/issues/3849#issuecomment-451386515 ) :warning:
For instance, if you're on v2.6.0, then check out v2.7.0, run the upgrade, check out the next tag, and run the next upgrade, etc.
2021-04-29 12:20:50 +00:00
Assuming you don't explicitly define a kubernetes version in your k8s_cluster.yml, you simply check out the next tag and run the upgrade-cluster.yml playbook
2019-10-16 11:35:41 +00:00
2021-04-29 12:20:50 +00:00
* If you do define kubernetes version in your inventory (e.g. group_vars/k8s_cluster.yml) then either make sure to update it before running upgrade-cluster, or specify the new version you're upgrading to: `ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml -e kube_version=v1.11.3`
2020-02-17 09:45:29 +00:00
2019-10-16 11:35:41 +00:00
Otherwise, the upgrade will leave your cluster at the same k8s version defined in your inventory vars.
The below example shows taking a cluster that was set up for v2.6.0 up to v2.10.0
2019-12-04 15:22:57 +00:00
```ShellSession
2019-10-16 11:35:41 +00:00
$ kubectl get node
NAME STATUS ROLES AGE VERSION
apollo Ready master,node 1h v1.10.4
boomer Ready master,node 42m v1.10.4
caprica Ready master,node 42m v1.10.4
$ git describe --tags
v2.6.0
$ git tag
...
v2.6.0
v2.7.0
v2.8.0
v2.8.1
v2.8.2
...
$ git checkout v2.7.0
Previous HEAD position was 8b3ce6e4 bump upgrade tests to v2.5.0 commit (#3087)
HEAD is now at 05dabb7e Fix Bionic networking restart error #3430 (#3431)
# NOTE: May need to sudo pip3 install -r requirements.txt when upgrading.
2019-12-04 15:22:57 +00:00
ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
2019-10-16 11:35:41 +00:00
...
$ kubectl get node
NAME STATUS ROLES AGE VERSION
apollo Ready master,node 1h v1.11.3
boomer Ready master,node 1h v1.11.3
caprica Ready master,node 1h v1.11.3
$ git checkout v2.8.0
Previous HEAD position was 05dabb7e Fix Bionic networking restart error #3430 (#3431)
HEAD is now at 9051aa52 Fix ubuntu-contiv test failed (#3808)
```
:info: NOTE: Review changes between the sample inventory and your inventory when upgrading versions. :info:
Some deprecations between versions that mean you can't just upgrade straight from 2.7.0 to 2.8.0 if you started with the sample inventory.
In this case, I set "kubeadm_enabled" to false, knowing that it is deprecated and removed by 2.9.0, to delay converting the cluster to kubeadm as long as I could.
2019-12-04 15:22:57 +00:00
```ShellSession
2019-10-16 11:35:41 +00:00
$ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
...
"msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
...
Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
yes
...
$ kubectl get node
NAME STATUS ROLES AGE VERSION
apollo Ready master,node 114m v1.12.3
boomer Ready master,node 114m v1.12.3
caprica Ready master,node 114m v1.12.3
$ git checkout v2.8.1
Previous HEAD position was 9051aa52 Fix ubuntu-contiv test failed (#3808)
HEAD is now at 2ac1c756 More Feature/2.8 backports for 2.8.1 (#3911)
$ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
...
"msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
...
Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
yes
...
$ kubectl get node
NAME STATUS ROLES AGE VERSION
apollo Ready master,node 2h36m v1.12.4
boomer Ready master,node 2h36m v1.12.4
caprica Ready master,node 2h36m v1.12.4
$ git checkout v2.8.2
Previous HEAD position was 2ac1c756 More Feature/2.8 backports for 2.8.1 (#3911)
HEAD is now at 4167807f Upgrade to 1.12.5 (#4066)
$ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
...
"msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
...
Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
yes
...
$ kubectl get node
NAME STATUS ROLES AGE VERSION
apollo Ready master,node 3h3m v1.12.5
boomer Ready master,node 3h3m v1.12.5
caprica Ready master,node 3h3m v1.12.5
$ git checkout v2.8.3
Previous HEAD position was 4167807f Upgrade to 1.12.5 (#4066)
HEAD is now at ea41fc5e backport cve-2019-5736 to release-2.8 (#4234)
$ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
...
"msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
...
Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
yes
...
$ kubectl get node
NAME STATUS ROLES AGE VERSION
apollo Ready master,node 5h18m v1.12.5
boomer Ready master,node 5h18m v1.12.5
caprica Ready master,node 5h18m v1.12.5
$ git checkout v2.8.4
Previous HEAD position was ea41fc5e backport cve-2019-5736 to release-2.8 (#4234)
HEAD is now at 3901480b go to k8s 1.12.7 (#4400)
$ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
...
"msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
...
Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
yes
...
$ kubectl get node
NAME STATUS ROLES AGE VERSION
apollo Ready master,node 5h37m v1.12.7
boomer Ready master,node 5h37m v1.12.7
caprica Ready master,node 5h37m v1.12.7
$ git checkout v2.8.5
Previous HEAD position was 3901480b go to k8s 1.12.7 (#4400)
HEAD is now at 6f97687d Release 2.8 robust san handling (#4478)
$ ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
...
"msg": "DEPRECATION: non-kubeadm deployment is deprecated from v2.9. Will be removed in next release."
...
Are you sure you want to deploy cluster using the deprecated non-kubeadm mode. (output is hidden):
yes
...
$ kubectl get node
NAME STATUS ROLES AGE VERSION
apollo Ready master,node 5h45m v1.12.7
boomer Ready master,node 5h45m v1.12.7
caprica Ready master,node 5h45m v1.12.7
$ git checkout v2.9.0
Previous HEAD position was 6f97687d Release 2.8 robust san handling (#4478)
HEAD is now at a4e65c7c Upgrade to Ansible >2.7.0 (#4471)
```
2021-04-29 12:20:50 +00:00
:warning: IMPORTANT: Some of the variable formats changed in the k8s_cluster.yml between 2.8.5 and 2.9.0 :warning:
2019-10-16 11:35:41 +00:00
If you do not keep your inventory copy up to date, **your upgrade will fail** and your first master will be left non-functional until fixed and re-run.
It is at this point the cluster was upgraded from non-kubeadm to kubeadm as per the deprecation warning.
2019-12-04 15:22:57 +00:00
```ShellSession
ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
2019-10-16 11:35:41 +00:00
...
$ kubectl get node
NAME STATUS ROLES AGE VERSION
apollo Ready master,node 6h54m v1.13.5
boomer Ready master,node 6h55m v1.13.5
caprica Ready master,node 6h54m v1.13.5
# Watch out: 2.10.0 is hiding between 2.1.2 and 2.2.0
$ git tag
...
v2.1.0
v2.1.1
v2.1.2
v2.10.0
v2.2.0
...
$ git checkout v2.10.0
Previous HEAD position was a4e65c7c Upgrade to Ansible >2.7.0 (#4471)
HEAD is now at dcd9c950 Add etcd role dependency on kube user to avoid etcd role failure when running scale.yml with a fresh node. (#3240) (#4479)
2019-12-04 15:22:57 +00:00
ansible-playbook -i inventory/mycluster/hosts.ini -b upgrade-cluster.yml
2019-10-16 11:35:41 +00:00
...
$ kubectl get node
NAME STATUS ROLES AGE VERSION
apollo Ready master,node 7h40m v1.14.1
boomer Ready master,node 7h40m v1.14.1
caprica Ready master,node 7h40m v1.14.1
```
2019-12-04 15:22:57 +00:00
## Upgrade order
2016-12-07 13:33:08 +00:00
As mentioned above, components are upgraded in the order in which they were
installed in the Ansible playbook. The order of component installation is as
follows:
2017-03-16 16:53:48 +00:00
* Docker
2021-04-02 06:20:11 +00:00
* Containerd
2017-03-16 16:53:48 +00:00
* etcd
* kubelet and kube-proxy
* network_plugin (such as Calico or Weave)
* kube-apiserver, kube-scheduler, and kube-controller-manager
* Add-ons (such as KubeDNS)
2017-09-26 09:38:58 +00:00
2018-07-06 06:12:13 +00:00
### Component-based upgrades
A deployer may want to upgrade specific components in order to minimize risk
or save time. This strategy is not covered by CI as of this writing, so it is
2018-09-13 13:50:42 +00:00
not guaranteed to work.
2018-07-06 06:12:13 +00:00
These commands are useful only for upgrading fully-deployed, healthy, existing
hosts. This will definitely not work for undeployed or partially deployed
hosts.
2018-07-10 10:37:37 +00:00
Upgrade docker:
2019-12-04 15:22:57 +00:00
```ShellSession
2018-07-10 10:37:37 +00:00
ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=docker
```
2018-07-06 06:12:13 +00:00
Upgrade etcd:
2019-12-04 15:22:57 +00:00
```ShellSession
2018-07-06 06:12:13 +00:00
ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=etcd
```
2021-08-26 07:21:23 +00:00
Upgrade etcd without rotating etcd certs:
```ShellSession
ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=etcd --limit=etcd --skip-tags=etcd-secrets
```
2018-07-06 06:12:13 +00:00
Upgrade kubelet:
2019-12-04 15:22:57 +00:00
```ShellSession
2018-07-06 06:12:13 +00:00
ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=node --skip-tags=k8s-gen-certs,k8s-gen-tokens
```
Upgrade Kubernetes master components:
2019-12-04 15:22:57 +00:00
```ShellSession
2018-07-06 06:12:13 +00:00
ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=master
```
Upgrade network plugins:
2019-12-04 15:22:57 +00:00
```ShellSession
2018-07-06 06:12:13 +00:00
ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=network
```
Upgrade all add-ons:
2019-12-04 15:22:57 +00:00
```ShellSession
2018-07-06 06:12:13 +00:00
ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=apps
```
Upgrade just helm (assuming `helm_enabled` is true):
2019-12-04 15:22:57 +00:00
```ShellSession
2018-07-06 06:12:13 +00:00
ansible-playbook -b -i inventory/sample/hosts.ini cluster.yml --tags=helm
```