[2.18] preinstall: Add nodelocaldns to supersede_nameserver if enabled

When a machine that use dhclient and resolvconf reboots, this will make /etc/resolv.conf remain close to the one before reboot
disable kubelet_authorization_mode_webhook by default
2022-09-27 09:06:03 +02:00 · 2022-08-31 11:44:14 +02:00 · 2022-07-12 09:24:39 +02:00 · 2022-07-12 09:24:39 +02:00 · 2022-07-12 09:24:39 +02:00 · 2022-04-09 11:02:04 -07:00
46 changed files with 275 additions and 145 deletions
--- a/.gitlab-ci/packet.yml
+++ b/.gitlab-ci/packet.yml
@ -70,7 +70,7 @@ packet_centos7-flannel-addons-ha:
  stage: deploy-part2
  when: on_success

-packet_centos8-crio:
+packet_almalinux8-crio:
  extends: .packet_pr
  stage: deploy-part2
  when: on_success
@ -145,17 +145,17 @@ packet_centos7-calico-ha-once-localhost:
  services:
    - docker:19.03.9-dind

-packet_centos8-kube-ovn:
+packet_almalinux8-kube-ovn:
  stage: deploy-part2
  extends: .packet_periodic
  when: on_success

-packet_centos8-calico:
+packet_almalinux8-calico:
  stage: deploy-part2
  extends: .packet_pr
  when: on_success

-packet_centos8-docker:
+packet_almalinux8-docker:
  stage: deploy-part2
  extends: .packet_pr
  when: on_success
@ -203,7 +203,7 @@ packet_ubuntu18-flannel-ha-once:
  when: manual

 # Calico HA eBPF
-packet_centos8-calico-ha-ebpf:
+packet_almalinux8-calico-ha-ebpf:
  stage: deploy-part2
  extends: .packet_pr
  when: manual
@ -255,7 +255,7 @@ packet_amazon-linux-2-aio:
  extends: .packet_pr
  when: manual

-packet_centos8-calico-nodelocaldns-secondary:
+packet_almalinux8-calico-nodelocaldns-secondary:
  stage: deploy-part2
  extends: .packet_pr
  when: manual
--- a/.gitlab-ci/vagrant.yml
+++ b/.gitlab-ci/vagrant.yml
@ -4,7 +4,7 @@ molecule_tests:
  tags: [c3.small.x86]
  only: [/^pr-.*$/]
  except: ['triggers']
-  image: quay.io/kubespray/vagrant:$KUBESPRAY_VERSION
+  image: quay.io/kubespray/vagrant:v2.18.0
  services: []
  stage: deploy-part1
  before_script:
@ -33,7 +33,7 @@ molecule_tests:
  tags: [c3.small.x86]
  only: [/^pr-.*$/]
  except: ['triggers']
-  image: quay.io/kubespray/vagrant:$KUBESPRAY_VERSION
+  image: quay.io/kubespray/vagrant:v2.18.0
  services: []
  before_script:
    - apt-get update && apt-get install -y python3-pip
--- a/4
+++ b/4
@ -1,5 +1,5 @@
-# Use imutable image tags rather than mutable tags (like ubuntu:18.04)
-FROM ubuntu:bionic-20200807
+# Use imutable image tags rather than mutable tags (like ubuntu:20.04)
+FROM ubuntu:focal-20220316

 RUN apt update -y \
    && apt install -y \
--- a/README.md
+++ b/README.md
@ -131,7 +131,7 @@ Note: Upstart/SysV init based OS types are not supported.
 ## Supported Components

 - Core
-  - [kubernetes](https://github.com/kubernetes/kubernetes) v1.22.5
+  - [kubernetes](https://github.com/kubernetes/kubernetes) v1.22.8
  - [etcd](https://github.com/coreos/etcd) v3.5.0
  - [docker](https://www.docker.com/) v20.10 (see note)
  - [containerd](https://containerd.io/) v1.5.8
--- a/2
+++ b/2
@ -26,6 +26,8 @@ SUPPORTED_OS = {
  "centos-bento"        => {box: "bento/centos-7.6",           user: "vagrant"},
  "centos8"             => {box: "centos/8",                   user: "vagrant"},
  "centos8-bento"       => {box: "bento/centos-8",             user: "vagrant"},
+  "almalinux8"          => {box: "almalinux/8",                user: "vagrant"},
+  "almalinux8-bento"    => {box: "bento/almalinux-8",          user: "vagrant"},
  "fedora34"            => {box: "fedora/34-cloud-base",       user: "vagrant"},
  "fedora35"            => {box: "fedora/35-cloud-base",       user: "vagrant"},
  "opensuse"            => {box: "bento/opensuse-leap-15.2",   user: "vagrant"},
--- a/contrib/offline/README.md
+++ b/contrib/offline/README.md
@ -28,16 +28,19 @@ manage-offline-container-images.sh   register

 This script generates the list of downloaded files and the list of container images by `roles/download/defaults/main.yml` file.

-Run this script will generates three files, all downloaded files url in files.list, all container images in images.list, all component version in generate.sh.
+Run this script will execute `generate_list.yml` playbook in kubespray root directory and generate four files,
+all downloaded files url in files.list, all container images in images.list, jinja2 templates in *.template.

 ```shell
-bash generate_list.sh
+./generate_list.sh
 tree temp
 temp
 ├── files.list
-├── generate.sh
-└── images.list
-0 directories, 3 files
+├── files.list.template
+├── images.list
+└── images.list.template
+0 directories, 5 files
 ```

-In some cases you may want to update some component version, you can edit `generate.sh` file, then run `bash generate.sh | grep 'https' > files.list` to update file.list or run `bash generate.sh | grep -v 'https'> images.list` to update images.list.
+In some cases you may want to update some component version, you can declare version variables in ansible inventory file or group_vars,
+then run `./generate_list.sh -i [inventory_file]` to update file.list and images.list.
--- a/contrib/offline/generate_list.sh
+++ b/contrib/offline/generate_list.sh
@ -5,53 +5,26 @@ CURRENT_DIR=$(cd $(dirname $0); pwd)
 TEMP_DIR="${CURRENT_DIR}/temp"
 REPO_ROOT_DIR="${CURRENT_DIR%/contrib/offline}"

-: ${IMAGE_ARCH:="amd64"}
-: ${ANSIBLE_SYSTEM:="linux"}
-: ${ANSIBLE_ARCHITECTURE:="x86_64"}
 : ${DOWNLOAD_YML:="roles/download/defaults/main.yml"}
-: ${KUBE_VERSION_YAML:="roles/kubespray-defaults/defaults/main.yaml"}

 mkdir -p ${TEMP_DIR}

-# ARCH used in convert {%- if image_arch != 'amd64' -%}-{{ image_arch }}{%- endif -%} to {{arch}}
-if [ "${IMAGE_ARCH}" != "amd64" ]; then ARCH="${IMAGE_ARCH}"; fi
-
-cat > ${TEMP_DIR}/generate.sh << EOF
-arch=${ARCH}
-image_arch=${IMAGE_ARCH}
-ansible_system=${ANSIBLE_SYSTEM}
-ansible_architecture=${ANSIBLE_ARCHITECTURE}
-EOF
-
-# generate all component version by $DOWNLOAD_YML
-grep 'kube_version:' ${REPO_ROOT_DIR}/${KUBE_VERSION_YAML} \
-| sed 's/: /=/g' >> ${TEMP_DIR}/generate.sh
-grep '_version:' ${REPO_ROOT_DIR}/${DOWNLOAD_YML} \
-| sed 's/: /=/g;s/{{/${/g;s/}}/}/g' | tr -d ' ' >> ${TEMP_DIR}/generate.sh
-sed -i 's/kube_major_version=.*/kube_major_version=${kube_version%.*}/g' ${TEMP_DIR}/generate.sh
-sed -i 's/crictl_version=.*/crictl_version=${kube_version%.*}.0/g' ${TEMP_DIR}/generate.sh
-
-# generate all download files url
+# generate all download files url template
 grep 'download_url:' ${REPO_ROOT_DIR}/${DOWNLOAD_YML} \
-| sed 's/: /=/g;s/ //g;s/{{/${/g;s/}}/}/g;s/|lower//g;s/^.*_url=/echo /g' >> ${TEMP_DIR}/generate.sh
+    | sed 's/^.*_url: //g;s/\"//g' > ${TEMP_DIR}/files.list.template

-# generate all images list
-grep -E '_repo:|_tag:' ${REPO_ROOT_DIR}/${DOWNLOAD_YML} \
-| sed "s#{%- if image_arch != 'amd64' -%}-{{ image_arch }}{%- endif -%}#{{arch}}#g" \
-| sed 's/: /=/g;s/{{/${/g;s/}}/}/g' | tr -d ' ' >> ${TEMP_DIR}/generate.sh
+# generate all images list template
 sed -n '/^downloads:/,/download_defaults:/p' ${REPO_ROOT_DIR}/${DOWNLOAD_YML} \
-| sed -n "s/repo: //p;s/tag: //p" | tr -d ' ' | sed 's/{{/${/g;s/}}/}/g' \
-| sed 'N;s#\n# #g' | tr ' ' ':' | sed 's/^/echo /g' >> ${TEMP_DIR}/generate.sh
+    | sed -n "s/repo: //p;s/tag: //p" | tr -d ' ' \
+    | sed 'N;s#\n# #g' | tr ' ' ':' | sed 's/\"//g' > ${TEMP_DIR}/images.list.template

-# special handling for https://github.com/kubernetes-sigs/kubespray/pull/7570
-sed -i 's#^coredns_image_repo=.*#coredns_image_repo=${kube_image_repo}$(if printf "%s\\n%s\\n" v1.21 ${kube_version%.*} | sort --check=quiet --version-sort; then echo -n /coredns/coredns;else echo -n /coredns; fi)#' ${TEMP_DIR}/generate.sh
-sed -i 's#^coredns_image_tag=.*#coredns_image_tag=$(if printf "%s\\n%s\\n" v1.21 ${kube_version%.*} | sort --check=quiet --version-sort; then echo -n ${coredns_version};else echo -n ${coredns_version/v/}; fi)#' ${TEMP_DIR}/generate.sh
-
-# add kube-* images to images list
+# add kube-* images to images list template
 KUBE_IMAGES="kube-apiserver kube-controller-manager kube-scheduler kube-proxy"
-echo "${KUBE_IMAGES}" | tr ' ' '\n' | xargs -L1 -I {} \
-echo 'echo ${kube_image_repo}/{}:${kube_version}' >> ${TEMP_DIR}/generate.sh
+for i in $KUBE_IMAGES; do
+    echo "{{ kube_image_repo }}/$i:{{ kube_version }}" >> ${TEMP_DIR}/images.list.template
+done

-# print files.list and images.list
-bash ${TEMP_DIR}/generate.sh | grep 'https' | sort > ${TEMP_DIR}/files.list
-bash ${TEMP_DIR}/generate.sh | grep -v 'https' | sort > ${TEMP_DIR}/images.list
+# run ansible to expand templates
+/bin/cp ${CURRENT_DIR}/generate_list.yml ${REPO_ROOT_DIR}
+
+(cd ${REPO_ROOT_DIR} && ansible-playbook $* generate_list.yml && /bin/rm generate_list.yml) || exit 1
--- a/contrib/offline/generate_list.yml
+++ b/contrib/offline/generate_list.yml
@ -0,0 +1,20 @@
+---
+- hosts: localhost
+  become: false
+
+  roles:
+    # Just load default variables from roles.
+    - role: kubespray-defaults
+      when: false
+    - role: download
+      when: false
+
+  tasks:
+    - name: Generate files.list and images.list files from templates
+      template:
+        src: ./contrib/offline/temp/{{ item }}.list.template
+        dest: ./contrib/offline/temp/{{ item }}.list
+        mode: 0644
+      with_items:
+        - files
+        - images
--- a/docs/ci.md
+++ b/docs/ci.md
@ -6,9 +6,9 @@ To generate this Matrix run `./tests/scripts/md-table/main.py`

 | OS / CNI | calico | canal | cilium | flannel | kube-ovn | kube-router | macvlan | weave |
 |---| --- | --- | --- | --- | --- | --- | --- | --- |
+almalinux8 |  :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: |
 amazon |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 centos7 |  :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :white_check_mark: | :x: | :x: |
-centos8 |  :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :x: | :x: | :x: |
 debian10 |  :white_check_mark: | :x: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
 debian11 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 debian9 |  :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: |
@ -24,9 +24,9 @@ ubuntu20 |  :white_check_mark: | :x: | :x: | :white_check_mark: | :x: | :x: | :x

 | OS / CNI | calico | canal | cilium | flannel | kube-ovn | kube-router | macvlan | weave |
 |---| --- | --- | --- | --- | --- | --- | --- | --- |
+almalinux8 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 amazon |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 centos7 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
-centos8 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 debian10 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 debian11 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 debian9 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
@ -42,9 +42,9 @@ ubuntu20 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |

 | OS / CNI | calico | canal | cilium | flannel | kube-ovn | kube-router | macvlan | weave |
 |---| --- | --- | --- | --- | --- | --- | --- | --- |
+almalinux8 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 amazon |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 centos7 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :white_check_mark: |
-centos8 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 debian10 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 debian11 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
 debian9 |  :x: | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
--- a/docs/containerd.md
+++ b/docs/containerd.md
@ -28,4 +28,13 @@ containerd_registries:
    - "https://registry-1.docker.io"
 ```

+`containerd_registries` is ignored for pulling images when `image_command_tool=nerdctl`
+(the default for `container_manager=containerd`). Use `crictl` instead, it supports
+`containerd_registries` but lacks proper multi-arch support (see
+[#8375](https://github.com/kubernetes-sigs/kubespray/issues/8375)):
+
+```yaml
+image_command_tool: crictl
+```
+
 [containerd]: https://containerd.io/
--- a/docs/vars.md
+++ b/docs/vars.md
@ -52,7 +52,11 @@ Kubernetes needs some parameters in order to get deployed. These are the
 following default cluster parameters:

 * *cluster_name* - Name of cluster (default is cluster.local)
-* *container_manager* - Container Runtime to install in the nodes (default is docker)
+* *container_manager* - Container Runtime to install in the nodes (default is containerd)
+* *image_command_tool* - Tool used to pull images (default depends on `container_manager`
+  and is `nerdctl` for `containerd`, `crictl` for `crio`, `docker` for `docker`)
+* *image_command_tool_on_localhost* - Tool used to pull images on localhost
+  (default is equal to `image_command_tool`)
 * *dns_domain* - Name of cluster DNS domain (default is cluster.local)
 * *kube_network_plugin* - Plugin to use for container networking
 * *kube_service_addresses* - Subnet for cluster IPs (default is
--- a/inventory/sample/group_vars/all/containerd.yml
+++ b/inventory/sample/group_vars/all/containerd.yml
@ -33,10 +33,11 @@
 ## An obvious use case is allowing insecure-registry access to self hosted registries.
 ## Can be ipaddress and domain_name.
 ## example define mirror.registry.io or 172.19.16.11:5000
+## set "name": "url". insecure url must be started http://
 ## Port number is also needed if the default HTTPS port is not used.
 # containerd_insecure_registries:
-#   - mirror.registry.io
-#   - 172.19.16.11:5000
+#   "localhost": "http://127.0.0.1"
+#   "172.19.16.11:5000": "http://172.19.16.11:5000"

 # containerd_registries:
 #   "docker.io": "https://registry-1.docker.io"
--- a/inventory/sample/group_vars/all/offline.yml
+++ b/inventory/sample/group_vars/all/offline.yml
@ -55,6 +55,7 @@
 # [Optional] runc,containerd: only if you set container_runtime: containerd
 # runc_download_url: "{{ files_repo }}/{{ runc_version }}/runc.{{ image_arch }}"
 # containerd_download_url: "{{ files_repo }}/containerd/v{{ containerd_version }}/containerd-{{ containerd_version }}-linux-{{ image_arch }}.tar.gz"
+# nerdctl_download_url: "{{ files_repo }}/nerdctl/v{{ nerdctl_version }}/nerdctl-{{ nerdctl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"

 ## CentOS/Redhat/AlmaLinux
 ### For EL7, base and extras repo must be available, for EL8, baseos and appstream
--- a/inventory/sample/group_vars/k8s_cluster/addons.yml
+++ b/inventory/sample/group_vars/k8s_cluster/addons.yml
@ -134,6 +134,7 @@ cert_manager_enabled: false
 #   -----BEGIN CERTIFICATE-----
 #   [REPLACE with your CA certificate]
 #   -----END CERTIFICATE-----
+# cert_manager_leader_election_namespace: kube-system

 # MetalLB deployment
 metallb_enabled: false
--- a/inventory/sample/group_vars/k8s_cluster/k8s-cluster.yml
+++ b/inventory/sample/group_vars/k8s_cluster/k8s-cluster.yml
@ -17,7 +17,7 @@ kube_token_dir: "{{ kube_config_dir }}/tokens"
 kube_api_anonymous_auth: true

 ## Change this to use another Kubernetes version, e.g. a current beta release
-kube_version: v1.22.5
+kube_version: v1.22.8

 # Where the binaries will be downloaded.
 # Note: ensure that you've enough disk space (about 1G)
--- a/requirements-2.10.txt
+++ b/requirements-2.10.txt
@ -6,5 +6,5 @@ netaddr==0.7.19
 pbr==5.4.4
 jmespath==0.9.5
 ruamel.yaml==0.16.10
-ruamel.yaml.clib==0.2.4
+ruamel.yaml.clib==0.2.6
 MarkupSafe==1.1.1
--- a/requirements-2.11.txt
+++ b/requirements-2.11.txt
@ -6,5 +6,5 @@ netaddr==0.7.19
 pbr==5.4.4
 jmespath==0.9.5
 ruamel.yaml==0.16.10
-ruamel.yaml.clib==0.2.4
+ruamel.yaml.clib==0.2.6
 MarkupSafe==1.1.1
--- a/requirements-2.9.txt
+++ b/requirements-2.9.txt
@ -4,6 +4,6 @@ netaddr==0.7.19
 pbr==5.4.4
 jmespath==0.9.5
 ruamel.yaml==0.16.10
-ruamel.yaml.clib==0.2.4 ; python_version >= '3.5'
+ruamel.yaml.clib==0.2.6 ; python_version >= '3.5'
 ruamel.yaml.clib==0.2.2 ; python_version < '3.5'
 MarkupSafe==1.1.1
--- a/roles/bootstrap-os/molecule/default/molecule.yml
+++ b/roles/bootstrap-os/molecule/default/molecule.yml
@ -25,6 +25,10 @@ platforms:
    box: centos/7
    cpus: 1
    memory: 512
+  - name: almalinux8
+    box: almalinux/8
+    cpus: 1
+    memory: 512
  - name: debian9
    box: generic/debian9
    cpus: 1
--- a/roles/container-engine/containerd/molecule/default/molecule.yml
+++ b/roles/container-engine/containerd/molecule/default/molecule.yml
@ -23,8 +23,8 @@ platforms:
      - kube_control_plane
      - kube_node
      - k8s_cluster
-  - name: centos8
-    box: generic/centos8
+  - name: almalinux8
+    box: almalinux/8
    cpus: 1
    memory: 1024
    groups:
--- a/roles/container-engine/containerd/templates/config.toml.j2
+++ b/roles/container-engine/containerd/templates/config.toml.j2
@ -54,12 +54,14 @@ oom_score = {{ containerd_oom_score }}
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."{{ registry }}"]
          endpoint = ["{{ ([ addr ] | flatten ) | join('","') }}"]
 {% endfor %}
-{% for addr in containerd_insecure_registries %}
-        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."{{ addr }}"]
+{% if containerd_insecure_registries is defined and containerd_insecure_registries|length>0 %}
+{% for registry, addr in containerd_insecure_registries.items() %}
+        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."{{ registry }}"]
          endpoint = ["{{ ([ addr ] | flatten ) | join('","') }}"]
-        [plugins."io.containerd.grpc.v1.cri".registry.configs."{{ addr }}".tls]
+        [plugins."io.containerd.grpc.v1.cri".registry.configs."{{ registry }}".tls]
          insecure_skip_verify = true
 {% endfor %}
+{% endif %}
 {% for registry in containerd_registry_auth if registry['registry'] is defined %}
 {% if (registry['username'] is defined and registry['password'] is defined) or registry['auth'] is defined %}
      [plugins."io.containerd.grpc.v1.cri".registry.configs."{{ registry['registry'] }}".auth]
--- a/roles/container-engine/cri-o/molecule/default/molecule.yml
+++ b/roles/container-engine/cri-o/molecule/default/molecule.yml
@ -13,14 +13,8 @@ platforms:
    memory: 1024
    groups:
      - kube_control_plane
-  - name: centos7
-    box: centos/7
-    cpus: 2
-    memory: 1024
-    groups:
-      - kube_control_plane
-  - name: centos8
-    box: centos/8
+  - name: almalinux8
+    box: almalinux/8
    cpus: 2
    memory: 1024
    groups:
--- a/roles/container-engine/docker/molecule/default/molecule.yml
+++ b/roles/container-engine/docker/molecule/default/molecule.yml
@ -7,10 +7,14 @@ lint: |
  set -e
  yamllint -c ../../../.yamllint .
 platforms:
-  - name: kubespray-docker
+  - name: ubuntu18
    box: generic/ubuntu1804
    cpus: 2
    memory: 1024
+  - name: almalinux8
+    box: almalinux/8
+    cpus: 2
+    memory: 1024
 provisioner:
  name: ansible
  env:
--- a/roles/container-engine/gvisor/molecule/default/molecule.yml
+++ b/roles/container-engine/gvisor/molecule/default/molecule.yml
@ -16,8 +16,8 @@ platforms:
    nested: true
    groups:
      - kube_control_plane
-  - name: centos8
-    box: generic/centos8
+  - name: almalinux8
+    box: almalinux/8
    cpus: 1
    memory: 1024
    nested: true
--- a/roles/download/defaults/main.yml
+++ b/roles/download/defaults/main.yml
@ -45,9 +45,28 @@ download_validate_certs: true
 # Use the first kube_control_plane if download_localhost is not set
 download_delegate: "{% if download_localhost %}localhost{% else %}{{ groups['kube_control_plane'][0] }}{% endif %}"

+# The docker_image_info_command might seems weird but we are using raw/endraw and `{{ `{{` }}` to manage the double jinja2 processing
+docker_image_pull_command: "{{ docker_bin_dir }}/docker pull"
+docker_image_info_command: "{{ docker_bin_dir }}/docker images -q | xargs -i {{ '{{' }} docker_bin_dir }}/docker inspect -f {% raw %}'{{ '{{' }} if .RepoTags }}{{ '{{' }} join .RepoTags \",\" }}{{ '{{' }} end }}{{ '{{' }} if .RepoDigests }},{{ '{{' }} join .RepoDigests \",\" }}{{ '{{' }} end }}' {% endraw %} {} | tr '\n' ','"
+nerdctl_image_info_command: "{{ bin_dir }}/nerdctl -n k8s.io images --format '{% raw %}{{ '{{' }} .Repository {{ '}}' }}:{{ '{{' }} .Tag {{ '}}' }}{% endraw %}' 2>/dev/null | grep -v ^:$ | tr '\n' ','"
+nerdctl_image_pull_command: "{{ bin_dir }}/nerdctl -n k8s.io pull --quiet {{ nerdctl_extra_flags }}"
+crictl_image_info_command: "{{ bin_dir }}/crictl images --verbose | awk -F ': ' '/RepoTags|RepoDigests/ {print $2}' | tr '\n' ','"
+crictl_image_pull_command: "{{ bin_dir }}/crictl pull"
+
+image_command_tool: "{%- if container_manager == 'containerd' -%}nerdctl{%- elif container_manager == 'crio' -%}crictl{%- else -%}{{ container_manager }}{%- endif -%}"
+image_command_tool_on_localhost: "{{ image_command_tool }}"
+
+image_pull_command: "{{ lookup('vars', image_command_tool + '_image_pull_command') }}"
+image_info_command: "{{ lookup('vars', image_command_tool + '_image_info_command') }}"
+image_pull_command_on_localhost: "{{ lookup('vars', image_command_tool_on_localhost + '_image_pull_command') }}"
+image_info_command_on_localhost: "{{ lookup('vars', image_command_tool_on_localhost + '_image_info_command') }}"
+
 # Arch of Docker images and needed packages
 image_arch: "{{host_architecture | default('amd64')}}"

+# Nerdctl insecure flag set
+nerdctl_extra_flags: '{%- if containerd_insecure_registries is defined and containerd_insecure_registries|length>0 -%}--insecure-registry{%- else -%}{%- endif -%}'
+
 # Versions
 kubeadm_version: "{{ kube_version }}"
 etcd_version: v3.5.0
@ -55,6 +74,10 @@ crun_version: 1.3
 runc_version: v1.0.3
 kata_containers_version: 2.2.3
 gvisor_version: 20210921
+containerd_version: 1.5.8
+
+# this is relevant when container_manager == 'docker'
+docker_containerd_version: 1.4.12

 # gcr and kubernetes image repo define
 gcr_image_repo: "gcr.io"
@ -140,14 +163,23 @@ crictl_checksums:
 # Kubernetes versions above Kubespray's current target version are untested and should be used with caution.
 kubelet_checksums:
  arm:
+    v1.23.5: 9505cf63fb56a1d90d1db9c1507587621455a152ef16d871e802875e1e7b4587
+    v1.23.4: e67a51013ed59ea3df0ad1d54863d483cc99247584992b8cad6dd612135a70c5
+    v1.23.2: f9e83b3bd99b9e70cd98a5f8dc75a89d3d51548d51e4e05615cdc48d6144f908
    v1.23.1: 29868f172ef171ae990deafcdc13af7fe5b00f0a546ae81c267c4ad01231c3ce
    v1.23.0: 7417fc7cd624a85887f0a28054f58f7534143579fe85285d0b68c8984c95f2ba
+    v1.22.8: 7bc14bfca0efb5af6d7e56218f5c51862596cd9927843f8456a36e70e8e64da8
+    v1.22.7: 3709a794b33081b3f5f5ff1c6f9ab1614c3723d1da0a31c74c37ccdec456e94f
+    v1.22.6: 9957157375a343664db35be75281d610df85e1986a98cc3db1398bd0e53c36f4
    v1.22.5: d901629aa537a0bff0907557810adb3cdc4a31f58035c57a45be011d836e2c8f
    v1.22.4: 725a89d9752bbce91443b20108f784ae5efc950db26abb618eb4d0a2364b2ada
    v1.22.3: 1c08dbe0b90d5b03fa386fadf5fa1af4db8e41bf8fa77888d54119ff188d130d
    v1.22.2: 941e639b0f859eba65df0c66be82808ea6be697ed5dbf4df8e602dcbfa683aa3
    v1.22.1: f42bc00f274be7ce0578b359cbccc48ead03894b599f5bf4d10e44c305fbab65
    v1.22.0: 4354dc8db1d8ca336eb940dd73adcd3cf17cbdefbf11889602420f6ee9c6c4bb
+    v1.21.11: 37fd7e30e532a51d6eef48b29a3b6a52f47827741f48458e1a7285e9ea9f88e9
+    v1.21.10: 5ab184ab71032662e68c3a933ce10fac59c0b02ff48db40b5438e617367d4689
+    v1.21.9: c359937e497184577efda5c850874af92ab626331da0cbf208d087e3f03b6114
    v1.21.8: 4f2db85f6151b985ccadbb18f5667b7002476139bc606da15d7adcd0f3996285
    v1.21.7: 0e8f96bc1d8977c99d5a04a089008af3bbeb38b7d4730fbe81bed479b2d1ba57
    v1.21.6: 20571caa4edcab5c17c448099cff74f0c0c54087c91888a23fc59407b8836127
@ -157,6 +189,7 @@ kubelet_checksums:
    v1.21.2: 577e09db9e8c11a57eeaa060fddc907df2b026b5270768201adcaafd9c6aa7b7
    v1.21.1: 2d2d17654a4abf66307c81c513228f29719aa02f5d5855dbd67a1337ab47c804
    v1.21.0: c2a3c926842c892ca3a124fb0526970e6ce1d92cb3ac35028b251a5672bb1af1
+    v1.20.15: 72520d4eda38b984e643fc87b89dbcf7ef5f8e4b7f2a0379d92c3d95751c245a
    v1.20.14: e0ed41fe013ac237e9e52e7210b959fc94208b6deff011f855a88f8edfa8b0b5
    v1.20.13: 67ca239729c2e1d739fa4aac9830c9769bcb000dbdce6937b37aa44339110d47
    v1.20.12: ee3220389c4ef66b0e4408ee9aa4517068794b76f15a64524449b6ba9829dc92
@ -173,14 +206,23 @@ kubelet_checksums:
    v1.20.1: 576d89284c8fa758d7d34d789444e9a5a03f24b6400cc94bf103c1c796898dc7
    v1.20.0: 132ea2f821ec1537af34b3a1414071416149972ce93619f71730c8dd741e357c
  arm64:
+    v1.23.5: 61f7e3ae0eb00633d3b5163c046cfcae7e73b5f26d4ffcf343f3a45904323583
+    v1.23.4: c4f09c9031a34549fbaa48231b115fee6e170ce6832dce26d4b50b040aad2311
+    v1.23.2: 65372ad077a660dfb8a863432c8a22cd0b650122ca98ce2e11f51a536449339f
    v1.23.1: c24e4ab211507a39141d227595610383f7c5686cae3795b7d75eebbce8606f3d
    v1.23.0: a546fb7ccce69c4163e4a0b19a31f30ea039b4e4560c23fd6e3016e2b2dfd0d9
+    v1.22.8: 604c672908a3b3cbbcf9d109d8d5fef0879992ddcf0d3e0766079d3bb7d0ca3e
+    v1.22.7: 8291d304c0ba4faec4336336d4cdd5159f5c90652b8b0d6be0cb5ce8f8bf92e3
+    v1.22.6: fbb823fe82b16c6f37911e907d3e4921f4642d5d48eb60e56aba1d7be0665430
    v1.22.5: e68536cff9172d1562edddd7194d20302472a064009bf7c0ed8d79d030cb61aa
    v1.22.4: c0049ab240b27a9dd57be2bb98356c62582d975ba2f790a61b34f155b12ab7e6
    v1.22.3: d0570f09bd5137ff2f672a0b177a6b78fd294a42db21f094dc02c613436ce8d1
    v1.22.2: f5fe3d6f4b2df5a794ebf325dc17fcdfe905a188e25f7c7e47d9cd15f14f8c2d
    v1.22.1: d5ffd67d8285fb224a1c49622fd739131f7b941e3d68f233dec96e72c9ebee63
    v1.22.0: cea637a7da4f1097b16b0195005351c07032a820a3d64c3ff326b9097cfac930
+    v1.21.11: ec0df7cf90f3422d674f9881e33d6e329a12e0f5bb438b422999493fd4370edf
+    v1.21.10: 5278427751381b90299e4ef330f41ca6b691aab39c3100cd200344ce6a7481c9
+    v1.21.9: 8797c78961cb71a757f35714d2735bb8bdbea94fc13d567bc0f1cf4f8e49e880
    v1.21.8: 1d880cd437457b6a52c95fa5cfb62f05bdcea8fc29b87aaa5535a67c89a279d4
    v1.21.7: 02adf21a8de206cf64c4bff5723adb08377ecdcc38ff1efbfefd3abe2e415bb8
    v1.21.6: 041441623c31bc6b0295342b8a2a5930d87545473e7c761ea79f3ff186c0ff52
@ -190,6 +232,7 @@ kubelet_checksums:
    v1.21.2: 525cf5506595e70bffc4c1845b3c535c7121fa2ee3daac6ca3edc69d8d63b89f
    v1.21.1: 5b37d7fc2da65a25896447685166769333b5896488de21bc9667edb4e799905e
    v1.21.0: 17832b192be5ea314714f7e16efd5e5f65347974bbbf41def6b02f68931380c4
+    v1.20.15: 8e3f08018fb66ee8b01d468779dc88608ce86f7a8bada85b5c5ddaae0280aeca
    v1.20.14: 2e4a857c675fea138efb45a91ac382c1067f5f903b73c1b0cfdb7a91f9429465
    v1.20.13: 065d1a0ab7909b77f829661d41da6bccef6116dc28becf3a845971915d70693d
    v1.20.12: e52854ee593b542ae8ad5cf2e29383d14369dffafacec979f76e6ad3bf514093
@ -206,14 +249,23 @@ kubelet_checksums:
    v1.20.1: 359025e549ec6e16ffa1aead7312934233a983a918f54686a2c34ff9a6ba6016
    v1.20.0: 47ab6c4273fc3bb0cb8ec9517271d915890c5a6b0e54b2991e7a8fbbe77b06e4
  amd64:
+    v1.23.5: 253b9db2299b09b91e4c09781ce1d2db6bad2099cf16ba210245159f48d0d5e4
+    v1.23.4: ec3db57edcce219c24ef37f4a6a2eef5a1543e4a9bd15e7ecc993b9f74950d91
+    v1.23.2: c3c4be17910935d234b776288461baf7a9c6a7414d1f1ac2ef8d3a1af4e41ab6
    v1.23.1: 7ff47abf62096a41005d18c6d482cf73f26b613854173327fa9f2b98720804d4
    v1.23.0: 4756ff345dd80704b749d87efb8eb294a143a1f4a251ec586197d26ad20ea518
+    v1.22.8: 2e6d1774f18c4d4527c3b9197a64ea5705edcf1b547c77b3e683458d771f3ce7
+    v1.22.7: cfc96b5f781bfbfdcb05115f4e26a5a6afc9d74bb4a5647c057b2c13086fb24d
+    v1.22.6: 7b009835b0ab74aa16ebf57f5179893035e0cf5994e1bcf9b783275921a0393a
    v1.22.5: 2be340f236a25881969eaa7d58b2279a4e31dc393cab289a74c78c0c37ba2154
    v1.22.4: 8d014cfe511d8c0a127b4e65ae2a6e60db592f9b1b512bb822490ea35958b10d
    v1.22.3: 3f00a5f98cec024abace5bcc3580b80afc78181caf52e100fc800e588774d6eb
    v1.22.2: 0fd6572e24e3bebbfd6b2a7cb7adced41dad4a828ef324a83f04b46378a8cb24
    v1.22.1: 2079780ad2ff993affc9b8e1a378bf5ee759bf87fdc446e6a892a0bbd7353683
    v1.22.0: fec5c596f7f815f17f5d7d955e9707df1ef02a2ca5e788b223651f83376feb7f
+    v1.21.11: ea22e3683016643344c5839a317b5e7b0061fdded321339a6d545766765bb10a
+    v1.21.10: 8e0dab1cb93e61771fba594484a37a6079073ed2d707cf300c472e79b2f91bf0
+    v1.21.9: 1fa0c296df6af71fca1bdd94f9fb19c7051b4b3f8cf19c353192cb96b413fcf2
    v1.21.8: 32f7eb6af9f1fd4e8b944f4f59582d455572147745e9fc04d044c383bd995c98
    v1.21.7: 59f8d7da2e994f59a369ea1705e4933949fc142bf47693e0918f4811c2e1c7b5
    v1.21.6: 422c29a1ba3bfeb2fc26ebd1c3596847fbbeeeef0ce2694515504513dc907813
@ -223,6 +275,7 @@ kubelet_checksums:
    v1.21.2: aaf144b19c0676e1fe34a93dc753fb38f4de057a0e2d7521b0bef4e82f8ccc28
    v1.21.1: e77ff3ea404b2e69519ea4dce41cbdf11ae2bcba75a86d409a76eecda1c76244
    v1.21.0: 681c81b7934ae2bf38b9f12d891683972d1fbbf6d7d97e50940a47b139d41b35
+    v1.20.15: 137b40720623588ad2e8feee9f73160c86298b328eeba28e207778806ca03614
    v1.20.14: 01342d70251276346aa5afa9653b49391959d0e4e32c905e1c66244f74b7af1f
    v1.20.13: 4f6a11f84e9b4af4d20a40e2dc6a5fcbae77479d6745fc95ba16d8f4596a15e5
    v1.20.12: a53cae386e08efce25a9ee228c916e80b86335691f327a0ee8bc76ac56adeca2
@ -240,14 +293,23 @@ kubelet_checksums:
    v1.20.0: ff2422571c4c1e9696e367f5f25466b96fb6e501f28aed29f414b1524a52dea0
 kubectl_checksums:
  arm:
+    v1.23.5: 58420bc549e1683a4529066b38b2ac657611ed3b70041be78fba3b29401415db
+    v1.23.4: bde3d7801cfe444d4e226d4669dfd518e4687e16c99efddd016c4bf3d529b198
+    v1.23.2: 6521719af33342f00ebb6cf020848e25152a63ed5f35a94440c08373b7a36173
    v1.23.1: 52001ed48e9e1c8b8623f3e6b0242111227721e5ddd08fa18046c65c406e35a5
    v1.23.0: 6152216d88fa4d32da58c67f78b63b3b99bf4d4d726ffb9fb74ea698dccc8644
+    v1.22.8: 08ffeb8924c315cd466fc930377ac545edd6ac4ebb8bf284218947256b6729f3
+    v1.22.7: be9a833a6eae7ee7698ee5cc18bacc2652207af07528e60a78f43a8139fffbfc
+    v1.22.6: a0dea833198a95ec85b4d55fe7e16333bcdc6a93290238c7473887e7e06f23a7
    v1.22.5: 6db514e45f62f611d7e5f862c1eec6009e6de07852cf3cbc37309db1ed76920f
    v1.22.4: 9ea171e868aeb64b187a039edd79b2c7ea2bedbd752c76e1c5e44c2486d21f72
    v1.22.3: 28e2817751c94940469755911fe3d6a93e288391377f5bb8db08cffa538e72fa
    v1.22.2: a16f7d70e65589d2dbd5d4f2115f6ccd4f089fe17a2961c286b809ad94eb052a
    v1.22.1: 50991ec4313ee42da03d60e21b90bc15e3252c97db189d1b66aad5bbb555997b
    v1.22.0: 6d7c787416a148acffd49746837df4cebb1311c652483dc3d2c8d24ce1cc897e
+    v1.21.11: 16e0065ac097d42cf791ac0be297c1a86ef48c72ba2a32748ac2c7ad51a58175
+    v1.21.10: faefbe444bd78cf217ef1c72abce816f335e06a1e08afa8a1d9530ba04d24ee9
+    v1.21.9: 543f0425d76c71d3d3ac2af8eaa7ca7dd6aa1919f01312303f328b6d805f3e20
    v1.21.8: e501c517dbe041a10a56ef7e0c2a041c194ff2e1f2c3c993966211d486a9b3d0
    v1.21.7: 9c0f7af2c7e4217ccd389d72c212880d50b9d46fe94f83efb8e33e51112be086
    v1.21.6: 9100bc13498f770a5a1524665a9dc2470d3a15518e53aba68c700f10f3def978
@ -257,6 +319,7 @@ kubectl_checksums:
    v1.21.2: 898c2cd54b651873a8fb18bcb0792eb4772a78f845d758fa9b0eee278aede869
    v1.21.1: d963971fd796b04ccaa389cf59900834e01c151a52c531585ac68aae779c0d91
    v1.21.0: 6d79f6ebec2eda45b0808a895fa5d06bd8611c02d34b584eaa94857f365b25aa
+    v1.20.15: 9c32db99c544a4e3d11172ff11d166cc61cf839b9d8280cdbd1c0d2dd8cfe1b8
    v1.20.14: 30a0b11050e227006d5f5cf48a7db29da004faa6b3f20c8e0bf398a017266475
    v1.20.13: 2026bee7860bf299b39cf650a0c7564306063f0529570e78a8762f09751826ea
    v1.20.12: f0999194f5d7e33e0e4ae6fa929ecaf62ab6d23906e2f9742149dcfb9de52a63
@ -273,14 +336,23 @@ kubectl_checksums:
    v1.20.1: f164e6efd49a6c4a278d1317f00ee1691e7cf8531a49d10ea069f5180d3540ed
    v1.20.0: bec2b30e37056ae354ec5a86d77634a9b832ade8d9aeea7a4514cee21cb7821e
  arm64:
+    v1.23.5: 15cd560c04def7bbe5ee3f6f75e2cfd3913371c7e76354f4b2d5d6f536b70e39
+    v1.23.4: aa45dba48791eeb78a994a2723c462d155af4e39fdcfbcb39ce9c96f604a967a
+    v1.23.2: 6e7bb8ddc5fc8fa89a4c31aba02942718b092a5107585bd09a83c95039c7510b
    v1.23.1: c0c24c7f6a974390e15148a575c84878e925f32328ff96ae173ec762678e4524
    v1.23.0: 1d77d6027fc8dfed772609ad9bd68f611b7e4ce73afa949f27084ad3a92b15fe
+    v1.22.8: 48105735b74e941a84dec6bd53637c023ad53dc5fadd9bf616347cb339c76b47
+    v1.22.7: 44342131947bc61e6b03103e7e1302d16fa3e5b2e2cd67e27194f66223ecf798
+    v1.22.6: b43199fe66a58f292f2c685b922330819190eb22ac41cc5c10c33fdf9f2bbc29
    v1.22.5: a122ef299d75c0bec1dc1e28670d358e13743144e68223c8178462ba5c436e1d
    v1.22.4: 3fcec0284c0fdfc22e89a5b73ebd7f51120cc3505a11a4f6d6f819d46a40b26a
    v1.22.3: ebeac516cc073cfe9550f114ca326f762d958cb91a33c8c9d03ede6ba94a6088
    v1.22.2: c5bcc7e5321d34ac42c4635ad4f6fe8bd4698e9c879dc3367be542a0b301297b
    v1.22.1: 5c7ef1e505c35a8dc0b708f6b6ecdad6723875bb85554e9f9c3fe591e030ae5c
    v1.22.0: 8d9cc92dcc942f5ea2b2fc93c4934875d9e0e8ddecbde24c7d4c4e092cfc7afc
+    v1.21.11: 2d51a37128d823520f5f2b70436f5e3ae426eeacd16d671ae7806d421e4f57d8
+    v1.21.10: d0a88f897824954ec104895eae5f9ff9a173b162d1c9245c274cfe8db323fb37
+    v1.21.9: 6e2893b5de590fd9587ba327c048e5318e9e12e2acdc5a83c995c57ae822e6e4
    v1.21.8: ec122a1c239798c8a233377113b71bed808191dd931137f0631faa2d91fddb2a
    v1.21.7: 50e5d76831af7b83228a5191ae10313c33639d03fadd89ad3cd492d280be4f88
    v1.21.6: a193997181cdfa00be0420ac6e7f4cfbf6cedd6967259c5fda1d558fa9f4efe0
@ -290,6 +362,7 @@ kubectl_checksums:
    v1.21.2: 5753051ed464d0f1af05a3ca351577ba5680a332d5b2fa7738f287c8a40d81cf
    v1.21.1: d7e1163f4127efd841e5f5db6eacced11c2a3b20384457341b19ca295d0c535f
    v1.21.0: a4dd7100f547a40d3e2f83850d0bab75c6ea5eb553f0a80adcf73155bef1fd0d
+    v1.20.15: d479febfb2e967bd86240b5c0b841e40e39e1ef610afd6f224281a23318c13dc
    v1.20.14: bfafaa9208b6c2e5806971e0b4fd9e98ba22940bfb42072b34426bc151125e9d
    v1.20.13: e58610d292120d0d8ebb098f2ef9d15a63dbedbf2be36e820e48eca97bf836f9
    v1.20.12: 2a315fcee435ae61a2d75f5d37051735dc0ee6a1abfe4ff032655de7bdee4adb
@ -306,14 +379,23 @@ kubectl_checksums:
    v1.20.1: 1ed8762306d4d09574150ffc666a04e5a79ca08e53570cd34977fab4fc7c4611
    v1.20.0: 25e4465870c99167e6c466623ed8f05a1d20fbcb48cab6688109389b52d87623
  amd64:
+    v1.23.5: 715da05c56aa4f8df09cb1f9d96a2aa2c33a1232f6fd195e3ffce6e98a50a879
+    v1.23.4: 3f0398d4c8a5ff633e09abd0764ed3b9091fafbe3044970108794b02731c72d6
+    v1.23.2: 5b55b58205acbafa7f4e3fc69d9ce5a9257be63455db318e24db4ab5d651cbde
    v1.23.1: 156fd5e7ebbedf3c482fd274089ad75a448b04cf42bc53f370e4e4ea628f705e
    v1.23.0: 2d0f5ba6faa787878b642c151ccb2c3390ce4c1e6c8e2b59568b3869ba407c4f
+    v1.22.8: 761bf1f648056eeef753f84c8365afe4305795c5f605cd9be6a715483fe7ca6b
+    v1.22.7: 4dd14c5b61f112b73a5c9c844011a7887c4ffd6b91167ca76b67197dee54d388
+    v1.22.6: 1ab07643807a45e2917072f7ba5f11140b40f19675981b199b810552d6af5c53
    v1.22.5: fcb54488199c5340ff1bc0e8641d0adacb27bb18d87d0899a45ddbcc45468611
    v1.22.4: 21f24aa723002353eba1cc2668d0be22651f9063f444fd01626dce2b6e1c568c
    v1.22.3: 0751808ca8d7daba56bf76b08848ef5df6b887e9d7e8a9030dd3711080e37b54
    v1.22.2: aeca0018958c1cae0bf2f36f566315e52f87bdab38b440df349cd091e9f13f36
    v1.22.1: 78178a8337fc6c76780f60541fca7199f0f1a2e9c41806bded280a4a5ef665c9
    v1.22.0: 703e70d49b82271535bc66bc7bd469a58c11d47f188889bd37101c9772f14fa1
+    v1.21.11: 9c45ce24ad412701beeac8d9f0004787209d76dd66390915f38a8682358484cb
+    v1.21.10: 24ce60269b1ffe1ca151af8bfd3905c2427ebef620bc9286484121adf29131c0
+    v1.21.9: 195d5387f2a6ca7b8ab5c2134b4b6cc27f29372f54b771947ba7c18ee983fbe6
    v1.21.8: 84eaef3da0b508666e58917ebe9a6b32dcc6367bddf6e4489b909451877e3e70
    v1.21.7: d25d6b6f67456cc059680e7443c424eb613d9e840850a7be5195cff73fed41b8
    v1.21.6: 810eadc2673e0fab7044f88904853e8f3f58a4134867370bf0ccd62c19889eaa
@ -323,6 +405,7 @@ kubectl_checksums:
    v1.21.2: 55b982527d76934c2f119e70bf0d69831d3af4985f72bb87cd4924b1c7d528da
    v1.21.1: 58785190e2b4fc6891e01108e41f9ba5db26e04cebb7c1ac639919a931ce9233
    v1.21.0: 9f74f2fa7ee32ad07e17211725992248470310ca1988214518806b39b1dad9f0
+    v1.20.15: d283552d3ef3b0fd47c08953414e1e73897a1b3f88c8a520bb2e7de4e37e96f3
    v1.20.14: 7b582833331dc98a20c01e27006c2949fe7141bb197d20d824ac7efe85230ea3
    v1.20.13: 0ea0e50382bb16a134b812caba15f7b9bb805ca21a593c8958a94aa0a09509cd
    v1.20.12: 1e1cee1294d6b2f5cae547123df729ca38a07e8c0c1c2b782fb5f31a6541f89b
@ -340,14 +423,23 @@ kubectl_checksums:
    v1.20.0: a5895007f331f08d2e082eb12458764949559f30bcc5beae26c38f3e2724262c
 kubeadm_checksums:
  arm:
+    v1.23.5: 9ea3e52cb236f446a33cf69e4ed6ac28a76103c1e351b2675cb9bfcb77222a61
+    v1.23.4: 9ca72cf1e6bbbe91bf634a18571c84f3fc36ba5fcd0526b14432e87b7262a5ee
+    v1.23.2: 63a6ca7dca76475ddef84e4ff84ef058ee2003d0e453b85a52729094025d158e
    v1.23.1: 77baac1659f7f474ba066ef8ca67a86accc4e40d117e73c6c76a2e62689d8369
    v1.23.0: b59790cdce297ac0937cc9ce0599979c40bc03601642b467707014686998dbda
+    v1.22.8: f55fce83ae69b0f660a0fbdd2d05681d2e29a1119d7cce890fe1f50724bdcc60
+    v1.22.7: 26b3d79d88e81bf354d716fa48210b0358d2f6ca99cba06eb7640ac1e32724b8
+    v1.22.6: ad23ad06e83f2466f78652221f73fd58d23d6122b3395c24d9a3be779f6afa49
    v1.22.5: f0c95c9b86287ec8570388f8fc26ad05ac342f69876a08cb6cb5aa2ffcc1febd
    v1.22.4: 5e52ee3c3f0f5bffd9f0d9e7b3e215b5ab239feb425d47d8bd609bd4b1fb1d61
    v1.22.3: d3c76311c582e48889bdb3e3ef1948ce0292983a0c13d37c7e8ae5c6024291f5
    v1.22.2: 6ccc26494160e19468b0cb55d56b2d5c62d21424fac79cb66402224c2bf73a0d
    v1.22.1: cc08281c5261e860df9a0b5040b8aa2e6d202a243daf25556f5f6d3fd8f2e1e9
    v1.22.0: 6a002deb0ee191001d5c0e0435e9a995d70aa376d55075c5f61e70ce198433b8
+    v1.21.11: 4c59ecfaac03987a839310fe69867021225d06ebf62b04929ae3e13e6dcfa6b3
+    v1.21.10: ba6fdf511ce4521b89d2674d31014a64da78f351763adbbb32ae13b19535f0be
+    v1.21.9: 855e5dff65343245136eb13c0ce89f830a5a22269536834cf50fdb2e47789ad5
    v1.21.8: 1318ef6e4ef7203d08f66d39a2367122d0d4b92b81da0cc0bde87c94a56746d4
    v1.21.7: d72e2d91994647839b7982c04f30ac89c953719c180a0cccb607ffae36bde7e6
    v1.21.6: 02951dae946dd5588ccda71b6e28f0d91adf7a94b57792b412635fcce7099d74
@ -357,6 +449,7 @@ kubeadm_checksums:
    v1.21.2: 75251be6394ca3fedf578a1ba3bc7d7f01e60402b83658ccf2439d1c48379846
    v1.21.1: d2a6b582ae5407f2dcd3da902060cadbe5212577ffc9f546245e0d83d4490582
    v1.21.0: 878e4c848412c3d6ac35323e6be1bd09f9648069ea1caa8e618fd2a6a6925b2f
+    v1.20.15: 6e7159ad50d4d2be0f764d6b2c4af051d0ce14aedf7299eb1ba1d327b24a2882
    v1.20.14: 08459769f4e76d4ce0e1f9cda7573e45f817ab4d2bd40933a7ef3636e60b2747
    v1.20.13: 82fa0db6663656ebe97810c843f5206280345353eabb82a5cd72c3bf4e629e46
    v1.20.12: ede10f9cd12a1f718f7b9d3915bfe0d34e7e54d3cd08e04e49643e2eefaecb36
@ -373,14 +466,23 @@ kubeadm_checksums:
    v1.20.1: 0e63af77d148e64cd1710f98d56967d5490df6a978658f6d5e2cd6b9cbf3c61a
    v1.20.0: 38829599cfcab3274e9cf1aff942db06d7ec091510399722159fa3fa49a6fef0
  arm64:
+    v1.23.5: 22a8468abc5d45b3415d694ad52cc8099114248c3d1fcf4297ec2b336f5cc274
+    v1.23.4: 90fd5101e321053cdb66d165879a9cde18f19ba9bb8eae152fd4f4fcbe497be1
+    v1.23.2: a29fcde7f92e1abfe992e99f415d3aee0fa381478b4a3987e333438b5380ddff
    v1.23.1: eb865da197f4595dec21e6fb1fa1751ef25ac66b64fa77fd4411bbee33352a40
    v1.23.0: 989d117128dcaa923b2c7a917a03f4836c1b023fe1ee723541e0e39b068b93a6
+    v1.22.8: 67f09853d10434347eb75dbb9c63d57011ba3e4f7e1b320a0c30612b8185be8c
+    v1.22.7: 2ae0287769a70f442757e49af0ecd9ca2c6e5748e8ba72cb822d669a7aeeb8fa
+    v1.22.6: bc10e4fb42a182515f4232205bea53f90270b8f80ec1a6c1cc3301bff05e86b7
    v1.22.5: 47aa54533289277ac13419c16ffd1a2c35c7af2d6a571261e3d728990bc5fc7d
    v1.22.4: 3dfb128e108a3f07c53cae777026f529784a057628c721062d8fdd94b6870b69
    v1.22.3: dcd1ecfb7f51fb3929b9c63a984b00cf6baa6136e1d58f943ee2c9a47af5875d
    v1.22.2: 77b4c6a56ae0ec142f54a6f5044a7167cdd7193612b04b77bf433ffe1d1918ef
    v1.22.1: 85df7978b2e5bb78064ed0bcce14a39d105a1a3968bb92ee5d2f96a1fa09ed12
    v1.22.0: 9fc14b993de2c275b54445255d7770bd1d6cdb49f4cf9c227c5b035f658a2351
+    v1.21.11: 97117a6d984ff88628654494181b62502cbf4c310af70d4de92dab35482900e5
+    v1.21.10: 7607bfd40317a24a276e452b46a26a7298dde2988fce826f1ee0fe9355eae786
+    v1.21.9: 8947309c985911a99fb0a6e30f9ca85d9b7adc1215149e45e5be150c7e5e5de9
    v1.21.8: abf2d57cb42e8dfbcb3632dd278991bcf422891cc91e3967e00f7f45183bb43e
    v1.21.7: d2d17f37f1e4de446cf75f60a2a6f7fba3cbc8e27a1d176cfa0fa48862fad4bc
    v1.21.6: 498325da2521ce67b27902967daf4087153c5797070e03bf0bdd7c846f4d61a8
@ -390,6 +492,7 @@ kubeadm_checksums:
    v1.21.2: 245125dc436f649466123a2d2c922d17f300cbc20d2b75edad5e42d734ead4a3
    v1.21.1: 1c9a93ac74f2756c1eb40a9d18bb7e146eeab0b33177c0f66f5e617ed7261d1b
    v1.21.0: 50bb95d1827455346b5643dcf83a52520733c3a582b8b1ffb50f04a8e66f00e7
+    v1.20.15: 76e132da8185a7c33e6178348d05570a1f78878d5e118c7402c9ad02e8b6fb77
    v1.20.14: 08c5da652f3be528fb1a97200d962cc209f3335c0e750206765aef22bedae70f
    v1.20.13: cae4663ef8242f04f134acd25d27b26455e6dc4e163e69495f0eaf5c6d797c1e
    v1.20.12: 8e6fa3e3e0d18170bf7d9e4a457c353989a8eac89c0a398457ba9576363dfed7
@ -406,14 +509,23 @@ kubeadm_checksums:
    v1.20.1: 91eb63944e9a033bb40dfc638913ca76e07d6877a24d25636bf833459ae90010
    v1.20.0: 16faf8bf3c94cb21dcae131cbfbd0961fc5fef37cb7e37f872ff249e235e0f46
  amd64:
+    v1.23.5: 8eebded187ee84c97003074eaa347e34131fef3acdf3e589a9b0200f94687667
+    v1.23.4: c91912c9fd34a50492f889e08ff94c447fdceff150b588016fecc9051a1e56b8
+    v1.23.2: 58487391ec37489bb32fe532e367995e9ecaeafdb65c2113ff3675e7a8407219
    v1.23.1: 4d5766cb90050ee84e15df5e09148072da2829492fdb324521c4fa6d74d3aa34
    v1.23.0: e21269a058d4ad421cf5818d4c7825991b8ba51cd06286932a33b21293b071b0
+    v1.22.8: fc10b4e5b66c9bfa6dc297bbb4a93f58051a6069c969905ef23c19680d8d49dc
+    v1.22.7: 7e4be37fc5ddeeae732886bf83c374198813e76d84ed2f6590145e08ece1a8b2
+    v1.22.6: 0bf8e47ad91215cd8c5e0ded565645aeb1ad6f0a9223a2486eb913bff929d472
    v1.22.5: a512be0fa429f43d3457472efd73529cd2ba2cd54ef714faf6b69486beea054f
    v1.22.4: 33b799df2941f12a53ffe995d86a385c35d3c543f9d2c00c0cdb47ec91a98c5c
    v1.22.3: 3964e6fd46052eb4a9672421d8e8ce133b83b45abb77481b688dc6375390e480
    v1.22.2: 4ff09d3cd2118ee2670bc96ed034620a9a1ea6a69ef38804363d4710a2f90d8c
    v1.22.1: 50a5f0d186d7aefae309539e9cc7d530ef1a9b45ce690801655c2bee722d978c
    v1.22.0: 90a48b92a57ff6aef63ff409e2feda0713ca926b2cd243fe7e88a84c483456cc
+    v1.21.11: 3514ea5acaae9c2779a341deb24832df17722cb612fa7a78d34f602f91e94d17
+    v1.21.10: 61aaadd98806d979b65e031a144d9379390d26ccb5383d47bdd8b7c727e94a7b
+    v1.21.9: 3333116f9f0d72e0598f52dcbef7ecab1ce88192fdcfd5384ca919fdc075e8d5
    v1.21.8: 51d266e91e2aec0e994c046b4d80901a1b1e7be05e30b83461f0563571f1224d
    v1.21.7: c4480121b629a0f563f718aa11440ae26a569e37e0229c093a5785c90725a03c
    v1.21.6: fef4b40acd982da99294be07932eabedd476113ce5dc38bb9149522e32dada6d
@ -423,6 +535,7 @@ kubeadm_checksums:
    v1.21.2: 6a83e52e51f41d67658a13ce8ac9deb77a6d82a71ced2d106756f6d38756ec00
    v1.21.1: 1553c07a6a777c4cf71d45d5892915f0ea6586b8a80f9fea39e7a659d6315d42
    v1.21.0: 7bdaf0d58f0d286538376bc40b50d7e3ab60a3fe7a0709194f53f1605129550f
+    v1.20.15: e6469323cfea76a8b7d1031d00e9c19af7a763df32da013786624f24507ea8a2
    v1.20.14: 84d29c334dc67b5ca6ed14b547cd23618d7afb1698b916357bf534906bb9f58b
    v1.20.13: 63bd2d0ff5df371941503b8858e61222a799f3a18eefad1802bbeba42b668a36
    v1.20.12: 992f29f17c3775d577035e5d0c3292652cb57c00fb606f96bf39e7441884568b
@ -557,6 +670,7 @@ kata_containers_binary_checksums:
    2.0.4: 0
    2.1.1: 0
    2.2.2: 0
+    2.2.3: 0
    2.3.0: 0
  amd64:
    2.0.4: 022a60c2d92a5ab9a5eb83d5a95154a2d06fdc2206b2a473d902ccc86766371a
@ -568,6 +682,7 @@ kata_containers_binary_checksums:
    2.0.4: 0
    2.1.1: 0
    2.2.2: 0
+    2.2.3: 0
    2.3.0: 0

 gvisor_runsc_binary_checksums:
--- a/roles/download/tasks/prep_download.yml
+++ b/roles/download/tasks/prep_download.yml
@ -5,44 +5,6 @@
  tags:
    - facts

-# The docker image_info_command might seems weird but we are using raw/endraw and `{{ `{{` }}` to manage the double jinja2 processing
-# done here and when `image_info_command` is used (first the raw/endraw allow to store the command, then the second processing replace `{{`
- name: prep_download | Set image pull/info command for docker
-  set_fact:
-    image_pull_command: "{{ docker_bin_dir }}/docker pull"
-    image_info_command: "{{ docker_bin_dir }}/docker images -q | xargs -i {{ '{{' }} docker_bin_dir }}/docker inspect -f {% raw %}'{{ '{{' }} if .RepoTags }}{{ '{{' }} join .RepoTags \",\" }}{{ '{{' }} end }}{{ '{{' }} if .RepoDigests }},{{ '{{' }} join .RepoDigests \",\" }}{{ '{{' }} end }}' {% endraw %} {} | tr '\n' ','"
-  when: container_manager == 'docker'
-
- name: prep_download | Set image pull/info command for containerd
-  set_fact:
-    image_info_command: "{{ bin_dir }}/nerdctl -n k8s.io images --format '{% raw %}{{ '{{' }} .Repository {{ '}}' }}:{{ '{{' }} .Tag {{ '}}' }}{% endraw %}' 2>/dev/null | grep -v ^:$ | tr '\n' ','"
-    image_pull_command: "{{ bin_dir }}/nerdctl -n k8s.io pull --quiet"
-  when: container_manager == 'containerd'
-
- name: prep_download | Set image pull/info command for crio
-  set_fact:
-    image_info_command: "{{ bin_dir }}/crictl images --verbose | awk -F ': ' '/RepoTags|RepoDigests/ {print $2}' | tr '\n' ','"
-    image_pull_command: "{{ bin_dir }}/crictl pull"
-  when: container_manager == 'crio'
-
- name: prep_download | Set image pull/info command for docker on localhost
-  set_fact:
-    image_pull_command_on_localhost: "{{ docker_bin_dir }}/docker pull"
-    image_info_command_on_localhost: "{{ docker_bin_dir }}/docker images"
-  when: container_manager_on_localhost == 'docker'
-
- name: prep_download | Set image pull/info command for containerd on localhost
-  set_fact:
-    image_info_command_on_localhost: "{{ bin_dir }}/nerdctl -n k8s.io images --format '{% raw %}{{ '{{' }} .Repository {{ '}}' }}:{{ '{{' }} .Tag {{ '}}' }}{% endraw %}' 2>/dev/null | grep -v ^:$ | tr '\n' ','"
-    image_pull_command_on_localhost: "{{ bin_dir }}/nerdctl -n k8s.io pull --quiet"
-  when: container_manager_on_localhost == 'containerd'
-
- name: prep_download | Set image pull/info command for crio on localhost
-  set_fact:
-    image_info_command_on_localhost: "{{ bin_dir }}/crictl images --verbose | awk -F ': ' '/RepoTags|RepoDigests/ {print $2}' | tr '\n' ','"
-    image_pull_command_on_localhost: "{{ bin_dir }}/crictl pull"
-  when: container_manager_on_localhost == 'crio'
-
 - name: prep_download | On localhost, check if passwordless root is possible
  command: "true"
  delegate_to: localhost
--- a/roles/kubernetes-apps/ansible/defaults/main.yml
+++ b/roles/kubernetes-apps/ansible/defaults/main.yml
@ -24,6 +24,7 @@ nodelocaldns_secondary_prometheus_port: 9255
 dns_autoscaler_cpu_requests: 20m
 dns_autoscaler_memory_requests: 10Mi
 dns_autoscaler_deployment_nodeselector: "kubernetes.io/os: linux"
+# dns_autoscaler_extra_tolerations: [{effect: NoSchedule, operator: "Exists"}]

 # etcd metrics
 # etcd_metrics_service_labels:
@ -81,3 +82,6 @@ dashboard_master_toleration: true
 # Override dashboard default settings
 dashboard_token_ttl: 900
 dashboard_skip_login: false
+
+# Policy Controllers
+# policy_controller_extra_tolerations: [{effect: NoSchedule, operator: "Exists"}]
--- a/roles/kubernetes-apps/ansible/templates/coredns-deployment.yml.j2
+++ b/roles/kubernetes-apps/ansible/templates/coredns-deployment.yml.j2
@ -34,7 +34,7 @@ spec:
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          effect: NoSchedule
-{% if dns_extra_tolerations | default(None) %}
+{% if dns_extra_tolerations is defined %}
        {{ dns_extra_tolerations | list | to_nice_yaml(indent=2) | indent(8) }}
 {% endif %}
      affinity:
--- a/roles/kubernetes-apps/ansible/templates/dns-autoscaler.yml.j2
+++ b/roles/kubernetes-apps/ansible/templates/dns-autoscaler.yml.j2
@ -45,6 +45,9 @@ spec:
          key: node-role.kubernetes.io/master
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
+{% if dns_autoscaler_extra_tolerations is defined %}
+        {{ dns_autoscaler_extra_tolerations | list | to_nice_yaml(indent=2) | indent(8) }}
+{% endif %}
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
--- a/roles/kubernetes-apps/ingress_controller/cert_manager/defaults/main.yml
+++ b/roles/kubernetes-apps/ingress_controller/cert_manager/defaults/main.yml
@ -1,3 +1,7 @@
 ---
 cert_manager_namespace: "cert-manager"
 cert_manager_user: 1001
+
+## Change leader election namespace when deploying on GKE Autopilot that forbid the changes on kube-system namespace.
+## See https://github.com/jetstack/cert-manager/issues/3717
+cert_manager_leader_election_namespace: kube-system
--- a/roles/kubernetes-apps/ingress_controller/cert_manager/templates/cert-manager.yml.j2
+++ b/roles/kubernetes-apps/ingress_controller/cert_manager/templates/cert-manager.yml.j2
@ -630,7 +630,7 @@ apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
  name: cert-manager-cainjector:leaderelection
-  namespace: {{ cert_manager_namespace }}
+  namespace: {{ cert_manager_leader_election_namespace }}
  labels:
    app: cainjector
    app.kubernetes.io/name: cainjector
@ -664,7 +664,7 @@ apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
  name: cert-manager:leaderelection
-  namespace: {{ cert_manager_namespace }}
+  namespace: {{ cert_manager_leader_election_namespace }}
  labels:
    app: cert-manager
    app.kubernetes.io/name: cert-manager
@ -719,7 +719,7 @@ apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: cert-manager-cainjector:leaderelection
-  namespace: {{ cert_manager_namespace }}
+  namespace: {{ cert_manager_leader_election_namespace }}
  labels:
    app: cainjector
    app.kubernetes.io/name: cainjector
@ -742,7 +742,7 @@ apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
  name: cert-manager:leaderelection
-  namespace: {{ cert_manager_namespace }}
+  namespace: {{ cert_manager_leader_election_namespace }}
  labels:
    app: cert-manager
    app.kubernetes.io/name: cert-manager
@ -866,7 +866,7 @@ spec:
          imagePullPolicy: {{ k8s_image_pull_policy }}
          args:
          - --v=2
-          - --leader-election-namespace=kube-system
+          - --leader-election-namespace={{ cert_manager_leader_election_namespace }}
          env:
          - name: POD_NAMESPACE
            valueFrom:
@ -928,7 +928,7 @@ spec:
          args:
          - --v=2
          - --cluster-resource-namespace=$(POD_NAMESPACE)
-          - --leader-election-namespace=kube-system
+          - --leader-election-namespace={{ cert_manager_leader_election_namespace }}
          ports:
          - containerPort: 9402
            protocol: TCP
--- a/roles/kubernetes-apps/policy_controller/calico/templates/calico-kube-controllers.yml.j2
+++ b/roles/kubernetes-apps/policy_controller/calico/templates/calico-kube-controllers.yml.j2
@ -28,6 +28,9 @@ spec:
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          effect: NoSchedule
+{% if policy_controller_extra_tolerations is defined %}
+        {{ policy_controller_extra_tolerations | list | to_nice_yaml(indent=2) | indent(8) }}
+{% endif %}
      priorityClassName: system-cluster-critical
      containers:
        - name: calico-kube-controllers
--- a/roles/kubernetes/preinstall/tasks/0040-set_facts.yml
+++ b/roles/kubernetes/preinstall/tasks/0040-set_facts.yml
@ -182,7 +182,7 @@
    nameserverentries:
      nameserver {{ ( ( [nodelocaldns_ip] if enable_nodelocaldns else []) + coredns_server|d([]) + nameservers|d([]) + cloud_resolver|d([]) + configured_nameservers|d([])) | unique | join(',nameserver ') }}
    supersede_nameserver:
-      supersede domain-name-servers {{ ( coredns_server|d([]) + nameservers|d([]) + cloud_resolver|d([])) | unique | join(', ') }};
+      supersede domain-name-servers {{ ( ( [nodelocaldns_ip] if enable_nodelocaldns else []) + coredns_server|d([]) + nameservers|d([]) + cloud_resolver|d([])) | unique | join(', ') }};

 - name: gather os specific variables
  include_vars: "{{ item }}"
--- a/roles/kubernetes/preinstall/tasks/0080-system-configurations.yml
+++ b/roles/kubernetes/preinstall/tasks/0080-system-configurations.yml
@ -79,6 +79,24 @@
    reload: yes
  when: enable_dual_stack_networks | bool

+- name: Check if we need to set fs.may_detach_mounts
+  stat:
+    path: /proc/sys/fs/may_detach_mounts
+    get_attributes: no
+    get_checksum: no
+    get_mime: no
+  register: fs_may_detach_mounts
+  ignore_errors: true  # noqa ignore-errors
+
+- name: Set fs.may_detach_mounts if needed
+  sysctl:
+    sysctl_file: "{{ sysctl_file_path }}"
+    name: fs.may_detach_mounts
+    value: 1
+    state: present
+    reload: yes
+  when: fs_may_detach_mounts.stat.exists | d(false)
+
 - name: Ensure kube-bench parameters are set
  sysctl:
    sysctl_file: /etc/sysctl.d/bridge-nf-call.conf
--- a/roles/kubernetes/preinstall/tasks/0100-dhclient-hooks.yml
+++ b/roles/kubernetes/preinstall/tasks/0100-dhclient-hooks.yml
@ -13,7 +13,6 @@
    marker: "# Ansible entries {mark}"
    mode: 0644
  notify: Preinstall | propagate resolvconf to k8s components
-  when: dhclientconffile is defined

 - name: Configure dhclient hooks for resolv.conf (non-RH)
  template:
--- a/roles/kubernetes/preinstall/tasks/0110-dhclient-hooks-undo.yml
+++ b/roles/kubernetes/preinstall/tasks/0110-dhclient-hooks-undo.yml
@ -9,12 +9,10 @@
    state: absent
    backup: yes
    marker: "# Ansible entries {mark}"
-  when: dhclientconffile is defined
  notify: Preinstall | propagate resolvconf to k8s components

 - name: Remove kubespray specific dhclient hook
  file:
    path: "{{ dhclienthookfile }}"
    state: absent
-  when: dhclienthookfile is defined
  notify: Preinstall | propagate resolvconf to k8s components
--- a/roles/kubernetes/preinstall/tasks/main.yml
+++ b/roles/kubernetes/preinstall/tasks/main.yml
@ -75,6 +75,7 @@
  when:
    - dns_mode != 'none'
    - resolvconf_mode == 'host_resolvconf'
+    - dhclientconffile is defined
    - not ansible_os_family in ["Flatcar", "Flatcar Container Linux by Kinvolk"]
  tags:
    - bootstrap-os
@ -84,6 +85,7 @@
  when:
    - dns_mode != 'none'
    - resolvconf_mode != 'host_resolvconf'
+    - dhclientconffile is defined
    - not ansible_os_family in ["Flatcar", "Flatcar Container Linux by Kinvolk"]
  tags:
    - bootstrap-os
--- a/roles/kubespray-defaults/defaults/main.yaml
+++ b/roles/kubespray-defaults/defaults/main.yaml
@ -15,7 +15,7 @@ is_fedora_coreos: false
 disable_swap: true

 ## Change this to use another Kubernetes version, e.g. a current beta release
-kube_version: v1.22.5
+kube_version: v1.22.8

 ## The minimum version working
 kube_version_min_required: v1.20.0
@ -254,7 +254,7 @@ kubelet_shutdown_grace_period: 60s
 kubelet_shutdown_grace_period_critical_pods: 20s

 # Whether to deploy the container engine
-deploy_container_engine: inventory_hostname in groups['k8s_cluster'] or etcd_deployment_type != 'host'
+deploy_container_engine: "{{ inventory_hostname in groups['k8s_cluster'] or etcd_deployment_type != 'host' }}"

 # Container for runtime
 container_manager: containerd
@ -340,12 +340,8 @@ docker_plugins: []
 etcd_kubeadm_enabled: false

 # Containerd options - thse are relevant when container_manager == 'containerd'
-containerd_version: 1.5.8
 containerd_use_systemd_cgroup: true

-# Docker options - this is relevant when container_manager == 'docker'
-docker_containerd_version: 1.4.12
-
 ## An obvious use case is allowing insecure-registry access to self hosted registries.
 ## Can be ipaddress and domain_name.
 ## example define mirror.registry.io or 172.19.16.11:5000
@ -448,7 +444,7 @@ rbac_enabled: "{{ 'RBAC' in authorization_modes }}"
 kubelet_authentication_token_webhook: true

 # When enabled, access to the kubelet API requires authorization by delegation to the API server
-kubelet_authorization_mode_webhook: true
+kubelet_authorization_mode_webhook: false

 # kubelet uses certificates for authenticating to the Kubernetes API
 # Automatically generate a new key and request a new certificate from the Kubernetes API as the current certificate approaches expiration
--- a/test-infra/image-builder/roles/kubevirt-images/defaults/main.yml
+++ b/test-infra/image-builder/roles/kubevirt-images/defaults/main.yml
@ -62,6 +62,13 @@ images:
    converted: true
    tag: "latest"

+  almalinux-8:
+    filename: AlmaLinux-8-GenericCloud-latest.x86_64.qcow2
+    url: https://repo.almalinux.org/almalinux/8.5/cloud/x86_64/images/AlmaLinux-8-GenericCloud-8.5-20211119.x86_64.qcow2
+    checksum: sha256:d629247b12802157be127db53a7fcb484b80fceae9896d750c953a51a8c6688f
+    converted: true
+    tag: "latest"
+
  debian-9:
    filename: debian-9-openstack-amd64.qcow2
    url: https://cdimage.debian.org/cdimage/openstack/current-9/debian-9-openstack-amd64.qcow2
--- a/tests/cloud_playbooks/roles/packet-ci/defaults/main.yml
+++ b/tests/cloud_playbooks/roles/packet-ci/defaults/main.yml
@ -21,6 +21,7 @@ mode: aio
 cloud_init:
  centos-7: "I2Nsb3VkLWNvbmZpZwpzeXN0ZW1faW5mbzoKICBkaXN0cm86IHJoZWwKdXNlcnM6CiAtIG5hbWU6IGt1YmVzcHJheQogICBncm91cHM6IHdoZWVsCiAgIHN1ZG86ICdBTEw9KEFMTCkgTk9QQVNTV0Q6QUxMJwogICBzaGVsbDogL2Jpbi9iYXNoCiAgIGxvY2tfcGFzc3dkOiBGYWxzZQogICBob21lOiAvaG9tZS9rdWJlc3ByYXkKICAgc3NoX2F1dGhvcml6ZWRfa2V5czoKICAgICAtIHNzaC1yc2EgQUFBQUIzTnphQzF5YzJFQUFBQURBUUFCQUFBQkFRQ2FuVGkvZUt4MCt0SFlKQWVEaHErc0ZTMk9iVVAxL0k2OWY3aVYzVXRrS2xUMjBKZlcxZjZGZVh0LzA0VmYyN1dRcStOcXM2dkdCcUQ5UVhTWXVmK3QwL3M3RVBMalRlaTltZTFtcHFyK3VUZStLRHRUUDM5cGZEMy9lVkNhZUI3MjZHUDJGa2FEMEZ6cG1FYjY2TzNOcWh4T1E5Nkd4LzlYVHV3L0szbGxqNE9WRDZHcmpSM0I3YzRYdEVCc1pjWnBwTUovb0gxbUd5R1hkaDMxbVdRU3FBUk8vUDhVOEd3dDArSEdwVXdoL2hkeTN0K1NZb1RCMkd3VmIwem95Vnd0VnZmRFF6c204ZnEzYXY0S3ZlejhrWXVOREp2MDV4NGx2VVpnUjE1WkRSWHNBbmRoUXlxb1hkQ0xBZTArZWFLWHE5QmtXeEtGYjloUGUwQVVqamE1Cgo="
  centos-8: "I2Nsb3VkLWNvbmZpZwpzeXN0ZW1faW5mbzoKICBkaXN0cm86IHJoZWwKdXNlcnM6CiAtIG5hbWU6IGt1YmVzcHJheQogICBncm91cHM6IHdoZWVsCiAgIHN1ZG86ICdBTEw9KEFMTCkgTk9QQVNTV0Q6QUxMJwogICBzaGVsbDogL2Jpbi9iYXNoCiAgIGxvY2tfcGFzc3dkOiBGYWxzZQogICBob21lOiAvaG9tZS9rdWJlc3ByYXkKICAgc3NoX2F1dGhvcml6ZWRfa2V5czoKICAgICAtIHNzaC1yc2EgQUFBQUIzTnphQzF5YzJFQUFBQURBUUFCQUFBQkFRQ2FuVGkvZUt4MCt0SFlKQWVEaHErc0ZTMk9iVVAxL0k2OWY3aVYzVXRrS2xUMjBKZlcxZjZGZVh0LzA0VmYyN1dRcStOcXM2dkdCcUQ5UVhTWXVmK3QwL3M3RVBMalRlaTltZTFtcHFyK3VUZStLRHRUUDM5cGZEMy9lVkNhZUI3MjZHUDJGa2FEMEZ6cG1FYjY2TzNOcWh4T1E5Nkd4LzlYVHV3L0szbGxqNE9WRDZHcmpSM0I3YzRYdEVCc1pjWnBwTUovb0gxbUd5R1hkaDMxbVdRU3FBUk8vUDhVOEd3dDArSEdwVXdoL2hkeTN0K1NZb1RCMkd3VmIwem95Vnd0VnZmRFF6c204ZnEzYXY0S3ZlejhrWXVOREp2MDV4NGx2VVpnUjE1WkRSWHNBbmRoUXlxb1hkQ0xBZTArZWFLWHE5QmtXeEtGYjloUGUwQVVqamE1Cgo="
+  almalinux-8: "I2Nsb3VkLWNvbmZpZwpzeXN0ZW1faW5mbzoKICBkaXN0cm86IHJoZWwKdXNlcnM6CiAtIG5hbWU6IGt1YmVzcHJheQogICBncm91cHM6IHdoZWVsCiAgIHN1ZG86ICdBTEw9KEFMTCkgTk9QQVNTV0Q6QUxMJwogICBzaGVsbDogL2Jpbi9iYXNoCiAgIGxvY2tfcGFzc3dkOiBGYWxzZQogICBob21lOiAvaG9tZS9rdWJlc3ByYXkKICAgc3NoX2F1dGhvcml6ZWRfa2V5czoKICAgICAtIHNzaC1yc2EgQUFBQUIzTnphQzF5YzJFQUFBQURBUUFCQUFBQkFRQ2FuVGkvZUt4MCt0SFlKQWVEaHErc0ZTMk9iVVAxL0k2OWY3aVYzVXRrS2xUMjBKZlcxZjZGZVh0LzA0VmYyN1dRcStOcXM2dkdCcUQ5UVhTWXVmK3QwL3M3RVBMalRlaTltZTFtcHFyK3VUZStLRHRUUDM5cGZEMy9lVkNhZUI3MjZHUDJGa2FEMEZ6cG1FYjY2TzNOcWh4T1E5Nkd4LzlYVHV3L0szbGxqNE9WRDZHcmpSM0I3YzRYdEVCc1pjWnBwTUovb0gxbUd5R1hkaDMxbVdRU3FBUk8vUDhVOEd3dDArSEdwVXdoL2hkeTN0K1NZb1RCMkd3VmIwem95Vnd0VnZmRFF6c204ZnEzYXY0S3ZlejhrWXVOREp2MDV4NGx2VVpnUjE1WkRSWHNBbmRoUXlxb1hkQ0xBZTArZWFLWHE5QmtXeEtGYjloUGUwQVVqamE1Cgo="
  debian-9: "I2Nsb3VkLWNvbmZpZwp1c2VyczoKIC0gbmFtZToga3ViZXNwcmF5CiAgIHN1ZG86IEFMTD0oQUxMKSBOT1BBU1NXRDpBTEwKICAgc2hlbGw6IC9iaW4vYmFzaAogICBob21lOiAvaG9tZS91YnVudHUKICAgc3NoX2F1dGhvcml6ZWRfa2V5czoKICAgICAtIHNzaC1yc2EgQUFBQUIzTnphQzF5YzJFQUFBQURBUUFCQUFBQkFRQ2FuVGkvZUt4MCt0SFlKQWVEaHErc0ZTMk9iVVAxL0k2OWY3aVYzVXRrS2xUMjBKZlcxZjZGZVh0LzA0VmYyN1dRcStOcXM2dkdCcUQ5UVhTWXVmK3QwL3M3RVBMalRlaTltZTFtcHFyK3VUZStLRHRUUDM5cGZEMy9lVkNhZUI3MjZHUDJGa2FEMEZ6cG1FYjY2TzNOcWh4T1E5Nkd4LzlYVHV3L0szbGxqNE9WRDZHcmpSM0I3YzRYdEVCc1pjWnBwTUovb0gxbUd5R1hkaDMxbVdRU3FBUk8vUDhVOEd3dDArSEdwVXdoL2hkeTN0K1NZb1RCMkd3VmIwem95Vnd0VnZmRFF6c204ZnEzYXY0S3ZlejhrWXVOREp2MDV4NGx2VVpnUjE1WkRSWHNBbmRoUXlxb1hkQ0xBZTArZWFLWHE5QmtXeEtGYjloUGUwQVVqamE1"
  debian-10: "I2Nsb3VkLWNvbmZpZwp1c2VyczoKIC0gbmFtZToga3ViZXNwcmF5CiAgIHN1ZG86IEFMTD0oQUxMKSBOT1BBU1NXRDpBTEwKICAgc2hlbGw6IC9iaW4vYmFzaAogICBob21lOiAvaG9tZS91YnVudHUKICAgc3NoX2F1dGhvcml6ZWRfa2V5czoKICAgICAtIHNzaC1yc2EgQUFBQUIzTnphQzF5YzJFQUFBQURBUUFCQUFBQkFRQ2FuVGkvZUt4MCt0SFlKQWVEaHErc0ZTMk9iVVAxL0k2OWY3aVYzVXRrS2xUMjBKZlcxZjZGZVh0LzA0VmYyN1dRcStOcXM2dkdCcUQ5UVhTWXVmK3QwL3M3RVBMalRlaTltZTFtcHFyK3VUZStLRHRUUDM5cGZEMy9lVkNhZUI3MjZHUDJGa2FEMEZ6cG1FYjY2TzNOcWh4T1E5Nkd4LzlYVHV3L0szbGxqNE9WRDZHcmpSM0I3YzRYdEVCc1pjWnBwTUovb0gxbUd5R1hkaDMxbVdRU3FBUk8vUDhVOEd3dDArSEdwVXdoL2hkeTN0K1NZb1RCMkd3VmIwem95Vnd0VnZmRFF6c204ZnEzYXY0S3ZlejhrWXVOREp2MDV4NGx2VVpnUjE1WkRSWHNBbmRoUXlxb1hkQ0xBZTArZWFLWHE5QmtXeEtGYjloUGUwQVVqamE1"
  debian-11: "I2Nsb3VkLWNvbmZpZwp1c2VyczoKIC0gbmFtZToga3ViZXNwcmF5CiAgIHN1ZG86IEFMTD0oQUxMKSBOT1BBU1NXRDpBTEwKICAgc2hlbGw6IC9iaW4vYmFzaAogICBob21lOiAvaG9tZS91YnVudHUKICAgc3NoX2F1dGhvcml6ZWRfa2V5czoKICAgICAtIHNzaC1yc2EgQUFBQUIzTnphQzF5YzJFQUFBQURBUUFCQUFBQkFRQ2FuVGkvZUt4MCt0SFlKQWVEaHErc0ZTMk9iVVAxL0k2OWY3aVYzVXRrS2xUMjBKZlcxZjZGZVh0LzA0VmYyN1dRcStOcXM2dkdCcUQ5UVhTWXVmK3QwL3M3RVBMalRlaTltZTFtcHFyK3VUZStLRHRUUDM5cGZEMy9lVkNhZUI3MjZHUDJGa2FEMEZ6cG1FYjY2TzNOcWh4T1E5Nkd4LzlYVHV3L0szbGxqNE9WRDZHcmpSM0I3YzRYdEVCc1pjWnBwTUovb0gxbUd5R1hkaDMxbVdRU3FBUk8vUDhVOEd3dDArSEdwVXdoL2hkeTN0K1NZb1RCMkd3VmIwem95Vnd0VnZmRFF6c204ZnEzYXY0S3ZlejhrWXVOREp2MDV4NGx2VVpnUjE1WkRSWHNBbmRoUXlxb1hkQ0xBZTArZWFLWHE5QmtXeEtGYjloUGUwQVVqamE1"
--- a/tests/files/packet_almalinux8-calico-ha-ebpf.yml
+++ b/tests/files/packet_almalinux8-calico-ha-ebpf.yml
@ -1,6 +1,6 @@
 ---
 # Instance settings
-cloud_image: centos-8
+cloud_image: almalinux-8
 mode: ha
 vm_memory: 3072Mi

--- a/tests/files/packet_almalinux8-calico-nodelocaldns-secondary.yml
+++ b/tests/files/packet_almalinux8-calico-nodelocaldns-secondary.yml
@ -1,6 +1,6 @@
 ---
 # Instance settings
-cloud_image: centos-8
+cloud_image: almalinux-8
 mode: default
 vm_memory: 3072Mi

--- a/tests/files/packet_almalinux8-calico.yml
+++ b/tests/files/packet_almalinux8-calico.yml
@ -1,6 +1,6 @@
 ---
 # Instance settings
-cloud_image: centos-8
+cloud_image: almalinux-8
 mode: default
 vm_memory: 3072Mi

--- a/tests/files/packet_almalinux8-crio.yml
+++ b/tests/files/packet_almalinux8-crio.yml
@ -1,6 +1,6 @@
 ---
 # Instance settings
-cloud_image: centos-8
+cloud_image: almalinux-8
 mode: default

 # Kubespray settings
--- a/tests/files/packet_almalinux8-docker.yml
+++ b/tests/files/packet_almalinux8-docker.yml
@ -1,6 +1,6 @@
 ---
 # Instance settings
-cloud_image: centos-8
+cloud_image: almalinux-8
 mode: default
 vm_memory: 3072Mi

--- a/tests/files/packet_almalinux8-kube-ovn.yml
+++ b/tests/files/packet_almalinux8-kube-ovn.yml
@ -1,6 +1,6 @@
 ---
 # Instance settings
-cloud_image: centos-8
+cloud_image: almalinux-8
 mode: default
 vm_memory: 3072Mi
Author	SHA1	Message	Date
Zhong Jianxin	70d4f70c3c	[2.18] preinstall: Add nodelocaldns to supersede_nameserver if enabled When a machine that use dhclient and resolvconf reboots, this will make /etc/resolv.conf remain close to the one before reboot	2022-09-27 09:06:03 +02:00
Cristian Calin	56f9af8668	disable kubelet_authorization_mode_webhook by default	2022-08-31 11:44:14 +02:00
Sébastien Masset	29d81f7617	Aligned existing handling of extra DNS tolerations Signed-off-by: Sébastien Masset <smt.masset@gmail.com>	2022-07-12 09:24:39 +02:00
Sébastien Masset	2546eb2559	Added new configuration item for extra tolerations in DNS autoscaler Signed-off-by: Sébastien Masset <smt.masset@gmail.com>	2022-07-12 09:24:39 +02:00
Sébastien Masset	47af8b7ea5	Added new configuration item for extra tolerations in policy controllers Signed-off-by: Sébastien Masset <smt.masset@gmail.com>	2022-07-12 09:24:39 +02:00
Kenichi Omichi	2cc5f04bad	Removed quotation of nerdctl_extra_flags. (#8699 ) The quotations in the variable nerdctl_extra_flags are not required for the `nerdctl_image_pull_command` and throw the following error when executing the cluster-playbook with `container_insecure_registries` set: unknown flag: --insecure-registry\\\" This happens as the complete nerdctl_image_pull_command string variable gets split into an array string for the cmd task. The escaped quotation doesn't get escaped properly and is added to the cmd-string array as part of the command. This leads to a wrong written insecure-registry flag, which throws this error.	2022-04-09 11:02:04 -07:00
Mathieu Parent	d4d03a01f2	Fix image_command_tool ignored since PR #8601 (#8684 ) Signed-off-by: Mathieu Parent <mathieu.parent@insee.fr>	2022-04-06 02:00:58 -07:00
Kenichi Omichi	bdaa33c3bc	Fix quotation of nerdctl_extra_flags (#8668 ) (#8685 ) Due to missing quotation of nerdctl_extra_flags, ansible-playbook was failed: Using module file /usr/local/lib/python3.6/dist-packages/ansible/modules/command.py Pipelining is enabled. [..] File "/usr/lib/python3.8/shlex.py", line 191, in read_token raise ValueError("No closing quotation") This fixes the issue. T-Eberle investigated the issue and found the solution. Thank you T-Eberle!	2022-04-05 15:25:25 -07:00
rtsp	58bea67b68	[2.18] cert-manager: Backport cert-manager leader election namespace fixes from master (#8681 ) cherry-picked from * `ccd3180` cert-manager: Fix incorrect leader election namespace lead to insufficient permission (#8433) * `e791089` cert-manager: Allow to change leader election namespace for GKE Autopilot support (#8424)	2022-04-04 11:10:11 -07:00
Kenichi Omichi	f091b1cfd7	[2.18] Run 0100-dhclient-hooks if dhcpclient is enabled (#8658 ) (#8661 ) * Run 0100-dhclient-hooks if dhcpclient is enabled (#8658) If running Kubespray on static IP environments, a task was failed like: TASK [kubernetes/preinstall : Configure dhclient hooks for resolv.conf (RH-only)] fatal: [ak8s2]: FAILED! => { "changed": false, "checksum": "..", "msg": "Destination directory /etc/dhcp/dhclient.d does not exist"} This adds a check for dhclientconffile for running 0100-dhclient-hooks to run the task only if dhcpclient is enabled. * Remove centos7 molecule while opensuse mirror is flaky Co-authored-by: Florian Ruynat <16313165+floryut@users.noreply.github.com>	2022-03-30 10:08:25 -07:00
Cristian Calin	e7508d7d21	[sysctl] set fs.may_detach_mounts=1 even when CRIs don't set it themselves (#8635 ) (#8642 )	2022-03-22 05:31:44 -07:00
Cristian Calin	59c05d3713	[container image] use focal (ubuntu 20.04) base image for our docker builds (#8631 ) (#8633 )	2022-03-21 01:03:09 -07:00
Calin Cristian Andrei	ae1f8d8578	[kubernetes] make 1.22.8 the new default	2022-03-18 11:26:41 -07:00
Calin Cristian Andrei	aafdcc1b68	[backport-2.18] update kubernetes hashes for 1.23, 1.22 and 1.21	2022-03-18 11:26:41 -07:00
Takuya Murakami	019bcbc893	Update config.toml.j2 (#8340 ) (#8602 ) * Update config.toml.j2 i think this commit code is not completed works exam registry address : a.com:5000 insecure registry must be http://a.com:5000 but this code add insecure a.com:5000 (without http://) If there is no http, containerd accesses with https even if insecure_skip_verify = true solution is code edit * Update config.toml.j2 * Update containerd.yml * Update containerd.yml * Update containerd.yml * Update config.toml.j2 (cherry picked from commit `dda557ed23`) Co-authored-by: Choi Yongbeom <59861163+mircyb@users.noreply.github.com>	2022-03-09 06:22:13 -08:00
Takuya Murakami	0c43883e5c	[PATCH] nerdctl insecure registry config (#8339 ) (#8601 ) Backport #8339 to 2.18-release Cherry-pick `24f1402a14` Co-authored-by: Choi Yongbeom <59861163+mircyb@users.noreply.github.com>	2022-03-08 14:32:22 -08:00
Takuya Murakami	92d6c2d9a8	feat(offline): Improve generate_list.sh to generate offline file list using ansible (#8537 ) (#8538 ) (#8606 ) Use jinja2 template and ansible to expand variables.	2022-03-07 05:32:55 -08:00
Kenichi Omichi	411902e9ff	Update quay.io/kubespray/vagrant (#8605 ) quay.io/kubespray/vagrant image is used for molecule_tests. The tag was v2.17.1 on release-2.18 branch but the image contains vagrant-2.2.15 which has a bug related to a virtual machine creation. That caused kubespray CI failures. This updates the image to use a newer vagrant.	2022-03-06 00:12:52 -08:00
Kenichi Omichi	c4a2745523	Move containerd_version to defaults/main.yml (#8379 ) (#8513 ) All container image versions were defined in download/defaults/main.yml except containerd. The inconsistency caused the offline script(generate_list.sh) could not output the URL of containerd image. This moves the definition into a valid file. In addition, this adds host_os to generate_list.sh for downloading krew from a valid URL.	2022-02-13 09:55:50 -08:00
Kenichi Omichi	d1609e3111	CI: Replace CentOS 8 with AlmaLinux 8 before CentOS 8 EOL end of 2021 (#8297 ) (#8514 ) Co-authored-by: Cristian Calin <6627509+cristicalin@users.noreply.github.com>	2022-02-07 23:42:53 -08:00
Cristian Calin	6abffe9c37	[2.18] update kubernetes hashes and make 1.22.6 the default (#8467 ) * [kubernetes] add hashes for 1.23.2, 1.22.6, 1.21.9 and 1.20.15 * [kubernetes] make 1.22.6 the default version	2022-01-25 05:20:30 -08:00
Boris Barnier	a5cd98f6cf	Fix kata_containers_binary_checksums for arm64 (#8460 )	2022-01-24 00:19:57 -08:00
Mathieu Parent	38d85cfafd	Document image_command_tool and image_command_tool_on_localhost (#8409 ) Signed-off-by: Mathieu Parent <mathieu.parent@insee.fr> (cherry picked from commit `43d128362f`)	2022-01-17 02:25:30 -08:00
Mathieu Parent	7fffe6730c	Allow to choose container manager commands (#8380 ) This allow to workaround #8375 by using image_command_tool=crictl when containerd_registries is used for containerd. Also changes image_info_command_on_localhost for docker to return digests. (cherry picked from commit `cfd9873bbc`) The cherry-pick was adapted because nerdctl_extra_flags is not in the release-2.18 branch (#8339).	2022-01-17 02:25:30 -08:00
Mathieu Parent	0b99ea69a9	Add missing example offline nerdctl_download_url (#8373 ) (cherry picked from commit `c11e4ba9a7`)	2022-01-17 02:25:30 -08:00
Mathieu Parent	1928f946be	Avoid yanked ruamel.yaml.clib version (#8372 ) See https://pypi.org/project/ruamel.yaml.clib/#history Signed-off-by: Mathieu Parent <math.parent@gmail.com> (cherry picked from commit `7ae00947f5`)	2022-01-17 02:25:30 -08:00
rtsp	8a3c78e8b4	[2.18] Fix container engine still installed on dedicated etcd (#8404 ) * Fix container engine still installed on dedicated etcd node even if `etcd_deployment_type: host` (#8386) (cherry picked from commit `aa4a3d7`)	2022-01-11 00:31:16 -08:00