Add kubeadm option to etcd_deployment_type to replace the etcd_kubeadm_enabled variable (#8317)

* Add kubeadm option to etcd_deployment_type to replace the etcd_kubeadm_enabled variable

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* Add etcd kubeadm deployment documentation

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* Refactor warning for the deprecated 'etcd_kubeadm_enabled' variable

Signed-off-by: necatican <necaticanyildirim@gmail.com>
This commit is contained in:
Necatican Yıldırım 2022-02-22 19:53:16 +03:00 committed by GitHub
parent b9a27c91da
commit e9c8913248
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
24 changed files with 99 additions and 60 deletions

View file

@ -46,7 +46,7 @@
vars: vars:
etcd_cluster_setup: true etcd_cluster_setup: true
etcd_events_cluster_setup: "{{ etcd_events_cluster_enabled }}" etcd_events_cluster_setup: "{{ etcd_events_cluster_enabled }}"
when: not etcd_kubeadm_enabled| default(false) when: etcd_deployment_type != "kubeadm"
- hosts: k8s_cluster - hosts: k8s_cluster
gather_facts: False gather_facts: False
@ -59,7 +59,7 @@
vars: vars:
etcd_cluster_setup: false etcd_cluster_setup: false
etcd_events_cluster_setup: false etcd_events_cluster_setup: false
when: not etcd_kubeadm_enabled| default(false) when: etcd_deployment_type != "kubeadm"
- hosts: k8s_cluster - hosts: k8s_cluster
gather_facts: False gather_facts: False

View file

@ -13,7 +13,7 @@ _To use the CRI-O container runtime set the following variables:_
```yaml ```yaml
download_container: false download_container: false
skip_downloads: false skip_downloads: false
etcd_kubeadm_enabled: true etcd_deployment_type: host # optionally kubeadm
``` ```
## k8s_cluster/k8s_cluster.yml ## k8s_cluster/k8s_cluster.yml
@ -22,12 +22,6 @@ etcd_kubeadm_enabled: true
container_manager: crio container_manager: crio
``` ```
## etcd.yml
```yaml
etcd_deployment_type: host # optionally and mutually exclusive with etcd_kubeadm_enabled
```
## all/crio.yml ## all/crio.yml
Enable docker hub registry mirrors Enable docker hub registry mirrors

View file

@ -1,5 +1,21 @@
# etcd # etcd
## Deployment Types
It is possible to deploy etcd with three methods. To change the default deployment method (host), use the `etcd_deployment_type` variable. Possible values are `host`, `kubeadm`, and `docker`.
### Host
Host deployment is the default method. Using this method will result in etcd installed as a systemd service.
### Docker
Installs docker in etcd group members and runs etcd on docker containers. Only usable when `container_manager` is set to `docker`.
### Kubeadm
This deployment method is experimental and is only available for new deployments. This deploys etcd as a static pod in master hosts.
## Metrics ## Metrics
To expose metrics on a separate HTTP port, define it in the inventory with: To expose metrics on a separate HTTP port, define it in the inventory with:

View file

@ -308,6 +308,18 @@ caprica Ready master,node 7h40m v1.14.1
``` ```
## Upgrading to v2.19
`etcd_kubeadm_enabled` is being deprecated at v2.19. The same functionality is achievable by setting `etcd_deployment_type` to `kubeadm`.
Deploying etcd using kubeadm is experimental and is only available for either new or deployments where `etcd_kubeadm_enabled` was set to `true` while deploying the cluster.
From 2.19 and onward `etcd_deployment_type` variable will be placed in `group_vars/all/etcd.yml` instead of `group_vars/etcd.yml`, due to scope issues.
The placement of the variable is only important for `etcd_deployment_type: kubeadm` right now. However, since this might change in future updates, it is recommended to move the variable.
Upgrading is straightforward; no changes are required if `etcd_kubeadm_enabled` was not set to `true` when deploying.
If you have a cluster where `etcd` was deployed using `kubeadm`, you will need to remove `etcd_kubeadm_enabled` the variable. Then move `etcd_deployment_type` variable from `group_vars/etcd.yml` to `group_vars/all/etcd.yml` due to scope issues and set `etcd_deployment_type` to `kubeadm`.
## Upgrade order ## Upgrade order
As mentioned above, components are upgraded in the order in which they were As mentioned above, components are upgraded in the order in which they were

View file

@ -1,10 +1,4 @@
--- ---
## Directory where etcd data stored
etcd_data_dir: /var/lib/etcd
## Experimental kubeadm etcd deployment mode. Available only for new deployment
etcd_kubeadm_enabled: false
## Directory where the binaries will be installed ## Directory where the binaries will be installed
bin_dir: /usr/local/bin bin_dir: /usr/local/bin

View file

@ -0,0 +1,16 @@
---
## Directory where etcd data stored
etcd_data_dir: /var/lib/etcd
## Container runtime
## docker for docker, crio for cri-o and containerd for containerd.
## Additionally you can set this to kubeadm if you want to install etcd using kubeadm
## Kubeadm etcd deployment is experimental and only available for new deployments
## If this is not set, container manager will be inherited from the Kubespray defaults
## and not from k8s_cluster/k8s-cluster.yml, which might not be what you want.
## Also this makes possible to use different container manager for etcd nodes.
# container_manager: containerd
## Settings for etcd deployment type
# Set this to docker if you are using container_manager: docker
etcd_deployment_type: host

View file

@ -17,14 +17,3 @@
### ETCD: disable peer client cert authentication. ### ETCD: disable peer client cert authentication.
# This affects ETCD_PEER_CLIENT_CERT_AUTH variable # This affects ETCD_PEER_CLIENT_CERT_AUTH variable
# etcd_peer_client_auth: true # etcd_peer_client_auth: true
## Container runtime
## docker for docker, crio for cri-o and containerd for containerd.
## If this is not set, container manager will be inherited from the Kubespray defaults
## and not from k8s_cluster/k8s-cluster.yml, which might not be what you want.
## Also this makes possible to use different container manager for etcd nodes.
# container_manager: containerd
## Settings for etcd deployment type
# Set this to docker if you are using container_manager: docker
etcd_deployment_type: host

View file

@ -8,7 +8,7 @@ kind: ClusterConfiguration
imageRepository: {{ kube_image_repo }} imageRepository: {{ kube_image_repo }}
kubernetesVersion: {{ kube_version }} kubernetesVersion: {{ kube_version }}
etcd: etcd:
{% if etcd_kubeadm_enabled %} {% if etcd_deployment_type == "kubeadm" %}
local: local:
imageRepository: "{{ etcd_image_repo | regex_replace("/etcd$","") }}" imageRepository: "{{ etcd_image_repo | regex_replace("/etcd$","") }}"
imageTag: "{{ etcd_image_tag }}" imageTag: "{{ etcd_image_tag }}"

View file

@ -4,7 +4,7 @@
- name: Check unintentional include of this role - name: Check unintentional include of this role
assert: assert:
that: etcd_kubeadm_enabled that: etcd_deployment_type == "kubeadm"
- name: Check if etcdctl exist - name: Check if etcdctl exist
stat: stat:

View file

@ -2,9 +2,6 @@
# disable upgrade cluster # disable upgrade cluster
upgrade_cluster_setup: false upgrade_cluster_setup: false
# Experimental kubeadm etcd deployment mode. Available only for new deployment
etcd_kubeadm_enabled: false
# change to 0.0.0.0 to enable insecure access from anywhere (not recommended) # change to 0.0.0.0 to enable insecure access from anywhere (not recommended)
kube_apiserver_insecure_bind_address: 127.0.0.1 kube_apiserver_insecure_bind_address: 127.0.0.1

View file

@ -15,4 +15,4 @@
- name: Ensure etcdctl script is installed - name: Ensure etcdctl script is installed
import_role: import_role:
name: etcdctl name: etcdctl
when: etcd_kubeadm_enabled when: etcd_deployment_type == "kubeadm"

View file

@ -21,4 +21,4 @@
dest: "{{ kube_config_dir }}/manifests/kube-apiserver.yaml" dest: "{{ kube_config_dir }}/manifests/kube-apiserver.yaml"
regexp: '^ - --etcd-servers=' regexp: '^ - --etcd-servers='
line: ' - --etcd-servers={{ etcd_access_addresses }}' line: ' - --etcd-servers={{ etcd_access_addresses }}'
when: not etcd_kubeadm_enabled | default(false) when: etcd_deployment_type != "kubeadm"

View file

@ -18,7 +18,7 @@
--config={{ kube_config_dir }}/kubeadm-config.yaml --config={{ kube_config_dir }}/kubeadm-config.yaml
--ignore-preflight-errors=all --ignore-preflight-errors=all
--allow-experimental-upgrades --allow-experimental-upgrades
--etcd-upgrade={{ etcd_kubeadm_enabled | bool | lower }} --etcd-upgrade={{ etcd_deployment_type == "kubeadm" | bool | lower }}
--force --force
register: kubeadm_upgrade register: kubeadm_upgrade
# Retry is because upload config sometimes fails # Retry is because upload config sometimes fails
@ -39,7 +39,7 @@
--config={{ kube_config_dir }}/kubeadm-config.yaml --config={{ kube_config_dir }}/kubeadm-config.yaml
--ignore-preflight-errors=all --ignore-preflight-errors=all
--allow-experimental-upgrades --allow-experimental-upgrades
--etcd-upgrade={{ etcd_kubeadm_enabled | bool | lower }} --etcd-upgrade={{ etcd_deployment_type == "kubeadm" | bool | lower }}
--force --force
register: kubeadm_upgrade register: kubeadm_upgrade
when: inventory_hostname != first_kube_control_plane when: inventory_hostname != first_kube_control_plane

View file

@ -69,7 +69,7 @@
- name: Include kubeadm etcd extra tasks - name: Include kubeadm etcd extra tasks
include_tasks: kubeadm-etcd.yml include_tasks: kubeadm-etcd.yml
when: etcd_kubeadm_enabled when: etcd_deployment_type == "kubeadm"
- name: Include kubeadm secondary server apiserver fixes - name: Include kubeadm secondary server apiserver fixes
include_tasks: kubeadm-fix-apiserver.yml include_tasks: kubeadm-fix-apiserver.yml

View file

@ -33,7 +33,7 @@ apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration kind: ClusterConfiguration
clusterName: {{ cluster_name }} clusterName: {{ cluster_name }}
etcd: etcd:
{% if not etcd_kubeadm_enabled %} {% if etcd_deployment_type != "kubeadm" %}
external: external:
endpoints: endpoints:
{% for endpoint in etcd_access_addresses.split(',') %} {% for endpoint in etcd_access_addresses.split(',') %}
@ -42,7 +42,7 @@ etcd:
caFile: {{ etcd_cert_dir }}/{{ kube_etcd_cacert_file }} caFile: {{ etcd_cert_dir }}/{{ kube_etcd_cacert_file }}
certFile: {{ etcd_cert_dir }}/{{ kube_etcd_cert_file }} certFile: {{ etcd_cert_dir }}/{{ kube_etcd_cert_file }}
keyFile: {{ etcd_cert_dir }}/{{ kube_etcd_key_file }} keyFile: {{ etcd_cert_dir }}/{{ kube_etcd_key_file }}
{% elif etcd_kubeadm_enabled %} {% elif etcd_deployment_type == "kubeadm" %}
local: local:
imageRepository: "{{ etcd_image_repo | regex_replace("/etcd$","") }}" imageRepository: "{{ etcd_image_repo | regex_replace("/etcd$","") }}"
imageTag: "{{ etcd_image_tag }}" imageTag: "{{ etcd_image_tag }}"

View file

@ -10,6 +10,3 @@ kube_override_hostname: >-
{%- else -%} {%- else -%}
{{ inventory_hostname }} {{ inventory_hostname }}
{%- endif -%} {%- endif -%}
# Experimental kubeadm etcd deployment mode. Available only for new deployment
etcd_kubeadm_enabled: false

View file

@ -153,7 +153,7 @@
- name: Extract etcd certs from control plane if using etcd kubeadm mode - name: Extract etcd certs from control plane if using etcd kubeadm mode
include_tasks: kubeadm_etcd_node.yml include_tasks: kubeadm_etcd_node.yml
when: when:
- etcd_kubeadm_enabled - etcd_deployment_type == "kubeadm"
- inventory_hostname not in groups['kube_control_plane'] - inventory_hostname not in groups['kube_control_plane']
- kube_network_plugin in ["calico", "flannel", "canal", "cilium"] or cilium_deploy_additionally | default(false) | bool - kube_network_plugin in ["calico", "flannel", "canal", "cilium"] or cilium_deploy_additionally | default(false) | bool
- kube_network_plugin != "calico" or calico_datastore == "etcd" - kube_network_plugin != "calico" or calico_datastore == "etcd"

View file

@ -15,7 +15,7 @@
run_once: true run_once: true
when: when:
- not ignore_assert_errors - not ignore_assert_errors
- not etcd_kubeadm_enabled - etcd_deployment_type != "kubeadm"
- name: Stop if non systemd OS type - name: Stop if non systemd OS type
assert: assert:
@ -277,23 +277,41 @@
when: resolvconf_mode is defined when: resolvconf_mode is defined
run_once: true run_once: true
- name: Stop if etcd deployment type is not host or docker - name: Stop if etcd deployment type is not host, docker or kubeadm
assert: assert:
that: etcd_deployment_type in ['host', 'docker'] that: etcd_deployment_type in ['host', 'docker', 'kubeadm']
msg: "The etcd deployment type, 'etcd_deployment_type', must be host or docker" msg: "The etcd deployment type, 'etcd_deployment_type', must be host, docker or kubeadm"
when: when:
- inventory_hostname in groups.get('etcd',[]) - inventory_hostname in groups.get('etcd',[])
- not etcd_kubeadm_enabled
- name: Stop if etcd deployment type is not host when container_manager != docker - name: Stop if etcd deployment type is not host or kubeadm when container_manager != docker
assert: assert:
that: etcd_deployment_type == 'host' that: etcd_deployment_type in ['host', 'kubeadm']
msg: "The etcd deployment type, 'etcd_deployment_type', must be host when container_manager is not docker" msg: "The etcd deployment type, 'etcd_deployment_type', must be host or kubeadm when container_manager is not docker"
when: when:
- inventory_hostname in groups.get('etcd',[]) - inventory_hostname in groups.get('etcd',[])
- not etcd_kubeadm_enabled
- container_manager != 'docker' - container_manager != 'docker'
# TODO: Clean this task up when we drop backward compatibility support for `etcd_kubeadm_enabled`
- name: Stop if etcd deployment type is not host or kubeadm when container_manager != docker and etcd_kubeadm_enabled is not defined
block:
- name: Warn the user if they are still using `etcd_kubeadm_enabled`
debug:
msg: >
"WARNING! => `etcd_kubeadm_enabled` is deprecated and will be removed in a future release.
You can set `etcd_deployment_type` to `kubeadm` instead of setting `etcd_kubeadm_enabled` to `true`."
changed_when: true
- name: Stop if `etcd_kubeadm_enabled` is defined and `etcd_deployment_type` is not `kubadm` or `host`
assert:
that: etcd_deployment_type == 'kubeadm'
msg: >
It is not possible to use `etcd_kubeadm_enabled` when `etcd_deployment_type` is set to {{ etcd_deployment_type }}.
Unset the `etcd_kubeadm_enabled` variable and set `etcd_deployment_type` to desired deployment type (`host`, `kubeadm`, `docker`) instead."
when: etcd_kubeadm_enabled
run_once: yes
when: etcd_kubeadm_enabled is defined
- name: Stop if download_localhost is enabled but download_run_once is not - name: Stop if download_localhost is enabled but download_run_once is not
assert: assert:
that: download_run_once that: download_run_once

View file

@ -205,7 +205,7 @@
kube_etcd_cert_file: "apiserver-etcd-client.crt" kube_etcd_cert_file: "apiserver-etcd-client.crt"
kube_etcd_key_file: "apiserver-etcd-client.key" kube_etcd_key_file: "apiserver-etcd-client.key"
when: when:
- etcd_kubeadm_enabled - etcd_deployment_type == "kubeadm"
- name: check /usr readonly - name: check /usr readonly
stat: stat:

View file

@ -258,7 +258,7 @@ kubelet_shutdown_grace_period: 60s
kubelet_shutdown_grace_period_critical_pods: 20s kubelet_shutdown_grace_period_critical_pods: 20s
# Whether to deploy the container engine # Whether to deploy the container engine
deploy_container_engine: "{{ inventory_hostname in groups['k8s_cluster'] or etcd_deployment_type != 'host' }}" deploy_container_engine: "{{ inventory_hostname in groups['k8s_cluster'] or etcd_deployment_type == 'docker' }}"
# Container for runtime # Container for runtime
container_manager: containerd container_manager: containerd
@ -344,9 +344,6 @@ docker_registry_mirrors: []
## Empty by default so no plugins will be installed. ## Empty by default so no plugins will be installed.
docker_plugins: [] docker_plugins: []
# Experimental kubeadm etcd deployment mode. Available only for new deployment
etcd_kubeadm_enabled: false
# Containerd options - thse are relevant when container_manager == 'containerd' # Containerd options - thse are relevant when container_manager == 'containerd'
containerd_use_systemd_cgroup: true containerd_use_systemd_cgroup: true

View file

@ -22,3 +22,12 @@
- no_proxy is not defined - no_proxy is not defined
tags: tags:
- always - always
# TODO: Clean this task up when we drop backward compatibility support for `etcd_kubeadm_enabled`
- name: Set `etcd_deployment_type` to "kubeadm" if `etcd_kubeadm_enabled` is true
set_fact:
etcd_deployment_type: kubeadm
when:
- etcd_kubeadm_enabled is defined and etcd_kubeadm_enabled
tags:
- always

View file

@ -55,7 +55,7 @@
- { role: kubernetes/preinstall, tags: preinstall } - { role: kubernetes/preinstall, tags: preinstall }
- { role: container-engine, tags: "container-engine", when: deploy_container_engine } - { role: container-engine, tags: "container-engine", when: deploy_container_engine }
- { role: download, tags: download, when: "not skip_downloads" } - { role: download, tags: download, when: "not skip_downloads" }
- { role: etcd, tags: etcd, etcd_cluster_setup: false, when: "not etcd_kubeadm_enabled|default(false)" } - { role: etcd, tags: etcd, etcd_cluster_setup: false, when: "etcd_deployment_type != 'kubeadm'" }
- name: Target only workers to get kubelet installed and checking in on any new nodes(node) - name: Target only workers to get kubelet installed and checking in on any new nodes(node)
hosts: kube_node hosts: kube_node

View file

@ -5,6 +5,6 @@ mode: ha
# Kubespray settings # Kubespray settings
kube_network_plugin: flannel kube_network_plugin: flannel
etcd_kubeadm_enabled: true etcd_deployment_type: kubeadm
kubeadm_certificate_key: 3998c58db6497dd17d909394e62d515368c06ec617710d02edea31c06d741085 kubeadm_certificate_key: 3998c58db6497dd17d909394e62d515368c06ec617710d02edea31c06d741085
skip_non_kubeadm_warning: true skip_non_kubeadm_warning: true

View file

@ -70,7 +70,7 @@
vars: vars:
etcd_cluster_setup: true etcd_cluster_setup: true
etcd_events_cluster_setup: "{{ etcd_events_cluster_enabled }}" etcd_events_cluster_setup: "{{ etcd_events_cluster_enabled }}"
when: not etcd_kubeadm_enabled | default(false) when: etcd_deployment_type != "kubeadm"
- hosts: k8s_cluster - hosts: k8s_cluster
gather_facts: False gather_facts: False
@ -83,7 +83,7 @@
vars: vars:
etcd_cluster_setup: false etcd_cluster_setup: false
etcd_events_cluster_setup: false etcd_events_cluster_setup: false
when: not etcd_kubeadm_enabled | default(false) when: etcd_deployment_type != "kubeadm"
- name: Handle upgrades to master components first to maintain backwards compat. - name: Handle upgrades to master components first to maintain backwards compat.
gather_facts: False gather_facts: False