C12s/c12s-kubespray
3
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Add vSphere external cloud provider (#5959)

Browse source
This commit is contained in:
Pierre Lebrun 2020-04-21 00:47:39 +09:00 committed by GitHub
parent 536606c2ed
commit 03c8d0113c
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
22 changed files with 881 additions and 12 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

90
docs/vsphere-csi.md Normal file
View file

@ -0,0 +1,90 @@
# vSphere CSI Driver
vSphere CSI driver allows you to provision volumes over a vSphere deployment. The Kubernetes historic in-tree cloud provider is deprecated and will be removed in future versions.
To enable vSphere CSI driver, uncomment the `vsphere_csi_enabled` option in `group_vars/all/vsphere.yml` and set it to `true`.
To set the number of replicas for the vSphere CSI controller, you can change `vsphere_csi_controller_replicas` option in `group_vars/all/vsphere.yml`.
You need to source the vSphere credentials you use to deploy your machines that will host Kubernetes.
| Variable | Required | Type | Choices | Default | Comment |
|---------------------------------------------|----------|---------|----------------------------|---------------------------|----------------------------------------------------------------|
| external_vsphere_vcenter_ip | TRUE | string | | | IP/URL of the vCenter |
| external_vsphere_vcenter_port | TRUE | string | | "443" | Port of the vCenter API |
| external_vsphere_insecure | TRUE | string | "true", "false" | "true" | set to "true" if the host above uses a self-signed cert |
| external_vsphere_user | TRUE | string | | | User name for vCenter with required privileges |
| external_vsphere_password | TRUE | string | | | Password for vCenter |
| external_vsphere_datacenter | TRUE | string | | | Datacenter name to use |
| external_vsphere_kubernetes_cluster_id | TRUE | string | | "kubernetes-cluster-id" | Kubernetes cluster ID to use |
| vsphere_cloud_controller_image_tag | TRUE | string | | "latest" | Kubernetes cluster ID to use |
| vsphere_syncer_image_tag | TRUE | string | | "v1.0.2" | Syncer image tag to use |
| vsphere_csi_attacher_image_tag | TRUE | string | | "v1.1.1" | CSI attacher image tag to use |
| vsphere_csi_controller | TRUE | string | | "v1.0.2" | CSI controller image tag to use |
| vsphere_csi_controller_replicas | TRUE | integer | | 1 | Number of pods Kubernetes should deploy for the CSI controller |
| vsphere_csi_liveness_probe_image_tag | TRUE | string | | "v1.1.0" | CSI liveness probe image tag to use |
| vsphere_csi_provisioner_image_tag | TRUE | string | | "v1.2.2" | CSI provisioner image tag to use |
| vsphere_csi_node_driver_registrar_image_tag | TRUE | string | | "v1.1.0" | CSI node driver registrat image tag to use |
| vsphere_csi_driver_image_tag | TRUE | string | | "v1.0.2" | CSI driver image tag to use |
## Usage example
To test the dynamic provisioning using vSphere CSI driver, make sure to create a [storage policy](https://github.com/kubernetes/cloud-provider-vsphere/blob/master/docs/book/tutorials/kubernetes-on-vsphere-with-kubeadm.md#create-a-storage-policy) and [storage class](https://github.com/kubernetes/cloud-provider-vsphere/blob/master/docs/book/tutorials/kubernetes-on-vsphere-with-kubeadm.md#create-a-storageclass), then apply the following manifest:
```yml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-pvc-vsphere
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: Space-Efficient
---
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- image: nginx
imagePullPolicy: IfNotPresent
name: nginx
ports:
- containerPort: 80
protocol: TCP
volumeMounts:
- mountPath: /var/lib/www/html
name: csi-data-vsphere
volumes:
- name: csi-data-vsphere
persistentVolumeClaim:
claimName: csi-pvc-vsphere
readOnly: false
```
Apply this conf to your cluster: ```kubectl apply -f nginx.yml```
You should see the PVC provisioned and bound:
```ShellSession
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
csi-pvc-vsphere Bound pvc-dc7b1d21-ee41-45e1-98d9-e877cc1533ac 1Gi RWO Space-Efficient 10s
```
And the volume mounted to the Nginx Pod (wait until the Pod is Running):
```ShellSession
kubectl exec -it nginx -- df -h | grep /var/lib/www/html
/dev/sdb 976M 2.6M 907M 1% /var/lib/www/html
```
## More info
For further information about the vSphere CSI Driver, you can refer to the official [vSphere Cloud Provider documentation](https://cloud-provider-vsphere.sigs.k8s.io/container_storage_interface.html).

82
docs/vsphere.md
View file

@ -1,13 +1,75 @@
# vSphere cloud provider # vSphere
Kubespray can be deployed with vSphere as Cloud provider. This feature supports Kubespray can be deployed with vSphere as Cloud provider. This feature supports:
- Volumes - Volumes
- Persistent Volumes - Persistent Volumes
- Storage Classes and provisioning of volumes. - Storage Classes and provisioning of volumes
- vSphere Storage Policy Based Management for Containers orchestrated by Kubernetes. - vSphere Storage Policy Based Management for Containers orchestrated by Kubernetes
## Prerequisites ## Out-of-tree vSphere cloud provider
### Prerequisites
You need at first to configure your vSphere environment by following the [official documentation](https://github.com/kubernetes/cloud-provider-vsphere/blob/master/docs/book/tutorials/kubernetes-on-vsphere-with-kubeadm.md#prerequisites).
After this step you should have:
- vSphere upgraded to 6.7 U3 or later
- VM hardware upgraded to version 15 or higher
- UUID activated for each VM where Kubernetes will be deployed
### Kubespray configuration
First in `inventory/sample/group_vars/all.yml` you must set the cloud provider to `external` and external_cloud_provider to `external_cloud_provider`.
```yml
cloud_provider: "external"
external_cloud_provider: "vsphere"
```
Then, `inventory/sample/group_vars/vsphere.yml`, you need to declare your vCenter credentials and enable the vSphere CSI following the description below.
| Variable | Required | Type | Choices | Default | Comment |
|----------------------------------------|----------|---------|----------------------------|---------|---------------------------------------------------------------------------|
| external_vsphere_vcenter_ip | TRUE | string | | | IP/URL of the vCenter |
| external_vsphere_vcenter_port | TRUE | string | | "443" | Port of the vCenter API |
| external_vsphere_insecure | TRUE | string | "true", "false" | "true" | set to "true" if the host above uses a self-signed cert |
| external_vsphere_user | TRUE | string | | | User name for vCenter with required privileges |
| external_vsphere_password | TRUE | string | | | Password for vCenter |
| external_vsphere_datacenter | TRUE | string | | | Datacenter name to use |
| external_vsphere_kubernetes_cluster_id | TRUE | string | | "kubernetes-cluster-id" | Kubernetes cluster ID to use |
| vsphere_csi_enabled | TRUE | boolean | | false | Enable vSphere CSI |
Example configuration:
```yml
external_vsphere_vcenter_ip: "myvcenter.domain.com"
external_vsphere_vcenter_port: "443"
external_vsphere_insecure: "true"
external_vsphere_user: "administrator@vsphere.local"
external_vsphere_password: "K8s_admin"
external_vsphere_datacenter: "DATACENTER_name"
external_vsphere_kubernetes_cluster_id: "kubernetes-cluster-id"
vsphere_csi_enabled: true
```
For a more fine-grained CSI setup, refer to the [vsphere-csi](vsphere-csi.md) documentation.
### Deployment
Once the configuration is set, you can execute the playbook again to apply the new configuration:
```ShellSession
cd kubespray
ansible-playbook -i inventory/sample/hosts.ini -b -v cluster.yml
```
You'll find some useful examples [here](https://github.com/kubernetes/cloud-provider-vsphere/blob/master/docs/book/tutorials/kubernetes-on-vsphere-with-kubeadm.md#sample-manifests-to-test-csi-driver-functionality) to test your configuration.
## In-tree vSphere cloud provider ([deprecated](https://cloud-provider-vsphere.sigs.k8s.io/concepts/in_tree_vs_out_of_tree.html))
### Prerequisites (deprecated)
You need at first to configure your vSphere environment by following the [official documentation](https://kubernetes.io/docs/getting-started-guides/vsphere/#vsphere-cloud-provider). You need at first to configure your vSphere environment by following the [official documentation](https://kubernetes.io/docs/getting-started-guides/vsphere/#vsphere-cloud-provider).
@ -18,7 +80,7 @@ After this step you should have:
If you intend to leverage the [zone and region node labeling](https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#failure-domain-beta-kubernetes-io-region), create a tag category for both the zone and region in vCenter. The tags can then be applied at the host, cluster, datacenter, or folder level, and the cloud provider will walk the hierarchy to extract and apply the labels to the Kubernetes nodes. If you intend to leverage the [zone and region node labeling](https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#failure-domain-beta-kubernetes-io-region), create a tag category for both the zone and region in vCenter. The tags can then be applied at the host, cluster, datacenter, or folder level, and the cloud provider will walk the hierarchy to extract and apply the labels to the Kubernetes nodes.
## Kubespray configuration ### Kubespray configuration (deprecated)
First you must define the cloud provider in `inventory/sample/group_vars/all.yml` and set it to `vsphere`. First you must define the cloud provider in `inventory/sample/group_vars/all.yml` and set it to `vsphere`.
@ -26,7 +88,7 @@ First you must define the cloud provider in `inventory/sample/group_vars/all.yml
cloud_provider: vsphere cloud_provider: vsphere
``` ```
Then, in the same file, you need to declare your vCenter credential following the description below. Then, in the same file, you need to declare your vCenter credentials following the description below.
| Variable | Required | Type | Choices | Default | Comment | | Variable | Required | Type | Choices | Default | Comment |
|------------------------------|----------|---------|----------------------------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |------------------------------|----------|---------|----------------------------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
@ -45,7 +107,7 @@ Then, in the same file, you need to declare your vCenter credential following th
| vsphere_zone_category | FALSE | string | | | Name of the tag category used to set the `failure-domain.beta.kubernetes.io/zone` label on nodes (Optional, only used for Kubernetes >= 1.12.0) | | vsphere_zone_category | FALSE | string | | | Name of the tag category used to set the `failure-domain.beta.kubernetes.io/zone` label on nodes (Optional, only used for Kubernetes >= 1.12.0) |
| vsphere_region_category | FALSE | string | | | Name of the tag category used to set the `failure-domain.beta.kubernetes.io/region` label on nodes (Optional, only used for Kubernetes >= 1.12.0) | | vsphere_region_category | FALSE | string | | | Name of the tag category used to set the `failure-domain.beta.kubernetes.io/region` label on nodes (Optional, only used for Kubernetes >= 1.12.0) |
Example configuration Example configuration:
```yml ```yml
vsphere_vcenter_ip: "myvcenter.domain.com" vsphere_vcenter_ip: "myvcenter.domain.com"
@ -60,9 +122,9 @@ vsphere_scsi_controller_type: "pvscsi"
vsphere_resource_pool: "K8s-Pool" vsphere_resource_pool: "K8s-Pool"
``` ```
## Deployment ### Deployment (deprecated)
Once the configuration is set, you can execute the playbook again to apply the new configuration Once the configuration is set, you can execute the playbook again to apply the new configuration:
```ShellSession ```ShellSession
cd kubespray cd kubespray

4
inventory/sample/group_vars/all/all.yml
View file

@ -54,8 +54,8 @@ loadbalancer_apiserver_healthcheck_port: 8081
# cloud_provider: # cloud_provider:
## When cloud_provider is set to 'external', you can set the cloud controller to deploy ## When cloud_provider is set to 'external', you can set the cloud controller to deploy
## Supported cloud controllers are: 'openstack' ## Supported cloud controllers are: 'openstack' and 'vsphere'
## When openstack is used make sure to source in the openstack credentials ## When openstack or vsphere are used make sure to source in the required fields
# external_cloud_provider: # external_cloud_provider:
## Set these proxy values in order to update package manager and docker daemon to use proxies ## Set these proxy values in order to update package manager and docker daemon to use proxies

20
inventory/sample/group_vars/all/vsphere.yml Normal file
View file

@ -0,0 +1,20 @@
## Values for the external vSphere Cloud Provider
# external_vsphere_vcenter_ip: "myvcenter.domain.com"
# external_vsphere_vcenter_port: "443"
# external_vsphere_insecure: "true"
# external_vsphere_user: "administrator@vsphere.local"
# external_vsphere_password: "K8s_admin"
# external_vsphere_datacenter: "DATACENTER_name"
# external_vsphere_kubernetes_cluster_id: "kubernetes-cluster-id"
## Tags for the external vSphere Cloud Provider images
# external_vsphere_cloud_controller_image_tag: "latest"
# vsphere_syncer_image_tag: "v1.0.2"
# vsphere_csi_attacher_image_tag: "v1.1.1"
# vsphere_csi_controller: "v1.0.2"
# vsphere_csi_liveness_probe_image_tag: "v1.1.0"
# vsphere_csi_provisioner_image_tag: "v1.2.2"
## To use vSphere CSI plugin to provision volumes set this value to true
# vsphere_csi_enabled: true
# vsphere_csi_controller_replicas: 1

14
roles/kubernetes-apps/csi_driver/vsphere/defaults/main.yml Normal file
View file

@ -0,0 +1,14 @@
---
external_vsphere_vcenter_port: "443"
external_vsphere_insecure: "true"
external_vsphere_kubernetes_cluster_id: "kubernetes-cluster-id"
vsphere_syncer_image_tag: "v1.0.2"
vsphere_csi_attacher_image_tag: "v1.1.1"
vsphere_csi_controller: "v1.0.2"
vsphere_csi_liveness_probe_image_tag: "v1.1.0"
vsphere_csi_provisioner_image_tag: "v1.2.2"
vsphere_csi_node_driver_registrar_image_tag: "v1.1.0"
vsphere_csi_driver_image_tag: "v1.0.2"
vsphere_csi_controller_replicas: 1

44
roles/kubernetes-apps/csi_driver/vsphere/tasks/main.yml Normal file
View file

@ -0,0 +1,44 @@
---
- include_tasks: vsphere-credentials-check.yml
tags: vsphere-csi-driver
- name: vSphere CSI Driver | Generate CSI cloud-config
template:
src: "{{ item }}.j2"
dest: "{{ kube_config_dir }}/{{ item }}"
mode: 0640
with_items:
- vsphere-csi-cloud-config
when: inventory_hostname == groups['kube-master'][0]
tags: vsphere-csi-driver
- name: vSphere CSI Driver | Generate Manifests
template:
src: "{{ item }}.j2"
dest: "{{ kube_config_dir }}/{{ item }}"
with_items:
- vsphere-csi-controller-rbac.yml
- vsphere-csi-controller-ss.yml
- vsphere-csi-node.yml
register: vsphere_csi_manifests
when: inventory_hostname == groups['kube-master'][0]
tags: vsphere-csi-driver
- name: vSphere CSI Driver | Create a CSI secret
command: "{{ bin_dir }}/kubectl create secret generic vsphere-config-secret --from-file=csi-vsphere.conf={{ kube_config_dir }}/vsphere-csi-cloud-config -n kube-system"
when: inventory_hostname == groups['kube-master'][0]
tags: vsphere-csi-driver
- name: vSphere CSI Driver | Apply Manifests
kube:
kubectl: "{{ bin_dir }}/kubectl"
filename: "{{ kube_config_dir }}/{{ item.item }}"
state: "latest"
with_items:
- "{{ vsphere_csi_manifests.results }}"
when:
- inventory_hostname == groups['kube-master'][0]
- not item is skipped
loop_control:
label: "{{ item.item }}"
tags: vsphere-csi-driver

38
roles/kubernetes-apps/csi_driver/vsphere/tasks/vsphere-credentials-check.yml Normal file
View file

@ -0,0 +1,38 @@
---
- name: External vSphere Cloud Provider | check external_vsphere_vcenter_ip value
fail:
msg: "external_vsphere_vcenter_ip is missing"
when: external_vsphere_vcenter_ip is not defined or not external_vsphere_vcenter_ip
- name: External vSphere Cloud Provider | check external_vsphere_vcenter_port value
fail:
msg: "external_vsphere_vcenter_port is missing"
when: external_vsphere_vcenter_port is not defined or not external_vsphere_vcenter_port
- name: External vSphere Cloud Provider | check external_vsphere_insecure value
fail:
msg: "external_vsphere_insecure is missing"
when: external_vsphere_insecure is not defined or not external_vsphere_insecure
- name: External vSphere Cloud Provider | check external_vsphere_user value
fail:
msg: "external_vsphere_user is missing"
when: external_vsphere_user is not defined or not external_vsphere_user
- name: External vSphere Cloud Provider | check external_vsphere_password value
fail:
msg: "external_vsphere_password is missing"
when:
- external_vsphere_password is not defined or not external_vsphere_password
- name: External vSphere Cloud Provider | check external_vsphere_datacenter value
fail:
msg: "external_vsphere_datacenter is missing"
when:
- external_vsphere_datacenter is not defined or not external_vsphere_datacenter
- name: External vSphere Cloud Provider | check external_vsphere_kubernetes_cluster_id value
fail:
msg: "external_vsphere_kubernetes_cluster_id is missing"
when:
- external_vsphere_kubernetes_cluster_id is not defined or not external_vsphere_kubernetes_cluster_id

9
roles/kubernetes-apps/csi_driver/vsphere/templates/vsphere-csi-cloud-config.j2 Normal file
View file

@ -0,0 +1,9 @@
[Global]
cluster-id = "{{ external_vsphere_kubernetes_cluster_id }}"
[VirtualCenter "{{ external_vsphere_vcenter_ip }}"]
insecure-flag = "{{ external_vsphere_insecure }}"
user = "{{ external_vsphere_user }}"
password = "{{ external_vsphere_password }}"
port = "{{ external_vsphere_vcenter_port }}"
datacenters = "{{ external_vsphere_datacenter }}"

42
roles/kubernetes-apps/csi_driver/vsphere/templates/vsphere-csi-controller-rbac.yml.j2 Normal file
View file

@ -0,0 +1,42 @@
kind: ServiceAccount
apiVersion: v1
metadata:
name: vsphere-csi-controller
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: vsphere-csi-controller-role
rules:
- apiGroups: [""]
resources: ["nodes", "persistentvolumeclaims", "pods"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "update", "delete"]
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["csinodes"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list", "watch", "update"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: vsphere-csi-controller-binding
subjects:
- kind: ServiceAccount
name: vsphere-csi-controller
namespace: kube-system
roleRef:
kind: ClusterRole
name: vsphere-csi-controller-role
apiGroup: rbac.authorization.k8s.io

128
roles/kubernetes-apps/csi_driver/vsphere/templates/vsphere-csi-controller-ss.yml.j2 Normal file
View file

@ -0,0 +1,128 @@
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: vsphere-csi-controller
namespace: kube-system
spec:
serviceName: vsphere-csi-controller
replicas: {{ vsphere_csi_controller_replicas }}
updateStrategy:
type: "RollingUpdate"
selector:
matchLabels:
app: vsphere-csi-controller
template:
metadata:
labels:
app: vsphere-csi-controller
role: vsphere-csi
spec:
serviceAccountName: vsphere-csi-controller
nodeSelector:
node-role.kubernetes.io/master: ""
tolerations:
- operator: "Exists"
key: node-role.kubernetes.io/master
effect: NoSchedule
dnsPolicy: "Default"
containers:
- name: csi-attacher
image: {{ quay_image_repo }}/k8scsi/csi-attacher:{{ vsphere_csi_attacher_image_tag }}
args:
- "--v=4"
- "--timeout=300s"
- "--csi-address=$(ADDRESS)"
env:
- name: ADDRESS
value: /csi/csi.sock
volumeMounts:
- mountPath: /csi
name: socket-dir
- name: vsphere-csi-controller
image: {{ gcr_image_repo }}/cloud-provider-vsphere/csi/release/driver:{{ vsphere_csi_controller }}
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "rm -rf /var/lib/csi/sockets/pluginproxy/csi.vsphere.vmware.com"]
args:
- "--v=4"
imagePullPolicy: "Always"
env:
- name: CSI_ENDPOINT
value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
- name: X_CSI_MODE
value: "controller"
- name: VSPHERE_CSI_CONFIG
value: "/etc/cloud/csi-vsphere.conf"
volumeMounts:
- mountPath: /etc/cloud
name: vsphere-config-volume
readOnly: true
- mountPath: /var/lib/csi/sockets/pluginproxy/
name: socket-dir
ports:
- name: healthz
containerPort: 9808
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: healthz
initialDelaySeconds: 10
timeoutSeconds: 3
periodSeconds: 5
failureThreshold: 3
- name: liveness-probe
image: {{ quay_image_repo }}/k8scsi/livenessprobe:{{ vsphere_csi_liveness_probe_image_tag }}
args:
- "--csi-address=$(ADDRESS)"
env:
- name: ADDRESS
value: /var/lib/csi/sockets/pluginproxy/csi.sock
volumeMounts:
- mountPath: /var/lib/csi/sockets/pluginproxy/
name: socket-dir
- name: vsphere-syncer
image: {{ gcr_image_repo }}/cloud-provider-vsphere/csi/release/syncer:{{ vsphere_syncer_image_tag }}
args:
- "--v=2"
imagePullPolicy: "Always"
env:
- name: FULL_SYNC_INTERVAL_MINUTES
value: "30"
- name: VSPHERE_CSI_CONFIG
value: "/etc/cloud/csi-vsphere.conf"
volumeMounts:
- mountPath: /etc/cloud
name: vsphere-config-volume
readOnly: true
- name: csi-provisioner
image: {{ quay_image_repo }}/k8scsi/csi-provisioner:{{ vsphere_csi_provisioner_image_tag }}
args:
- "--v=4"
- "--timeout=300s"
- "--csi-address=$(ADDRESS)"
- "--feature-gates=Topology=true"
- "--strict-topology"
env:
- name: ADDRESS
value: /csi/csi.sock
volumeMounts:
- mountPath: /csi
name: socket-dir
volumes:
- name: vsphere-config-volume
secret:
secretName: vsphere-config-secret
- name: socket-dir
hostPath:
path: /var/lib/csi/sockets/pluginproxy/csi.vsphere.vmware.com
type: DirectoryOrCreate
---
apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
name: csi.vsphere.vmware.com
spec:
attachRequired: true
podInfoOnMount: false

121
roles/kubernetes-apps/csi_driver/vsphere/templates/vsphere-csi-node.yml.j2 Normal file
View file

@ -0,0 +1,121 @@
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: vsphere-csi-node
namespace: kube-system
spec:
selector:
matchLabels:
app: vsphere-csi-node
updateStrategy:
type: "RollingUpdate"
template:
metadata:
labels:
app: vsphere-csi-node
role: vsphere-csi
spec:
dnsPolicy: "Default"
containers:
- name: node-driver-registrar
image: {{ quay_image_repo }}/k8scsi/csi-node-driver-registrar:{{ vsphere_csi_node_driver_registrar_image_tag }}
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "rm -rf /registration/csi.vsphere.vmware.com /var/lib/kubelet/plugins_registry/csi.vsphere.vmware.com /var/lib/kubelet/plugins_registry/csi.vsphere.vmware.com-reg.sock"]
args:
- "--v=5"
- "--csi-address=$(ADDRESS)"
- "--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)"
env:
- name: ADDRESS
value: /csi/csi.sock
- name: DRIVER_REG_SOCK_PATH
value: /var/lib/kubelet/plugins_registry/csi.vsphere.vmware.com/csi.sock
securityContext:
privileged: true
volumeMounts:
- name: plugin-dir
mountPath: /csi
- name: registration-dir
mountPath: /registration
- name: vsphere-csi-node
image: {{ gcr_image_repo }}/cloud-provider-vsphere/csi/release/driver:{{ vsphere_csi_driver_image_tag }}
imagePullPolicy: "Always"
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
- name: X_CSI_MODE
value: "node"
- name: X_CSI_SPEC_REQ_VALIDATION
value: "false"
# needed only for topology aware setups
#- name: VSPHERE_CSI_CONFIG
# value: "/etc/cloud/csi-vsphere.conf" # here csi-vsphere.conf is the name of the file used for creating secret using "--from-file" flag
args:
- "--v=4"
securityContext:
privileged: true
capabilities:
add: ["SYS_ADMIN"]
allowPrivilegeEscalation: true
volumeMounts:
# needed only for topology aware setups
#- name: vsphere-config-volume
# mountPath: /etc/cloud
# readOnly: true
- name: plugin-dir
mountPath: /csi
- name: pods-mount-dir
mountPath: /var/lib/kubelet
# needed so that any mounts setup inside this container are
# propagated back to the host machine.
mountPropagation: "Bidirectional"
- name: device-dir
mountPath: /dev
ports:
- name: healthz
containerPort: 9808
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: healthz
initialDelaySeconds: 10
timeoutSeconds: 3
periodSeconds: 5
failureThreshold: 3
- name: liveness-probe
image: {{ quay_image_repo }}/k8scsi/livenessprobe:{{ vsphere_csi_liveness_probe_image_tag }}
args:
- "--csi-address=$(ADDRESS)"
env:
- name: ADDRESS
value: /csi/csi.sock
volumeMounts:
- name: plugin-dir
mountPath: /csi
volumes:
# needed only for topology aware setups
#- name: vsphere-config-volume
# secret:
# secretName: vsphere-config-secret
- name: registration-dir
hostPath:
path: /var/lib/kubelet/plugins_registry
type: DirectoryOrCreate
- name: plugin-dir
hostPath:
path: /var/lib/kubelet/plugins_registry/csi.vsphere.vmware.com
type: DirectoryOrCreate
- name: pods-mount-dir
hostPath:
path: /var/lib/kubelet
type: Directory
- name: device-dir
hostPath:
path: /dev

10
roles/kubernetes-apps/external_cloud_controller/meta/main.yml
View file

@ -10,3 +10,13 @@ dependencies:
tags: tags:
- external-cloud-controller - external-cloud-controller
- external-openstack - external-openstack
- role: kubernetes-apps/external_cloud_controller/vsphere
when:
- cloud_provider is defined
- cloud_provider == "external"
- external_cloud_provider is defined
- external_cloud_provider == "vsphere"
- inventory_hostname == groups['kube-master'][0]
tags:
- external-cloud-controller
- external-vsphere

5
roles/kubernetes-apps/external_cloud_controller/vsphere/defaults/main.yml Normal file
View file

@ -0,0 +1,5 @@
---
external_vsphere_vcenter_port: "443"
external_vsphere_insecure: "true"
external_vsphere_cloud_controller_image_tag: "latest"

45
roles/kubernetes-apps/external_cloud_controller/vsphere/tasks/main.yml Normal file
View file

@ -0,0 +1,45 @@
---
- include_tasks: vsphere-credentials-check.yml
tags: external-vsphere
- name: External vSphere Cloud Controller | Generate CPI cloud-config
template:
src: "{{ item }}.j2"
dest: "{{ kube_config_dir }}/{{ item }}"
mode: 0640
with_items:
- external-vsphere-cpi-cloud-config
when: inventory_hostname == groups['kube-master'][0]
tags: external-vsphere
- name: External vSphere Cloud Controller | Generate Manifests
template:
src: "{{ item }}.j2"
dest: "{{ kube_config_dir }}/{{ item }}"
with_items:
- external-vsphere-cpi-cloud-config-secret.yml
- external-vsphere-cloud-controller-manager-roles.yml
- external-vsphere-cloud-controller-manager-role-bindings.yml
- external-vsphere-cloud-controller-manager-ds.yml
register: external_vsphere_manifests
when: inventory_hostname == groups['kube-master'][0]
tags: external-vsphere
- name: External vSphere Cloud Provider Interface | Create a CPI configMap
command: "{{ bin_dir }}/kubectl create configmap cloud-config --from-file=vsphere.conf={{ kube_config_dir }}/external-vsphere-cpi-cloud-config -n kube-system"
when: inventory_hostname == groups['kube-master'][0]
tags: external-vsphere
- name: External vSphere Cloud Controller | Apply Manifests
kube:
kubectl: "{{ bin_dir }}/kubectl"
filename: "{{ kube_config_dir }}/{{ item.item }}"
state: "latest"
with_items:
- "{{ external_vsphere_manifests.results }}"
when:
- inventory_hostname == groups['kube-master'][0]
- not item is skipped
loop_control:
label: "{{ item.item }}"
tags: external-vsphere

32
roles/kubernetes-apps/external_cloud_controller/vsphere/tasks/vsphere-credentials-check.yml Normal file
View file

@ -0,0 +1,32 @@
---
- name: External vSphere Cloud Provider | check external_vsphere_vcenter_ip value
fail:
msg: "external_vsphere_vcenter_ip is missing"
when: external_vsphere_vcenter_ip is not defined or not external_vsphere_vcenter_ip
- name: External vSphere Cloud Provider | check external_vsphere_vcenter_port value
fail:
msg: "external_vsphere_vcenter_port is missing"
when: external_vsphere_vcenter_port is not defined or not external_vsphere_vcenter_port
- name: External vSphere Cloud Provider | check external_vsphere_insecure value
fail:
msg: "external_vsphere_insecure is missing"
when: external_vsphere_insecure is not defined or not external_vsphere_insecure
- name: External vSphere Cloud Provider | check external_vsphere_user value
fail:
msg: "external_vsphere_user is missing"
when: external_vsphere_user is not defined or not external_vsphere_user
- name: External vSphere Cloud Provider | check external_vsphere_password value
fail:
msg: "external_vsphere_password is missing"
when:
- external_vsphere_password is not defined or not external_vsphere_password
- name: External vSphere Cloud Provider | check external_vsphere_datacenter value
fail:
msg: "external_vsphere_datacenter is missing"
when:
- external_vsphere_datacenter is not defined or not external_vsphere_datacenter

71
roles/kubernetes-apps/external_cloud_controller/vsphere/templates/external-vsphere-cloud-controller-manager-ds.yml.j2 Normal file
View file

@ -0,0 +1,71 @@
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cloud-controller-manager
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: vsphere-cloud-controller-manager
namespace: kube-system
labels:
k8s-app: vsphere-cloud-controller-manager
spec:
selector:
matchLabels:
k8s-app: vsphere-cloud-controller-manager
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
k8s-app: vsphere-cloud-controller-manager
spec:
nodeSelector:
node-role.kubernetes.io/master: ""
securityContext:
runAsUser: 0
tolerations:
- key: node.cloudprovider.kubernetes.io/uninitialized
value: "true"
effect: NoSchedule
- key: node-role.kubernetes.io/master
effect: NoSchedule
serviceAccountName: cloud-controller-manager
containers:
- name: vsphere-cloud-controller-manager
image: {{ gcr_image_repo }}/cloud-provider-vsphere/cpi/release/manager:{{ external_vsphere_cloud_controller_image_tag }}
args:
- --v=2
- --cloud-provider=vsphere
- --cloud-config=/etc/cloud/vsphere.conf
volumeMounts:
- mountPath: /etc/cloud
name: vsphere-config-volume
readOnly: true
resources:
requests:
cpu: 200m
hostNetwork: true
volumes:
- name: vsphere-config-volume
configMap:
name: cloud-config
---
apiVersion: v1
kind: Service
metadata:
labels:
component: cloud-controller-manager
name: vsphere-cloud-controller-manager
namespace: kube-system
spec:
type: NodePort
ports:
- port: 43001
protocol: TCP
targetPort: 43001
selector:
component: cloud-controller-manager

35
roles/kubernetes-apps/external_cloud_controller/vsphere/templates/external-vsphere-cloud-controller-manager-role-bindings.yml.j2 Normal file
View file

@ -0,0 +1,35 @@
apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: servicecatalog.k8s.io:apiserver-authentication-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- apiGroup: ""
kind: ServiceAccount
name: cloud-controller-manager
namespace: kube-system
- apiGroup: ""
kind: User
name: cloud-controller-manager
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:cloud-controller-manager
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:cloud-controller-manager
subjects:
- kind: ServiceAccount
name: cloud-controller-manager
namespace: kube-system
- kind: User
name: cloud-controller-manager
kind: List
metadata: {}

75
roles/kubernetes-apps/external_cloud_controller/vsphere/templates/external-vsphere-cloud-controller-manager-roles.yml.j2 Normal file
View file

@ -0,0 +1,75 @@
apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:cloud-controller-manager
rules:
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- update
- apiGroups:
- ""
resources:
- nodes
verbs:
- '*'
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
- apiGroups:
- ""
resources:
- services
verbs:
- list
- patch
- update
- watch
- apiGroups:
- ""
resources:
- serviceaccounts
verbs:
- create
- get
- list
- watch
- update
- apiGroups:
- ""
resources:
- persistentvolumes
verbs:
- get
- list
- update
- watch
- apiGroups:
- ""
resources:
- endpoints
verbs:
- create
- get
- list
- watch
- update
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- watch
kind: List
metadata: {}

11
roles/kubernetes-apps/external_cloud_controller/vsphere/templates/external-vsphere-cpi-cloud-config-secret.yml.j2 Normal file
View file

@ -0,0 +1,11 @@
# This YAML file contains secret objects,
# which are necessary to run external vsphere cloud controller.
apiVersion: v1
kind: Secret
metadata:
name: cpi-global-secret
namespace: kube-system
stringData:
{{ external_vsphere_vcenter_ip }}.username: "{{ external_vsphere_user }}"
{{ external_vsphere_vcenter_ip }}.password: "{{ external_vsphere_password }}"

8
roles/kubernetes-apps/external_cloud_controller/vsphere/templates/external-vsphere-cpi-cloud-config.j2 Normal file
View file

@ -0,0 +1,8 @@
[Global]
port = "{{ external_vsphere_vcenter_port }}"
insecure-flag = "{{ external_vsphere_insecure }}"
secret-name = "cpi-global-secret"
secret-namespace = "kube-system"
[VirtualCenter "{{ external_vsphere_vcenter_ip }}"]
datacenters = "{{ external_vsphere_datacenter }}"

8
roles/kubernetes-apps/meta/main.yml
View file

@ -61,6 +61,14 @@ dependencies:
- gcp-pd-csi-driver - gcp-pd-csi-driver
- csi-driver - csi-driver
- role: kubernetes-apps/csi_driver/vsphere
when:
- vsphere_csi_enabled
tags:
- apps
- vsphere-csi-driver
- csi-driver
- role: kubernetes-apps/persistent_volumes - role: kubernetes-apps/persistent_volumes
when: when:
- persistent_volumes_enabled - persistent_volumes_enabled

1
roles/kubespray-defaults/defaults/main.yaml
View file

@ -310,6 +310,7 @@ cinder_csi_enabled: false
aws_ebs_csi_enabled: false aws_ebs_csi_enabled: false
azure_csi_enabled: false azure_csi_enabled: false
gcp_pd_csi_enabled: false gcp_pd_csi_enabled: false
vsphere_csi_enabled: false
persistent_volumes_enabled: false persistent_volumes_enabled: false
cephfs_provisioner_enabled: false cephfs_provisioner_enabled: false
rbd_provisioner_enabled: false rbd_provisioner_enabled: false