360aff4a57
* rename ansible groups to use _ instead of - k8s-cluster -> k8s_cluster k8s-node -> k8s_node calico-rr -> calico_rr no-floating -> no_floating Note: kube-node,k8s-cluster groups in upgrade CI need clean-up after v2.16 is tagged * ensure old groups are mapped to the new ones |
||
---|---|---|
.. | ||
modules/kubernetes-cluster | ||
sample-inventory | ||
templates | ||
default.tfvars | ||
main.tf | ||
output.tf | ||
README.md | ||
variables.tf | ||
versions.tf |
Kubernetes on Exoscale with Terraform
Provision a Kubernetes cluster on Exoscale using Terraform and Kubespray
Overview
The setup looks like following
Kubernetes cluster
+-----------------------+
+---------------+ | +--------------+ |
| | | | +--------------+ |
| API server LB +---------> | | | |
| | | | | Master/etcd | |
+---------------+ | | | node(s) | |
| +-+ | |
| +--------------+ |
| ^ |
| | |
| v |
+---------------+ | +--------------+ |
| | | | +--------------+ |
| Ingress LB +---------> | | | |
| | | | | Worker | |
+---------------+ | | | node(s) | |
| +-+ | |
| +--------------+ |
+-----------------------+
Requirements
- Terraform 0.13.0 or newer
0.12 also works if you modify the provider block to include version and remove all versions.tf
files
Quickstart
NOTE: Assumes you are at the root of the kubespray repo
Copy the sample inventory for your cluster and copy the default terraform variables.
CLUSTER=my-exoscale-cluster
cp -r inventory/sample inventory/$CLUSTER
cp contrib/terraform/exoscale/default.tfvars inventory/$CLUSTER/
cd inventory/$CLUSTER
Edit default.tfvars
to match your setup. You MUST, at the very least, change ssh_public_keys
.
# Ensure $EDITOR points to your favorite editor, e.g., vim, emacs, VS Code, etc.
$EDITOR default.tfvars
For authentication you can use the credentials file ~/.cloudstack.ini
or ./cloudstack.ini
.
The file should look like something like this:
[cloudstack]
key = <API key>
secret = <API secret>
Follow the Exoscale IAM Quick-start to learn how to generate API keys.
Encrypted credentials
To have the credentials encrypted at rest, you can use sops and only decrypt the credentials at runtime.
cat << EOF > cloudstack.ini
[cloudstack]
key =
secret =
EOF
sops --encrypt --in-place --pgp <PGP key fingerprint> cloudstack.ini
sops cloudstack.ini
Run terraform to create the infrastructure
terraform init ../../contrib/terraform/exoscale
terraform apply -var-file default.tfvars ../../contrib/terraform/exoscale
If your cloudstack credentials file is encrypted using sops, run the following:
terraform init ../../contrib/terraform/exoscale
sops exec-file -no-fifo cloudstack.ini 'CLOUDSTACK_CONFIG={} terraform apply -var-file default.tfvars ../../contrib/terraform/exoscale'
You should now have a inventory file named inventory.ini
that you can use with kubespray.
You can now copy your inventory file and use it with kubespray to set up a cluster.
You can type terraform output
to find out the IP addresses of the nodes, as well as control-plane and data-plane load-balancer.
It is a good idea to check that you have basic SSH connectivity to the nodes. You can do that by:
ansible -i inventory.ini -m ping all
Example to use this with the default sample inventory:
ansible-playbook -i inventory.ini ../../cluster.yml -b -v
Teardown
The Kubernetes cluster cannot create any load-balancers or disks, hence, teardown is as simple as Terraform destroy:
terraform destroy -var-file default.tfvars ../../contrib/terraform/exoscale
Variables
Required
ssh_public_keys
: List of public SSH keys to install on all machineszone
: The zone where to run the clustermachines
: Machines to provision. Key of this object will be used as the name of the machinenode_type
: The role of this node (master|worker)size
: The size to useboot_disk
: The boot disk to useimage_name
: Name of the imageroot_partition_size
: Size (in GB) for the root partitionceph_partition_size
: Size (in GB) for the partition for rook to use as ceph storage. (Set to 0 to disable)node_local_partition_size
: Size (in GB) for the partition for node-local-storage. (Set to 0 to disable)
ssh_whitelist
: List of IP ranges (CIDR) that will be allowed to ssh to the nodesapi_server_whitelist
: List of IP ranges (CIDR) that will be allowed to connect to the API servernodeport_whitelist
: List of IP ranges (CIDR) that will be allowed to connect to the kubernetes nodes on port 30000-32767 (kubernetes nodeports)
Optional
prefix
: Prefix to use for all resources, required to be unique for all clusters in the same project (Defaults todefault
)
An example variables file can be found default.tfvars
Known limitations
Only single disk
Since Exoscale doesn't support additional disks to be mounted onto an instance, this script has the ability to create partitions for Rook and node-local-storage.
No Kubernetes API
The current solution doesn't use the Exoscale Kubernetes cloud controller. This means that we need to set up a HTTP(S) loadbalancer in front of all workers and set the Ingress controller to DaemonSet mode.