2020-07-22 06:54:05 +00:00
# Kubernetes on OpenStack with Terraform
2016-06-05 20:52:20 +00:00
Provision a Kubernetes cluster with [Terraform ](https://www.terraform.io ) on
2020-07-22 06:54:05 +00:00
OpenStack.
2016-06-05 20:52:20 +00:00
## Status
2020-07-22 06:54:05 +00:00
This will install a Kubernetes cluster on an OpenStack Cloud. It should work on
2017-12-05 12:48:47 +00:00
most modern installs of OpenStack that support the basic services.
2018-10-18 09:02:16 +00:00
### Known compatible public clouds
2020-12-25 20:10:27 +00:00
2018-10-18 09:02:16 +00:00
- [Auro ](https://auro.io/ )
2019-02-26 03:38:32 +00:00
- [Betacloud ](https://www.betacloud.io/ )
2018-10-18 09:02:16 +00:00
- [CityCloud ](https://www.citycloud.com/ )
- [DreamHost ](https://www.dreamhost.com/cloud/computing/ )
2018-11-05 11:39:41 +00:00
- [ELASTX ](https://elastx.se/ )
2018-10-18 09:02:16 +00:00
- [EnterCloudSuite ](https://www.entercloudsuite.com/ )
- [FugaCloud ](https://fuga.cloud/ )
2019-08-18 08:30:05 +00:00
- [Open Telekom Cloud ](https://cloud.telekom.de/ ) : requires to set the variable `wait_for_floatingip = "true"` in your cluster.tfvars
2018-10-19 10:39:34 +00:00
- [OVH ](https://www.ovh.com/ )
- [Rackspace ](https://www.rackspace.com/ )
2018-10-18 09:02:16 +00:00
- [Ultimum ](https://ultimum.io/ )
- [VexxHost ](https://vexxhost.com/ )
- [Zetta ](https://www.zetta.io/ )
2017-12-05 12:48:47 +00:00
## Approach
2020-12-25 20:10:27 +00:00
2017-12-05 12:48:47 +00:00
The terraform configuration inspects variables found in
[variables.tf ](variables.tf ) to create resources in your OpenStack cluster.
There is a [python script ](../terraform.py ) that reads the generated`.tfstate`
file to generate a dynamic inventory that is consumed by the main ansible script
to actually install kubernetes and stand up the cluster.
### Networking
2020-12-25 20:10:27 +00:00
2017-12-05 12:48:47 +00:00
The configuration includes creating a private subnet with a router to the
2018-02-23 20:08:45 +00:00
external net. It will allocate floating IPs from a pool and assign them to the
2017-12-05 12:48:47 +00:00
hosts where that makes sense. You have the option of creating bastion hosts
2018-02-23 20:08:45 +00:00
inside the private subnet to access the nodes there. Alternatively, a node with
a floating IP can be used as a jump host to nodes without.
2017-12-05 12:48:47 +00:00
2020-04-06 09:41:46 +00:00
#### Using an existing router
2020-12-25 20:10:27 +00:00
2020-04-06 09:41:46 +00:00
It is possible to use an existing router instead of creating one. To use an
existing router set the router\_id variable to the uuid of the router you wish
to use.
For example:
2020-12-25 20:10:27 +00:00
```ShellSession
2020-04-06 09:41:46 +00:00
router_id = "00c542e7-6f46-4535-ae95-984c7f0391a3"
```
2017-12-05 12:48:47 +00:00
### Kubernetes Nodes
2020-12-25 20:10:27 +00:00
2017-12-05 12:48:47 +00:00
You can create many different kubernetes topologies by setting the number of
different classes of hosts. For each class there are options for allocating
2018-02-23 20:08:45 +00:00
floating IP addresses or not.
2020-12-25 20:10:27 +00:00
2018-02-23 20:08:45 +00:00
- Master nodes with etcd
2017-12-05 12:48:47 +00:00
- Master nodes without etcd
- Standalone etcd hosts
- Kubernetes worker nodes
2018-02-23 20:08:45 +00:00
Note that the Ansible script will report an invalid configuration if you wind up
2018-07-31 19:15:49 +00:00
with an even number of etcd instances since that is not a valid configuration. This
restriction includes standalone etcd nodes that are deployed in a cluster along with
2019-02-11 22:10:10 +00:00
master nodes with etcd replicas. As an example, if you have three master nodes with
etcd replicas and three standalone etcd nodes, the script will fail since there are
2018-07-31 19:15:49 +00:00
now six total etcd replicas.
2017-12-05 12:48:47 +00:00
2020-12-25 20:10:27 +00:00
### GlusterFS shared file system
2018-02-23 20:08:45 +00:00
The Terraform configuration supports provisioning of an optional GlusterFS
2017-12-05 12:48:47 +00:00
shared file system based on a separate set of VMs. To enable this, you need to
2018-02-23 20:08:45 +00:00
specify:
2020-12-25 20:10:27 +00:00
2018-02-23 20:08:45 +00:00
- the number of Gluster hosts (minimum 2)
2017-12-05 12:48:47 +00:00
- Size of the non-ephemeral volumes to be attached to store the GlusterFS bricks
- Other properties related to provisioning the hosts
2020-08-28 09:28:53 +00:00
Even if you are using Flatcar Container Linux by Kinvolk for your cluster, you will still
2018-02-23 20:08:45 +00:00
need the GlusterFS VMs to be based on either Debian or RedHat based images.
2020-08-28 09:28:53 +00:00
Flatcar Container Linux by Kinvolk cannot serve GlusterFS, but can connect to it through
2017-12-05 12:48:47 +00:00
binaries available on hyperkube v1.4.3_coreos.0 or higher.
2016-06-05 20:52:20 +00:00
## Requirements
2019-08-18 08:30:05 +00:00
- [Install Terraform ](https://www.terraform.io/intro/getting-started/install.html ) 0.12 or later
2017-12-05 12:48:47 +00:00
- [Install Ansible ](http://docs.ansible.com/ansible/latest/intro_installation.html )
2018-02-23 20:08:45 +00:00
- you already have a suitable OS image in Glance
- you already have a floating IP pool created
- you have security groups enabled
2017-12-05 12:48:47 +00:00
- you have a pair of keys generated that can be used to secure the new hosts
2016-06-05 20:52:20 +00:00
2017-12-05 12:48:47 +00:00
## Module Architecture
2020-12-25 20:10:27 +00:00
2017-12-05 12:48:47 +00:00
The configuration is divided into three modules:
2020-12-25 20:10:27 +00:00
2017-12-05 12:48:47 +00:00
- Network
- IPs
- Compute
2016-06-05 20:52:20 +00:00
2017-12-05 12:48:47 +00:00
The main reason for splitting the configuration up in this way is to easily
accommodate situations where floating IPs are limited by a quota or if you have
any external references to the floating IP (e.g. DNS) that would otherwise have
to be updated.
2016-06-05 20:52:20 +00:00
2017-12-05 12:48:47 +00:00
You can force your existing IPs by modifying the compute variables in
2018-02-23 20:08:45 +00:00
`kubespray.tf` as follows:
2016-06-05 20:52:20 +00:00
2020-12-25 20:10:27 +00:00
```ini
2017-12-05 12:48:47 +00:00
k8s_master_fips = ["151.101.129.67"]
k8s_node_fips = ["151.101.129.68"]
2016-06-05 20:52:20 +00:00
```
2017-12-05 12:48:47 +00:00
## Terraform
2020-12-25 20:10:27 +00:00
2018-02-23 20:08:45 +00:00
Terraform will be used to provision all of the OpenStack resources with base software as appropriate.
2016-06-05 20:52:20 +00:00
2018-02-23 20:08:45 +00:00
### Configuration
2016-06-05 20:52:20 +00:00
2018-02-23 20:08:45 +00:00
#### Inventory files
2016-06-05 20:52:20 +00:00
2018-02-23 20:08:45 +00:00
Create an inventory directory for your cluster by copying the existing sample and linking the `hosts` script (used to build the inventory based on Terraform state):
```ShellSession
2020-12-25 20:10:27 +00:00
cp -LRp contrib/terraform/openstack/sample-inventory inventory/$CLUSTER
cd inventory/$CLUSTER
ln -s ../../contrib/terraform/openstack/hosts
ln -s ../../contrib
2018-02-23 20:08:45 +00:00
```
This will be the base for subsequent Terraform commands.
#### OpenStack access and credentials
No provider variables are hardcoded inside `variables.tf` because Terraform
2019-02-11 22:10:10 +00:00
supports various authentication methods for OpenStack: the older script and
environment method (using `openrc` ) as well as a newer declarative method, and
2018-02-23 23:05:29 +00:00
different OpenStack environments may support Identity API version 2 or 3.
2017-12-06 14:58:40 +00:00
These are examples and may vary depending on your OpenStack cloud provider,
for an exhaustive list on how to authenticate on OpenStack with Terraform
please read the [OpenStack provider documentation ](https://www.terraform.io/docs/providers/openstack/ ).
2018-02-23 23:05:29 +00:00
##### Declarative method (recommended)
2017-12-06 14:58:40 +00:00
2018-02-23 23:05:29 +00:00
The recommended authentication method is to describe credentials in a YAML file `clouds.yaml` that can be stored in:
2017-12-06 14:58:40 +00:00
2020-12-25 20:10:27 +00:00
- the current directory
- `~/.config/openstack`
- `/etc/openstack`
2017-12-06 14:58:40 +00:00
2018-02-23 20:08:45 +00:00
`clouds.yaml` :
2017-12-06 14:58:40 +00:00
2020-12-25 20:10:27 +00:00
```yaml
2017-12-06 14:58:40 +00:00
clouds:
mycloud:
auth:
auth_url: https://openstack:5000/v3
username: "username"
project_name: "projectname"
project_id: projectid
user_domain_name: "Default"
password: "password"
region_name: "RegionOne"
interface: "public"
identity_api_version: 3
```
If you have multiple clouds defined in your `clouds.yaml` file you can choose
2018-02-23 23:05:29 +00:00
the one you want to use with the environment variable `OS_CLOUD` :
2017-12-06 14:58:40 +00:00
2020-12-25 20:10:27 +00:00
```ShellSession
2017-12-06 14:58:40 +00:00
export OS_CLOUD=mycloud
```
2018-04-30 15:11:07 +00:00
##### Openrc method
2017-12-06 14:58:40 +00:00
When using classic environment variables, Terraform uses default `OS_*`
2018-02-24 00:54:13 +00:00
environment variables. A script suitable for your environment may be available
from Horizon under *Project* -> *Compute* -> *Access & Security* -> *API Access* .
2017-12-06 14:58:40 +00:00
2018-02-23 23:05:29 +00:00
With identity v2:
2016-06-05 20:52:20 +00:00
2020-12-25 20:10:27 +00:00
```ShellSession
2017-12-06 14:58:40 +00:00
source openrc
env | grep OS
OS_AUTH_URL=https://openstack:5000/v2.0
OS_PROJECT_ID=projectid
OS_PROJECT_NAME=projectname
OS_USERNAME=username
OS_PASSWORD=password
OS_REGION_NAME=RegionOne
OS_INTERFACE=public
OS_IDENTITY_API_VERSION=2
2016-06-05 20:52:20 +00:00
```
2018-02-24 00:54:13 +00:00
With identity v3:
2016-06-05 20:52:20 +00:00
2020-12-25 20:10:27 +00:00
```ShellSession
2017-12-06 14:58:40 +00:00
source openrc
env | grep OS
OS_AUTH_URL=https://openstack:5000/v3
OS_PROJECT_ID=projectid
OS_PROJECT_NAME=username
OS_PROJECT_DOMAIN_ID=default
OS_USERNAME=username
OS_PASSWORD=password
OS_REGION_NAME=RegionOne
OS_INTERFACE=public
OS_IDENTITY_API_VERSION=3
OS_USER_DOMAIN_NAME=Default
```
2020-12-25 20:10:27 +00:00
Terraform does not support a mix of DomainName and DomainID, choose one or the other:
2017-12-06 14:58:40 +00:00
2020-12-25 20:10:27 +00:00
- provider.openstack: You must provide exactly one of DomainID or DomainName to authenticate by Username
2017-12-06 14:58:40 +00:00
2020-12-25 20:10:27 +00:00
```ShellSession
2017-12-06 14:58:40 +00:00
unset OS_USER_DOMAIN_NAME
export OS_USER_DOMAIN_ID=default
2020-12-25 20:10:27 +00:00
```
2017-12-06 14:58:40 +00:00
or
2020-12-25 20:10:27 +00:00
```ShellSession
2017-12-06 14:58:40 +00:00
unset OS_PROJECT_DOMAIN_ID
set OS_PROJECT_DOMAIN_NAME=Default
2016-06-05 20:52:20 +00:00
```
2018-02-23 20:08:45 +00:00
#### Cluster variables
2020-12-25 20:10:27 +00:00
2017-12-05 12:48:47 +00:00
The construction of the cluster is driven by values found in
[variables.tf ](variables.tf ).
2019-08-18 08:30:05 +00:00
For your cluster, edit `inventory/$CLUSTER/cluster.tfvars` .
2017-09-27 09:59:09 +00:00
2017-12-05 12:48:47 +00:00
|Variable | Description |
|---------|-------------|
|`cluster_name` | All OpenStack resources will use the Terraform variable`cluster_name` (default`example`) in their name to make it easier to track. For example the first compute resource will be named`example-kubernetes-1`. |
2019-09-09 14:41:19 +00:00
|`az_list` | List of Availability Zones available in your OpenStack cluster. |
2017-12-05 12:48:47 +00:00
|`network_name` | The name to be given to the internal network that will be generated |
2019-08-21 12:09:15 +00:00
|`network_dns_domain` | (Optional) The dns_domain for the internal network that will be generated |
2017-12-06 14:58:40 +00:00
|`dns_nameservers`| An array of DNS name server names to be used by hosts in the internal subnet. |
2017-12-05 12:48:47 +00:00
|`floatingip_pool` | Name of the pool from which floating IPs will be allocated |
2020-10-12 06:54:47 +00:00
|`k8s_master_fips` | A list of floating IPs that you have already pre-allocated; they will be attached to master nodes instead of creating new random floating IPs. |
2021-11-19 15:58:52 +00:00
|`bastion_fips` | A list of floating IPs that you have already pre-allocated; they will be attached to bastion node instead of creating new random floating IPs. |
2017-12-05 12:48:47 +00:00
|`external_net` | UUID of the external network that will be routed to |
2019-02-26 04:13:16 +00:00
|`flavor_k8s_master`,`flavor_k8s_node`,`flavor_etcd`, `flavor_bastion` ,`flavor_gfs_node` | Flavor depends on your openstack installation, you can get available flavor IDs through `openstack flavor list` |
2017-12-05 12:48:47 +00:00
|`image`,`image_gfs` | Name of the image to use in provisioning the compute resources. Should already be loaded into glance. |
|`ssh_user`,`ssh_user_gfs` | The username to ssh into the image with. This usually depends on the image you have selected |
|`public_key_path` | Path on your local workstation to the public key file you wish to use in creating the key pairs |
|`number_of_k8s_masters`, `number_of_k8s_masters_no_floating_ip` | Number of nodes that serve as both master and etcd. These can be provisioned with or without floating IP addresses|
|`number_of_k8s_masters_no_etcd`, `number_of_k8s_masters_no_floating_ip_no_etcd` | Number of nodes that serve as just master with no etcd. These can be provisioned with or without floating IP addresses |
|`number_of_etcd` | Number of pure etcd nodes |
|`number_of_k8s_nodes`, `number_of_k8s_nodes_no_floating_ip` | Kubernetes worker nodes. These can be provisioned with or without floating ip addresses. |
|`number_of_bastions` | Number of bastion hosts to create. Scripts assume this is really just zero or one |
|`number_of_gfs_nodes_no_floating_ip` | Number of gluster servers to provision. |
| `gfs_volume_size_in_gb` | Size of the non-ephemeral volumes to be attached to store the GlusterFS bricks |
2021-04-29 12:20:50 +00:00
|`supplementary_master_groups` | To add ansible groups to the masters, such as `kube_node` for tainting them as nodes, empty by default. |
|`supplementary_node_groups` | To add ansible groups to the nodes, such as `kube_ingress` for running ingress controller pods, empty by default. |
2018-09-28 09:35:02 +00:00
|`bastion_allowed_remote_ips` | List of CIDR allowed to initiate a SSH connection, `["0.0.0.0/0"]` by default |
2019-04-21 08:57:44 +00:00
|`master_allowed_remote_ips` | List of CIDR blocks allowed to initiate an API connection, `["0.0.0.0/0"]` by default |
2019-04-15 14:22:08 +00:00
|`k8s_allowed_remote_ips` | List of CIDR allowed to initiate a SSH connection, empty by default |
2018-10-31 11:09:22 +00:00
|`worker_allowed_ports` | List of ports to open on worker nodes, `[{ "protocol" = "tcp", "port_range_min" = 30000, "port_range_max" = 32767, "remote_ip_prefix" = "0.0.0.0/0"}]` by default |
2021-01-21 15:55:06 +00:00
|`master_allowed_ports` | List of ports to open on master nodes, expected format is `[{ "protocol" = "tcp", "port_range_min" = 443, "port_range_max" = 443, "remote_ip_prefix" = "0.0.0.0/0"}]` , empty by default |
2019-05-09 09:16:50 +00:00
|`wait_for_floatingip` | Let Terraform poll the instance until the floating IP has been associated, `false` by default. |
2019-09-26 11:31:07 +00:00
|`node_root_volume_size_in_gb` | Size of the root volume for nodes, 0 to use ephemeral storage |
|`master_root_volume_size_in_gb` | Size of the root volume for masters, 0 to use ephemeral storage |
2021-07-14 05:52:26 +00:00
|`master_volume_type` | Volume type of the root volume for control_plane, 'Default' by default |
2019-09-26 11:31:07 +00:00
|`gfs_root_volume_size_in_gb` | Size of the root volume for gluster, 0 to use ephemeral storage |
|`etcd_root_volume_size_in_gb` | Size of the root volume for etcd nodes, 0 to use ephemeral storage |
|`bastion_root_volume_size_in_gb` | Size of the root volume for bastions, 0 to use ephemeral storage |
2019-12-09 09:15:10 +00:00
|`use_server_group` | Create and use openstack nova servergroups, default: false |
2020-09-17 09:48:45 +00:00
|`use_access_ip` | If 1, nodes with floating IPs will transmit internal cluster traffic via floating IPs; if 0 private IPs will be used instead. Default value is 1. |
2020-04-16 14:52:45 +00:00
|`k8s_nodes` | Map containing worker node definition, see explanation below |
##### k8s_nodes
2020-12-25 20:10:27 +00:00
2021-04-05 08:20:19 +00:00
Allows a custom definition of worker nodes giving the operator full control over individual node flavor and
2020-04-16 14:52:45 +00:00
availability zone placement. To enable the use of this mode set the `number_of_k8s_nodes` and
`number_of_k8s_nodes_no_floating_ip` variables to 0. Then define your desired worker node configuration
using the `k8s_nodes` variable.
For example:
2020-12-25 20:10:27 +00:00
```ini
2020-04-16 14:52:45 +00:00
k8s_nodes = {
"1" = {
"az" = "sto1"
"flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
"floating_ip" = true
},
"2" = {
"az" = "sto2"
"flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
"floating_ip" = true
},
"3" = {
"az" = "sto3"
"flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
"floating_ip" = true
}
}
```
Would result in the same configuration as:
2020-12-25 20:10:27 +00:00
```ini
2020-04-16 14:52:45 +00:00
number_of_k8s_nodes = 3
flavor_k8s_node = "83d8b44a-26a0-4f02-a981-079446926445"
az_list = ["sto1", "sto2", "sto3"]
```
And:
2020-12-25 20:10:27 +00:00
```ini
2020-04-16 14:52:45 +00:00
k8s_nodes = {
"ing-1" = {
"az" = "sto1"
"flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
"floating_ip" = true
},
"ing-2" = {
"az" = "sto2"
"flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
"floating_ip" = true
},
"ing-3" = {
"az" = "sto3"
"flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
"floating_ip" = true
},
"big-1" = {
"az" = "sto1"
"flavor" = "3f73fc93-ec61-4808-88df-2580d94c1a9b"
"floating_ip" = false
},
"big-2" = {
"az" = "sto2"
"flavor" = "3f73fc93-ec61-4808-88df-2580d94c1a9b"
"floating_ip" = false
},
"big-3" = {
"az" = "sto3"
"flavor" = "3f73fc93-ec61-4808-88df-2580d94c1a9b"
"floating_ip" = false
},
"small-1" = {
"az" = "sto1"
"flavor" = "7a6a998f-ac7f-4fb8-a534-2175b254f75e"
"floating_ip" = false
},
"small-2" = {
"az" = "sto2"
"flavor" = "7a6a998f-ac7f-4fb8-a534-2175b254f75e"
"floating_ip" = false
},
"small-3" = {
"az" = "sto3"
"flavor" = "7a6a998f-ac7f-4fb8-a534-2175b254f75e"
"floating_ip" = false
}
}
```
Would result in three nodes in each availability zone each with their own separate naming,
flavor and floating ip configuration.
The "schema":
2020-12-25 20:10:27 +00:00
```ini
2020-04-16 14:52:45 +00:00
k8s_nodes = {
"key | node name suffix, must be unique" = {
"az" = string
"flavor" = string
"floating_ip" = bool
},
}
```
2020-12-25 20:10:27 +00:00
2020-04-16 14:52:45 +00:00
All values are required.
2016-10-26 14:56:15 +00:00
2018-02-23 20:08:45 +00:00
#### Terraform state files
2017-12-06 14:58:40 +00:00
2018-02-23 20:08:45 +00:00
In the cluster's inventory folder, the following files might be created (either by Terraform
2017-12-06 14:58:40 +00:00
or manually), to prevent you from pushing them accidentally they are in a
`.gitignore` file in the `terraform/openstack` directory :
2020-12-25 20:10:27 +00:00
- `.terraform`
- `.tfvars`
- `.tfstate`
- `.tfstate.backup`
2017-12-06 14:58:40 +00:00
You can still add them manually if you want to.
2018-02-23 20:08:45 +00:00
### Initialization
2017-12-06 14:58:40 +00:00
2018-02-23 20:08:45 +00:00
Before Terraform can operate on your cluster you need to install the required
plugins. This is accomplished as follows:
2017-12-05 12:48:47 +00:00
2018-02-23 20:08:45 +00:00
```ShellSession
2020-12-25 20:10:27 +00:00
cd inventory/$CLUSTER
terraform init ../../contrib/terraform/openstack
2016-10-26 14:56:15 +00:00
```
2017-12-05 12:48:47 +00:00
2018-02-23 20:08:45 +00:00
This should finish fairly quickly telling you Terraform has successfully initialized and loaded necessary modules.
### Provisioning cluster
2020-12-25 20:10:27 +00:00
2018-02-23 20:08:45 +00:00
You can apply the Terraform configuration to your cluster with the following command
issued from your cluster's inventory directory (`inventory/$CLUSTER`):
2020-12-25 20:10:27 +00:00
2018-02-23 20:08:45 +00:00
```ShellSession
2020-12-25 20:10:27 +00:00
terraform apply -var-file=cluster.tfvars ../../contrib/terraform/openstack
2016-10-26 14:56:15 +00:00
```
2017-12-05 12:48:47 +00:00
if you chose to create a bastion host, this script will create
2021-04-29 12:20:50 +00:00
`contrib/terraform/openstack/k8s_cluster.yml` with an ssh command for Ansible to
2018-09-30 03:31:57 +00:00
be able to access your machines tunneling through the bastion's IP address. If
2017-12-05 12:48:47 +00:00
you want to manually handle the ssh tunneling to these machines, please delete
or move that file. If you want to use this, just leave it there, as ansible will
pick it up automatically.
2017-09-27 09:59:09 +00:00
2018-02-23 20:08:45 +00:00
### Destroying cluster
2020-12-25 20:10:27 +00:00
2018-02-23 20:08:45 +00:00
You can destroy your new cluster with the following command issued from the cluster's inventory directory:
2017-09-27 09:59:09 +00:00
2018-02-23 20:08:45 +00:00
```ShellSession
2020-12-25 20:10:27 +00:00
terraform destroy -var-file=cluster.tfvars ../../contrib/terraform/openstack
2017-09-27 09:59:09 +00:00
```
2018-02-24 00:54:13 +00:00
If you've started the Ansible run, it may also be a good idea to do some manual cleanup:
2020-12-25 20:10:27 +00:00
- remove SSH keys from the destroyed cluster from your `~/.ssh/known_hosts` file
- clean up any temporary cache files: `rm /tmp/$CLUSTER-*`
2018-02-24 00:54:13 +00:00
2018-02-23 20:08:45 +00:00
### Debugging
2020-12-25 20:10:27 +00:00
2017-12-05 12:48:47 +00:00
You can enable debugging output from Terraform by setting
2018-02-23 20:08:45 +00:00
`OS_DEBUG` to 1 and`TF_LOG` to`DEBUG` before running the Terraform command.
2017-09-27 09:59:09 +00:00
2018-02-23 20:08:45 +00:00
### Terraform output
2017-12-06 14:58:40 +00:00
2018-02-23 20:08:45 +00:00
Terraform can output values that are useful for configure Neutron/Octavia LBaaS or Cinder persistent volume provisioning as part of your Kubernetes deployment:
2017-12-06 14:58:40 +00:00
2020-12-25 20:10:27 +00:00
- `private_subnet_id` : the subnet where your instances are running is used for `openstack_lbaas_subnet_id`
- `floating_network_id` : the network_id where the floating IP are provisioned is used for `openstack_lbaas_floating_network_id`
2018-02-23 20:08:45 +00:00
## Ansible
### Node access
#### SSH
2017-09-27 09:59:09 +00:00
2017-12-05 12:48:47 +00:00
Ensure your local ssh-agent is running and your ssh key has been added. This
step is required by the terraform provisioner:
2017-09-27 09:59:09 +00:00
2020-12-25 20:10:27 +00:00
```ShellSession
eval $(ssh-agent -s)
ssh-add ~/.ssh/id_rsa
2017-12-05 12:48:47 +00:00
```
2018-02-23 20:08:45 +00:00
If you have deployed and destroyed a previous iteration of your cluster, you will need to clear out any stale keys from your SSH "known hosts" file ( `~/.ssh/known_hosts` ).
2019-06-10 06:01:05 +00:00
#### Metadata variables
The [python script ](../terraform.py ) that reads the
generated`.tfstate` file to generate a dynamic inventory recognizes
some variables within a "metadata" block, defined in a "resource"
block (example):
2020-12-25 20:10:27 +00:00
```ini
2019-06-10 06:01:05 +00:00
resource "openstack_compute_instance_v2" "example" {
...
metadata {
ssh_user = "ubuntu"
prefer_ipv6 = true
2020-12-25 20:10:27 +00:00
python_bin = "/usr/bin/python3"
2019-06-10 06:01:05 +00:00
}
...
}
```
As the example shows, these let you define the SSH username for
Ansible, a Python binary which is needed by Ansible if
`/usr/bin/python` doesn't exist, and whether the IPv6 address of the
instance should be preferred over IPv4.
2018-02-23 20:08:45 +00:00
#### Bastion host
2018-04-30 15:11:07 +00:00
Bastion access will be determined by:
2018-02-23 20:08:45 +00:00
2020-12-25 20:10:27 +00:00
- Your choice on the amount of bastion hosts (set by `number_of_bastions` terraform variable).
- The existence of nodes/masters with floating IPs (set by `number_of_k8s_masters` , `number_of_k8s_nodes` , `number_of_k8s_masters_no_etcd` terraform variables).
2018-04-30 15:11:07 +00:00
If you have a bastion host, your ssh traffic will be directly routed through it. This is regardless of whether you have masters/nodes with a floating IP assigned.
If you don't have a bastion host, but at least one of your masters/nodes have a floating IP, then ssh traffic will be tunneled by one of these machines.
So, either a bastion host, or at least master/node with a floating IP are required.
2018-02-23 20:08:45 +00:00
#### Test access
2017-12-05 12:48:47 +00:00
2020-08-28 09:28:53 +00:00
Make sure you can connect to the hosts. Note that Flatcar Container Linux by Kinvolk will have a state `FAILED` due to Python not being present. This is okay, because Python will be installed during bootstrapping, so long as the hosts are not `UNREACHABLE` .
2016-11-11 03:59:46 +00:00
2020-12-25 20:10:27 +00:00
```ShellSession
2018-02-23 20:08:45 +00:00
$ ansible -i inventory/$CLUSTER/hosts -m ping all
2017-12-05 12:48:47 +00:00
example-k8s_node-1 | SUCCESS => {
"changed": false,
"ping": "pong"
}
example-etcd-1 | SUCCESS => {
"changed": false,
"ping": "pong"
}
example-k8s-master-1 | SUCCESS => {
"changed": false,
"ping": "pong"
}
2016-11-11 03:59:46 +00:00
```
2018-02-23 20:08:45 +00:00
If it fails try to connect manually via SSH. It could be something as simple as a stale host key.
2017-10-18 12:26:14 +00:00
2018-02-23 20:08:45 +00:00
### Configure cluster variables
2017-07-10 15:53:57 +00:00
2019-02-11 22:55:41 +00:00
Edit `inventory/$CLUSTER/group_vars/all/all.yml` :
2020-12-25 20:10:27 +00:00
2018-02-23 20:08:45 +00:00
- **bin_dir**:
2020-12-25 20:10:27 +00:00
```yml
2017-07-10 15:53:57 +00:00
# Directory where the binaries will be installed
# Default:
# bin_dir: /usr/local/bin
2020-08-28 09:28:53 +00:00
# For Flatcar Container Linux by Kinvolk:
2017-07-10 15:53:57 +00:00
bin_dir: /opt/bin
```
2020-12-25 20:10:27 +00:00
2018-02-23 20:08:45 +00:00
- and **cloud_provider** :
2020-12-25 20:10:27 +00:00
```yml
2017-07-11 09:46:19 +00:00
cloud_provider: openstack
```
2020-12-25 20:10:27 +00:00
2021-04-29 12:20:50 +00:00
Edit `inventory/$CLUSTER/group_vars/k8s_cluster/k8s_cluster.yml` :
2020-12-25 20:10:27 +00:00
2018-02-23 20:08:45 +00:00
- Set variable **kube_network_plugin** to your desired networking plugin.
- **flannel** works out-of-the-box
- **calico** requires [configuring OpenStack Neutron ports ](/docs/openstack.md ) to allow service and pod subnets
2020-12-25 20:10:27 +00:00
```yml
2017-07-10 15:53:57 +00:00
# Choose network plugin (calico, weave or flannel)
# Can also be set to 'cloud', which lets the cloud provider setup appropriate routing
kube_network_plugin: flannel
```
2020-12-25 20:10:27 +00:00
2017-07-11 09:46:19 +00:00
- Set variable **resolvconf_mode**
2020-12-25 20:10:27 +00:00
```yml
2017-07-11 09:46:19 +00:00
# Can be docker_dns, host_resolvconf or none
# Default:
# resolvconf_mode: docker_dns
2020-08-28 09:28:53 +00:00
# For Flatcar Container Linux by Kinvolk:
2017-07-11 09:46:19 +00:00
resolvconf_mode: host_resolvconf
```
2020-12-25 20:10:27 +00:00
2019-08-20 06:45:32 +00:00
- Set max amount of attached cinder volume per host (default 256)
2020-12-25 20:10:27 +00:00
```yml
2019-08-20 06:45:32 +00:00
node_volume_attach_limit: 26
```
2017-07-10 15:53:57 +00:00
2018-02-23 20:08:45 +00:00
### Deploy Kubernetes
2016-06-05 20:52:20 +00:00
2020-12-25 20:10:27 +00:00
```ShellSession
ansible-playbook --become -i inventory/$CLUSTER/hosts cluster.yml
2016-06-05 20:52:20 +00:00
```
2018-02-23 20:08:45 +00:00
This will take some time as there are many tasks to run.
## Kubernetes
### Set up kubectl
2020-12-25 20:10:27 +00:00
2018-02-23 20:08:45 +00:00
1. [Install kubectl ](https://kubernetes.io/docs/tasks/tools/install-kubectl/ ) on your workstation
2. Add a route to the internal IP of a master node (if needed):
2020-12-25 20:10:27 +00:00
```ShellSession
2017-07-10 15:53:57 +00:00
sudo route add [master-internal-ip] gw [router-ip]
```
2020-12-25 20:10:27 +00:00
2017-07-10 15:53:57 +00:00
or
2020-12-25 20:10:27 +00:00
```ShellSession
2017-07-10 15:53:57 +00:00
sudo route add -net [internal-subnet]/24 gw [router-ip]
```
2020-12-25 20:10:27 +00:00
1. List Kubernetes certificates & keys:
```ShellSession
2019-03-20 12:56:57 +00:00
ssh [os-user]@[master-ip] sudo ls /etc/kubernetes/ssl/
2017-07-10 15:53:57 +00:00
```
2020-12-25 20:10:27 +00:00
1. Get `admin` 's certificates and keys:
```ShellSession
2019-03-20 12:56:57 +00:00
ssh [os-user]@[master-ip] sudo cat /etc/kubernetes/ssl/admin-kube-master-1-key.pem > admin-key.pem
ssh [os-user]@[master-ip] sudo cat /etc/kubernetes/ssl/admin-kube-master-1.pem > admin.pem
ssh [os-user]@[master-ip] sudo cat /etc/kubernetes/ssl/ca.pem > ca.pem
2017-07-10 15:53:57 +00:00
```
2020-12-25 20:10:27 +00:00
1. Configure kubectl:
2018-02-23 20:08:45 +00:00
```ShellSession
$ kubectl config set-cluster default-cluster --server=https://[master-internal-ip]:6443 \
2017-12-05 12:48:47 +00:00
--certificate-authority=ca.pem
2017-07-10 15:53:57 +00:00
2018-02-23 20:08:45 +00:00
$ kubectl config set-credentials default-admin \
2017-07-10 15:53:57 +00:00
--certificate-authority=ca.pem \
--client-key=admin-key.pem \
2017-12-05 12:48:47 +00:00
--client-certificate=admin.pem
2017-07-10 15:53:57 +00:00
2018-02-23 20:08:45 +00:00
$ kubectl config set-context default-system --cluster=default-cluster --user=default-admin
$ kubectl config use-context default-system
2017-07-10 15:53:57 +00:00
```
2020-12-25 20:10:27 +00:00
1. Check it:
```ShellSession
2017-07-10 15:53:57 +00:00
kubectl version
```
2017-12-05 12:48:47 +00:00
## GlusterFS
2020-12-25 20:10:27 +00:00
GlusterFS is not deployed by the standard `cluster.yml` playbook, see the
2018-02-23 20:08:45 +00:00
[GlusterFS playbook documentation ](../../network-storage/glusterfs/README.md )
2017-12-05 12:48:47 +00:00
for instructions.
2016-06-05 20:52:20 +00:00
2018-02-23 20:08:45 +00:00
Basically you will install Gluster as
2020-12-25 20:10:27 +00:00
2018-02-23 20:08:45 +00:00
```ShellSession
2020-12-25 20:10:27 +00:00
ansible-playbook --become -i inventory/$CLUSTER/hosts ./contrib/network-storage/glusterfs/glusterfs.yml
2016-06-05 20:52:20 +00:00
```
2017-12-05 12:48:47 +00:00
2018-02-23 20:08:45 +00:00
## What's next
Try out your new Kubernetes cluster with the [Hello Kubernetes service ](https://kubernetes.io/docs/tasks/access-application-cluster/service-access-application-cluster/ ).
2020-04-16 14:52:45 +00:00
## Appendix
### Migration from `number_of_k8s_nodes*` to `k8s_nodes`
2020-12-25 20:10:27 +00:00
2020-04-16 14:52:45 +00:00
If you currently have a cluster defined using the `number_of_k8s_nodes*` variables and wish
to migrate to the `k8s_nodes` style you can do it like so:
```ShellSession
$ terraform state list
module.compute.data.openstack_images_image_v2.gfs_image
module.compute.data.openstack_images_image_v2.vm_image
module.compute.openstack_compute_floatingip_associate_v2.k8s_master[0]
module.compute.openstack_compute_floatingip_associate_v2.k8s_node[0]
module.compute.openstack_compute_floatingip_associate_v2.k8s_node[1]
module.compute.openstack_compute_floatingip_associate_v2.k8s_node[2]
module.compute.openstack_compute_instance_v2.k8s_master[0]
module.compute.openstack_compute_instance_v2.k8s_node[0]
module.compute.openstack_compute_instance_v2.k8s_node[1]
module.compute.openstack_compute_instance_v2.k8s_node[2]
module.compute.openstack_compute_keypair_v2.k8s
module.compute.openstack_compute_servergroup_v2.k8s_etcd[0]
module.compute.openstack_compute_servergroup_v2.k8s_master[0]
module.compute.openstack_compute_servergroup_v2.k8s_node[0]
module.compute.openstack_networking_secgroup_rule_v2.bastion[0]
module.compute.openstack_networking_secgroup_rule_v2.egress[0]
module.compute.openstack_networking_secgroup_rule_v2.k8s
module.compute.openstack_networking_secgroup_rule_v2.k8s_allowed_remote_ips[0]
module.compute.openstack_networking_secgroup_rule_v2.k8s_allowed_remote_ips[1]
module.compute.openstack_networking_secgroup_rule_v2.k8s_allowed_remote_ips[2]
module.compute.openstack_networking_secgroup_rule_v2.k8s_master[0]
module.compute.openstack_networking_secgroup_rule_v2.worker[0]
module.compute.openstack_networking_secgroup_rule_v2.worker[1]
module.compute.openstack_networking_secgroup_rule_v2.worker[2]
module.compute.openstack_networking_secgroup_rule_v2.worker[3]
module.compute.openstack_networking_secgroup_rule_v2.worker[4]
module.compute.openstack_networking_secgroup_v2.bastion[0]
module.compute.openstack_networking_secgroup_v2.k8s
module.compute.openstack_networking_secgroup_v2.k8s_master
module.compute.openstack_networking_secgroup_v2.worker
module.ips.null_resource.dummy_dependency
module.ips.openstack_networking_floatingip_v2.k8s_master[0]
module.ips.openstack_networking_floatingip_v2.k8s_node[0]
module.ips.openstack_networking_floatingip_v2.k8s_node[1]
module.ips.openstack_networking_floatingip_v2.k8s_node[2]
module.network.openstack_networking_network_v2.k8s[0]
module.network.openstack_networking_router_interface_v2.k8s[0]
module.network.openstack_networking_router_v2.k8s[0]
module.network.openstack_networking_subnet_v2.k8s[0]
$ terraform state mv 'module.compute.openstack_compute_floatingip_associate_v2.k8s_node[0]' 'module.compute.openstack_compute_floatingip_associate_v2.k8s_nodes["1"]'
Move "module.compute.openstack_compute_floatingip_associate_v2.k8s_node[0]" to "module.compute.openstack_compute_floatingip_associate_v2.k8s_nodes[\"1\"]"
Successfully moved 1 object(s).
$ terraform state mv 'module.compute.openstack_compute_floatingip_associate_v2.k8s_node[1]' 'module.compute.openstack_compute_floatingip_associate_v2.k8s_nodes["2"]'
Move "module.compute.openstack_compute_floatingip_associate_v2.k8s_node[1]" to "module.compute.openstack_compute_floatingip_associate_v2.k8s_nodes[\"2\"]"
Successfully moved 1 object(s).
$ terraform state mv 'module.compute.openstack_compute_floatingip_associate_v2.k8s_node[2]' 'module.compute.openstack_compute_floatingip_associate_v2.k8s_nodes["3"]'
Move "module.compute.openstack_compute_floatingip_associate_v2.k8s_node[2]" to "module.compute.openstack_compute_floatingip_associate_v2.k8s_nodes[\"3\"]"
Successfully moved 1 object(s).
$ terraform state mv 'module.compute.openstack_compute_instance_v2.k8s_node[0]' 'module.compute.openstack_compute_instance_v2.k8s_node["1"]'
Move "module.compute.openstack_compute_instance_v2.k8s_node[0]" to "module.compute.openstack_compute_instance_v2.k8s_node[\"1\"]"
Successfully moved 1 object(s).
$ terraform state mv 'module.compute.openstack_compute_instance_v2.k8s_node[1]' 'module.compute.openstack_compute_instance_v2.k8s_node["2"]'
Move "module.compute.openstack_compute_instance_v2.k8s_node[1]" to "module.compute.openstack_compute_instance_v2.k8s_node[\"2\"]"
Successfully moved 1 object(s).
$ terraform state mv 'module.compute.openstack_compute_instance_v2.k8s_node[2]' 'module.compute.openstack_compute_instance_v2.k8s_node["3"]'
Move "module.compute.openstack_compute_instance_v2.k8s_node[2]" to "module.compute.openstack_compute_instance_v2.k8s_node[\"3\"]"
Successfully moved 1 object(s).
$ terraform state mv 'module.ips.openstack_networking_floatingip_v2.k8s_node[0]' 'module.ips.openstack_networking_floatingip_v2.k8s_node["1"]'
Move "module.ips.openstack_networking_floatingip_v2.k8s_node[0]" to "module.ips.openstack_networking_floatingip_v2.k8s_node[\"1\"]"
Successfully moved 1 object(s).
$ terraform state mv 'module.ips.openstack_networking_floatingip_v2.k8s_node[1]' 'module.ips.openstack_networking_floatingip_v2.k8s_node["2"]'
Move "module.ips.openstack_networking_floatingip_v2.k8s_node[1]" to "module.ips.openstack_networking_floatingip_v2.k8s_node[\"2\"]"
Successfully moved 1 object(s).
$ terraform state mv 'module.ips.openstack_networking_floatingip_v2.k8s_node[2]' 'module.ips.openstack_networking_floatingip_v2.k8s_node["3"]'
Move "module.ips.openstack_networking_floatingip_v2.k8s_node[2]" to "module.ips.openstack_networking_floatingip_v2.k8s_node[\"3\"]"
Successfully moved 1 object(s).
```
Of course for nodes without floating ips those steps can be omitted.