c12s-kubespray/contrib/terraform/aws
Cristian Calin 360aff4a57
Rename ansible groups to use _ instead of - (#7552)
* rename ansible groups to use _ instead of -

k8s-cluster -> k8s_cluster
k8s-node -> k8s_node
calico-rr -> calico_rr
no-floating -> no_floating

Note: kube-node,k8s-cluster groups in upgrade CI
      need clean-up after v2.16 is tagged

* ensure old groups are mapped to the new ones
2021-04-29 05:20:50 -07:00
..
docs rename almost all mentions of kargo 2017-06-16 13:25:46 -04:00
modules Replace kube-master with kube_control_plane (#7256) 2021-03-23 17:26:05 -07:00
sample-inventory Update to TF v0.12.12 (#5267) 2019-12-04 07:20:58 -08:00
templates Rename ansible groups to use _ instead of - (#7552) 2021-04-29 05:20:50 -07:00
.gitignore Rewrote AWS Terraform for Kargo 2017-03-06 12:52:02 +01:00
create-infrastructure.tf Replace kube-master with kube_control_plane (#7256) 2021-03-23 17:26:05 -07:00
credentials.tfvars.example Rewrote AWS Terraform for Kargo 2017-03-06 12:52:02 +01:00
output.tf Terraform quoted references are now deprecated (#6203) 2020-06-05 00:05:43 -07:00
README.md Replace kube-master with kube_control_plane (#7256) 2021-03-23 17:26:05 -07:00
terraform.tfvars Update to TF v0.12.12 (#5267) 2019-12-04 07:20:58 -08:00
terraform.tfvars.example Fixes for AWS Terraform Deployment and Updated Readme 2017-04-12 15:15:54 +02:00
variables.tf Fix terraform0.13 errors (#7077) 2020-12-23 05:08:26 -08:00

Kubernetes on AWS with Terraform

Overview

This project will create:

  • VPC with Public and Private Subnets in # Availability Zones
  • Bastion Hosts and NAT Gateways in the Public Subnet
  • A dynamic number of masters, etcd, and worker nodes in the Private Subnet
    • even distributed over the # of Availability Zones
  • AWS ELB in the Public Subnet for accessing the Kubernetes API from the internet

Requirements

  • Terraform 0.12.0 or newer

How to Use

  • Export the variables for your AWS credentials or edit credentials.tfvars:
export TF_VAR_AWS_ACCESS_KEY_ID="www"
export TF_VAR_AWS_SECRET_ACCESS_KEY ="xxx"
export TF_VAR_AWS_SSH_KEY_NAME="yyy"
export TF_VAR_AWS_DEFAULT_REGION="zzz"
  • Update contrib/terraform/aws/terraform.tfvars with your data. By default, the Terraform scripts use Ubuntu 18.04 LTS (Bionic) as base image. If you want to change this behaviour, see note "Using other distrib than Ubuntu" below.
  • Create an AWS EC2 SSH Key
  • Run with terraform apply --var-file="credentials.tfvars" or terraform apply depending if you exported your AWS credentials

Example:

terraform apply -var-file=credentials.tfvars
  • Terraform automatically creates an Ansible Inventory file called hosts with the created infrastructure in the directory inventory
  • Ansible will automatically generate an ssh config file for your bastion hosts. To connect to hosts with ssh using bastion host use generated ssh-bastion.conf. Ansible automatically detects bastion and changes ssh_args
ssh -F ./ssh-bastion.conf user@$ip
  • Once the infrastructure is created, you can run the kubespray playbooks and supply inventory/hosts with the -i flag.

Example (this one assumes you are using Ubuntu)

ansible-playbook -i ./inventory/hosts ./cluster.yml -e ansible_user=ubuntu -b --become-user=root --flush-cache

Using other distrib than Ubuntu If you want to use another distribution than Ubuntu 18.04 (Bionic) LTS, you can modify the search filters of the 'data "aws_ami" "distro"' in variables.tf.

For example, to use:

  • Debian Jessie, replace 'data "aws_ami" "distro"' in variables.tf with
data "aws_ami" "distro" {
  most_recent = true

  filter {
    name   = "name"
    values = ["debian-jessie-amd64-hvm-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["379101102735"]
}
  • Ubuntu 16.04, replace 'data "aws_ami" "distro"' in variables.tf with
data "aws_ami" "distro" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"]
}
  • Centos 7, replace 'data "aws_ami" "distro"' in variables.tf with
data "aws_ami" "distro" {
  most_recent = true

  filter {
    name   = "name"
    values = ["dcos-centos7-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["688023202711"]
}

Connecting to Kubernetes

You can use the following set of commands to get the kubeconfig file from your newly created cluster. Before running the commands, make sure you are in the project's root folder.

# Get the controller's IP address.
CONTROLLER_HOST_NAME=$(cat ./inventory/hosts | grep "\[kube_control_plane\]" -A 1 | tail -n 1)
CONTROLLER_IP=$(cat ./inventory/hosts | grep $CONTROLLER_HOST_NAME | grep ansible_host | cut -d'=' -f2)

# Get the hostname of the load balancer.
LB_HOST=$(cat inventory/hosts | grep apiserver_loadbalancer_domain_name | cut -d'"' -f2)

# Get the controller's SSH fingerprint.
ssh-keygen -R $CONTROLLER_IP > /dev/null 2>&1
ssh-keyscan -H $CONTROLLER_IP >> ~/.ssh/known_hosts 2>/dev/null

# Get the kubeconfig from the controller.
mkdir -p ~/.kube
ssh -F ssh-bastion.conf centos@$CONTROLLER_IP "sudo chmod 644 /etc/kubernetes/admin.conf"
scp -F ssh-bastion.conf centos@$CONTROLLER_IP:/etc/kubernetes/admin.conf ~/.kube/config
sed -i "s^server:.*^server: https://$LB_HOST:6443^" ~/.kube/config
kubectl get nodes

Troubleshooting

Remaining AWS IAM Instance Profile

If the cluster was destroyed without using Terraform it is possible that the AWS IAM Instance Profiles still remain. To delete them you can use the AWS CLI with the following command:

aws iam delete-instance-profile --region <region_name> --instance-profile-name <profile_name>

Ansible Inventory doesn't get created

It could happen that Terraform doesn't create an Ansible Inventory file automatically. If this is the case copy the output after inventory= and create a file named hostsin the directory inventory and paste the inventory into the file.

Architecture

Pictured is an AWS Infrastructure created with this Terraform project distributed over two Availability Zones.

AWS Infrastructure with Terraform