Integrate kubernetes/ingress-nginx 0.11.0 to Kubespray

This commit is contained in:
Wong Hoi Sing Edison 2018-02-13 09:08:44 +08:00
parent 9837b7926f
commit fd46442188
19 changed files with 584 additions and 0 deletions

View file

@ -100,6 +100,7 @@
- { role: kubespray-defaults}
- { role: kubernetes-apps/network_plugin, tags: network }
- { role: kubernetes-apps/policy_controller, tags: policy-controller }
- { role: kubernetes-apps/ingress_controller, tags: ingress-controller }
- { role: kubernetes-apps/external_provisioner, tags: external-provisioner }
- hosts: calico-rr

View file

@ -187,6 +187,12 @@ cephfs_provisioner_enabled: false
# cephfs_provisioner_secret: secret
# cephfs_provisioner_storage_class: cephfs
# Nginx ingress controller deployment
ingress_nginx_enabled: false
# ingress_nginx_namespace: "ingress-nginx"
# ingress_nginx_insecure_port: 80
# ingress_nginx_secure_port: 443
# Add Persistent Volumes Storage Class for corresponding cloud provider ( OpenStack is only supported now )
persistent_volumes_enabled: false

View file

@ -0,0 +1,283 @@
Installation Guide
==================
Contents
--------
- [Mandatory commands](#mandatory-commands)
- [Install without RBAC roles](#install-without-rbac-roles)
- [Install with RBAC roles](#install-with-rbac-roles)
- [Custom Provider](#custom-provider)
- [minikube](#minikube)
- [AWS](#aws)
- [GCE - GKE](#gce---gke)
- [Azure](#azure)
- [Baremetal](#baremetal)
- [Using Helm](#using-helm)
- [Verify installation](#verify-installation)
- [Detect installed version](#detect-installed-version)
- [Deploying the config-map](#deploying-the-config-map)
Generic Deployment
------------------
The following resources are required for a generic deployment.
### Mandatory commands
``` console
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/namespace.yaml \
| kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml \
| kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/configmap.yaml \
| kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/tcp-services-configmap.yaml \
| kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/udp-services-configmap.yaml \
| kubectl apply -f -
```
### Install without RBAC roles
``` console
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/without-rbac.yaml \
| kubectl apply -f -
```
### Install with RBAC roles
Please check the [RBAC](rbac.md) document.
``` console
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/rbac.yaml \
| kubectl apply -f -
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/with-rbac.yaml \
| kubectl apply -f -
```
Custom Service Provider Deployment
----------------------------------
There are cloud provider specific yaml files.
### minikube
For standard usage:
``` console
minikube addons enable ingress
```
For development:
1. Disable the ingress addon:
``` console
$ minikube addons disable ingress
```
2. Use the [docker daemon](https://github.com/kubernetes/minikube/blob/master/docs/reusing_the_docker_daemon.md)
3. [Build the image](../docs/development.md)
4. Perform [Mandatory commands](#mandatory-commands)
5. Install the `nginx-ingress-controller` deployment [without RBAC roles](#install-without-rbac-roles) or [with RBAC roles](#install-with-rbac-roles)
6. Edit the `nginx-ingress-controller` deployment to use your custom image. Local images can be seen by performing `docker images`.
``` console
$ kubectl edit deployment nginx-ingress-controller -n ingress-nginx
```
edit the following section:
``` yaml
image: <IMAGE-NAME>:<TAG>
imagePullPolicy: IfNotPresent
name: nginx-ingress-controller
```
7. Confirm the `nginx-ingress-controller` deployment exists:
``` console
$ kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
default-http-backend-66b447d9cf-rrlf9 1/1 Running 0 12s
nginx-ingress-controller-fdcdcd6dd-vvpgs 1/1 Running 0 11s
```
### AWS
In AWS we use an Elastic Load Balancer (ELB) to expose the NGINX Ingress controller behind a Service of `Type=LoadBalancer`.
This setup requires to choose in which layer (L4 or L7) we want to configure the ELB:
- [Layer 4](https://en.wikipedia.org/wiki/OSI_model#Layer_4:_Transport_Layer): use TCP as the listener protocol for ports 80 and 443.
- [Layer 7](https://en.wikipedia.org/wiki/OSI_model#Layer_7:_Application_Layer): use HTTP as the listener protocol for port 80 and terminate TLS in the ELB
Patch the nginx ingress controller deployment to add the flag `--publish-service`
``` console
kubectl patch deployment -n ingress-nginx nginx-ingress-controller --type='json' \
--patch="$(curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/publish-service-patch.yaml)"
```
For L4:
``` console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/aws/service-l4.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/aws/patch-configmap-l4.yaml
```
For L7:
Change line of the file `provider/aws/service-l7.yaml` replacing the dummy id with a valid one `"arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX"`
Then execute:
``` console
kubectl apply -f provider/aws/service-l7.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/aws/patch-configmap-l7.yaml
```
This example creates an ELB with just two listeners, one in port 80 and another in port 443
![Listeners](../docs/images/elb-l7-listener.png)
If the ingress controller uses RBAC run:
``` console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/patch-service-with-rbac.yaml
```
If not run:
``` console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/patch-service-without-rbac.yaml
```
### GCE - GKE
Patch the nginx ingress controller deployment to add the flag `--publish-service`
``` console
kubectl patch deployment -n ingress-nginx nginx-ingress-controller --type='json' \
--patch="$(curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/publish-service-patch.yaml)"
```
``` console
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/gce-gke/service.yaml \
| kubectl apply -f -
```
If the ingress controller uses RBAC run:
``` console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/patch-service-with-rbac.yaml
```
If not run:
``` console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/patch-service-without-rbac.yaml
```
**Important Note:** proxy protocol is not supported in GCE/GKE
### Azure
Patch the nginx ingress controller deployment to add the flag `--publish-service`
``` console
kubectl patch deployment -n ingress-nginx nginx-ingress-controller --type='json' \
--patch="$(curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/publish-service-patch.yaml)"
```
``` console
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/azure/service.yaml \
| kubectl apply -f -
```
If the ingress controller uses RBAC run:
``` console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/patch-service-with-rbac.yaml
```
If not run:
``` console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/patch-service-without-rbac.yaml
```
**Important Note:** proxy protocol is not supported in GCE/GKE
### Baremetal
Using [NodePort](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport):
``` console
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml \
| kubectl apply -f -
```
Using Helm
----------
NGINX Ingress controller can be installed via [Helm](https://helm.sh/) using the chart [stable/nginx](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress) from the official charts repository.
To install the chart with the release name `my-nginx`:
``` console
helm install stable/nginx-ingress --name my-nginx
```
If the kubernetes cluster has RBAC enabled, then run:
``` console
helm install stable/nginx-ingress --name my-nginx --set rbac.create=true
```
Verify installation
-------------------
To check if the ingress controller pods have started, run the following command:
``` console
kubectl get pods --all-namespaces -l app=ingress-nginx --watch
```
Once the operator pods are running, you can cancel the above command by typing `Ctrl+C`.
Now, you are ready to create your first ingress.
Detect installed version
------------------------
To detect which version of the ingress controller is running, exec into the pod and run `nginx-ingress-controller version` command.
``` console
POD_NAMESPACE=ingress-nginx
POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app=ingress-nginx -o jsonpath={.items[0].metadata.name})
kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version
```
Deploying the config-map
------------------------
A config map can be used to configure system components for the nginx-controller. In order to begin using a config-map
make sure it has been created and is being used in the deployment.
It is created as seen in the [Mandatory Commands](#mandatory-commands) section above.
``` console
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/configmap.yaml \
| kubectl apply -f -
```
and is setup to be used in the deployment [without-rbac](without-rbac.yaml) or [with-rbac](with-rbac.yaml) with the following line:
``` yaml
- --configmap=$(POD_NAMESPACE)/nginx-configuration
```
For information on using the config-map, see its [user-guide](../docs/user-guide/configmap.md).

View file

@ -0,0 +1,10 @@
---
ingress_nginx_default_backend_image_repo: gcr.io/google_containers/defaultbackend
ingress_nginx_default_backend_image_tag: 1.4
ingress_nginx_controller_image_repo: quay.io/kubernetes-ingress-controller/nginx-ingress-controller
ingress_nginx_controller_image_tag: 0.11.0
ingress_nginx_namespace: "ingress-nginx"
ingress_nginx_insecure_port: 80
ingress_nginx_secure_port: 443

View file

@ -0,0 +1,42 @@
---
- name: NGINX Ingress Controller | Create addon dir
file:
path: "{{ kube_config_dir }}/addons/ingress_nginx"
state: directory
owner: root
group: root
mode: 0755
- name: NGINX Ingress Controller | Create manifests
template:
src: "{{ item.file }}.j2"
dest: "{{ kube_config_dir }}/addons/ingress_nginx/{{ item.file }}"
with_items:
- { name: ingress-nginx-ns, file: ingress-nginx-ns.yml, type: ns }
- { name: ingress-nginx-sa, file: ingress-nginx-sa.yml, type: sa }
- { name: ingress-nginx-role, file: ingress-nginx-role.yml, type: role }
- { name: ingress-nginx-rolebinding, file: ingress-nginx-rolebinding.yml, type: rolebinding }
- { name: ingress-nginx-clusterrole, file: ingress-nginx-clusterrole.yml, type: clusterrole }
- { name: ingress-nginx-clusterrolebinding, file: ingress-nginx-clusterrolebinding.yml, type: clusterrolebinding }
- { name: ingress-nginx-cm, file: ingress-nginx-cm.yml, type: cm }
- { name: ingress-nginx-tcp-servicecs-cm, file: ingress-nginx-tcp-servicecs-cm.yml, type: cm }
- { name: ingress-nginx-udp-servicecs-cm, file: ingress-nginx-udp-servicecs-cm.yml, type: cm }
- { name: ingress-nginx-default-backend-svc, file: ingress-nginx-default-backend-svc.yml, type: svc }
- { name: ingress-nginx-default-backend-rs, file: ingress-nginx-default-backend-rs.yml, type: rs }
- { name: ingress-nginx-controller-ds, file: ingress-nginx-controller-ds.yml, type: ds }
register: ingress_nginx_manifests
when:
- inventory_hostname == groups['kube-master'][0]
- name: NGINX Ingress Controller | Apply manifests
kube:
name: "{{ item.item.name }}"
namespace: "{{ ingress_nginx_namespace }}"
kubectl: "{{ bin_dir }}/kubectl"
resource: "{{ item.item.type }}"
filename: "{{ kube_config_dir }}/addons/ingress_nginx/{{ item.item.file }}"
state: "latest"
with_items: "{{ ingress_nginx_manifests.results }}"
when:
- inventory_hostname == groups['kube-master'][0]

View file

@ -0,0 +1,25 @@
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: ingress-nginx
namespace: {{ ingress_nginx_namespace }}
rules:
- apiGroups: [""]
resources: ["configmaps", "endpoints", "nodes", "pods", "secrets"]
verbs: ["list", "watch"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get"]
- apiGroups: [""]
resources: ["services"]
verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
resources: ["ingresses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "patch"]
- apiGroups: ["extensions"]
resources: ["ingresses/status"]
verbs: ["update"]

View file

@ -0,0 +1,14 @@
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: ingress-nginx
namespace: {{ ingress_nginx_namespace }}
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: {{ ingress_nginx_namespace }}
roleRef:
kind: ClusterRole
name: ingress-nginx
apiGroup: rbac.authorization.k8s.io

View file

@ -0,0 +1,8 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: ingress-nginx
namespace: {{ ingress_nginx_namespace }}
labels:
k8s-app: ingress-nginx

View file

@ -0,0 +1,70 @@
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: ingress-nginx-controller
namespace: {{ ingress_nginx_namespace }}
labels:
k8s-app: ingress-nginx
version: v{{ ingress_nginx_controller_image_tag }}
annotations:
prometheus.io/port: '10254'
prometheus.io/scrape: 'true'
spec:
selector:
matchLabels:
k8s-app: ingress-nginx
version: v{{ ingress_nginx_controller_image_tag }}
template:
metadata:
labels:
k8s-app: ingress-nginx
version: v{{ ingress_nginx_controller_image_tag }}
spec:
serviceAccountName: ingress-nginx
containers:
- name: ingress-nginx-controller
image: {{ ingress_nginx_controller_image_repo }}:{{ ingress_nginx_controller_image_tag }}
imagePullPolicy: {{ k8s_image_pull_policy }}
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/ingress-nginx-default-backend
- --configmap=$(POD_NAMESPACE)/ingress-nginx
- --tcp-services-configmap=$(POD_NAMESPACE)/ingress-nginx-tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/ingress-nginx-udp-services
- --annotations-prefix=nginx.ingress.kubernetes.io
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
hostPort: {{ ingress_nginx_insecure_port }}
- name: https
containerPort: 443
hostPort: {{ ingress_nginx_secure_port }}
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1

View file

@ -0,0 +1,37 @@
---
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: ingress-nginx-default-backend-v{{ ingress_nginx_default_backend_image_tag }}
namespace: {{ ingress_nginx_namespace }}
labels:
k8s-app: ingress-nginx-default-backend
version: v{{ ingress_nginx_default_backend_image_tag }}
spec:
replicas: 1
selector:
matchLabels:
k8s-app: ingress-nginx-default-backend
version: v{{ ingress_nginx_default_backend_image_tag }}
template:
metadata:
labels:
k8s-app: ingress-nginx-default-backend
version: v{{ ingress_nginx_default_backend_image_tag }}
spec:
terminationGracePeriodSeconds: 60
containers:
- name: ingress-nginx-default-backend
# Any image is permissible as long as:
# 1. It serves a 404 page at /
# 2. It serves 200 on a /healthz endpoint
image: {{ ingress_nginx_default_backend_image_repo }}:{{ ingress_nginx_default_backend_image_tag }}
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
ports:
- containerPort: 8080

View file

@ -0,0 +1,14 @@
---
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx-default-backend
namespace: {{ ingress_nginx_namespace }}
labels:
k8s-app: ingress-nginx-default-backend
spec:
ports:
- port: 80
targetPort: 8080
selector:
k8s-app: ingress-nginx-default-backend

View file

@ -0,0 +1,5 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: {{ ingress_nginx_namespace }}

View file

@ -0,0 +1,24 @@
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: ingress-nginx
namespace: {{ ingress_nginx_namespace }}
rules:
- apiGroups: [""]
resources: ["configmaps", "pods", "secrets", "namespaces"]
verbs: ["get"]
- apiGroups: [""]
resources: ["configmaps"]
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
resourceNames: ["ingress-controller-leader-nginx"]
verbs: ["get", "update"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["create"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get"]

View file

@ -0,0 +1,14 @@
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: ingress-nginx
namespace: {{ ingress_nginx_namespace }}
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: {{ ingress_nginx_namespace }}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx

View file

@ -0,0 +1,6 @@
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: ingress-nginx
namespace: {{ ingress_nginx_namespace }}

View file

@ -0,0 +1,8 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: ingress-nginx-tcp-services
namespace: {{ ingress_nginx_namespace }}
labels:
k8s-app: ingress-nginx

View file

@ -0,0 +1,8 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: ingress-nginx-udp-services
namespace: {{ ingress_nginx_namespace }}
labels:
k8s-app: ingress-nginx

View file

@ -0,0 +1,8 @@
---
dependencies:
- role: kubernetes-apps/ingress_controller/ingress_nginx
when: ingress_nginx_enabled
tags:
- apps
- ingress-nginx
- ingress-controller

View file

@ -173,6 +173,7 @@ enable_network_policy: false
local_volume_provisioner_enabled: "{{ local_volumes_enabled | default('false') }}"
persistent_volumes_enabled: false
cephfs_provisioner_enabled: false
ingress_nginx_enabled: false
## When OpenStack is used, Cinder version can be explicitly specified if autodetection fails (Fixed in 1.9: https://github.com/kubernetes/kubernetes/issues/50461)
# openstack_blockstorage_version: "v1/v2/auto (default)"