C12s/c12s-kubespray
3
0
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity

Adds support for Multus (multiple interfaces) CNI plugin (#3166)

Browse source 
* Adds support for Multus (multiple interfaces) CNI plugin

Multus is a latin word for "Multi". As the name suggests, it acts as a
Multi plugin in Kubernetes and provides multiple network interface
support in a pod. Multus uses the concept of invoking delegates by
grouping multiple plugins into delegates and invoking them in the
sequential order of the CNI configuration file provided in json format.

* Change CNI version (0.1.0->0.3.1) of Contiv to be compatible with Multus
This commit is contained in:
Louis Woods 2018-11-04 01:07:38 -08:00 committed by k8s-ci-robot
parent 3c5f20190f
commit bc9e14a762
19 changed files with 344 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

15
.gitlab-ci.yml
View file

@ -300,6 +300,10 @@ before_script:
# stage: deploy-special
MOVED_TO_GROUP_VARS: "true"
.centos7_multus_calico_variables: &centos7_multus_calico_variables
# stage: deploy-part2
MOVED_TO_GROUP_VARS: "true"
.coreos_alpha_weave_ha_variables: &coreos_alpha_weave_ha_variables
# stage: deploy-special
MOVED_TO_GROUP_VARS: "true"
@ -638,6 +642,17 @@ gce_centos7-kube-router:
except: ['triggers']
only: ['master', /^pr-.*$/]
gce_centos7-multus-calico:
stage: deploy-part2
<<: *job
<<: *gce
variables:
<<: *gce_variables
<<: *centos7_multus_calico_variables
when: manual
except: ['triggers']
only: ['master', /^pr-.*$/]
gce_opensuse-canal:
stage: deploy-part2
<<: *job

3
README.md
View file

@ -125,6 +125,7 @@ Supported Components
- [weave](https://github.com/weaveworks/weave) v2.4.1
- [kube-router](https://github.com/cloudnativelabs/kube-router) v0.2.1
- Application
- [multus](https://github.com/intel/multus-cni) v3.1
- [cephfs-provisioner](https://github.com/kubernetes-incubator/external-storage) v2.1.0-k8s1.11
- [cert-manager](https://github.com/jetstack/cert-manager) v0.5.0
- [coredns](https://github.com/coredns/coredns) v1.2.5
@ -176,6 +177,8 @@ You can choose between 6 network plugins. (default: `calico`, except Vagrant use
iptables for network policies, and BGP for ods L3 networking (with optionally BGP peering with out-of-cluster BGP peers).
It can also optionally advertise routes to Kubernetes cluster Pods CIDRs, ClusterIPs, ExternalIPs and LoadBalancerIPs.
- [multus](docs/multus.md): Multus is a meta CNI plugin that provides multiple network interface support to pods. For each interface Multus delegates CNI calls to secondary CNI plugins such as Calico, macvlan, etc.
The choice is defined with the variable `kube_network_plugin`. There is also an
option to leverage built-in cloud provider networking instead.
See also [Network checker](docs/netcheck.md).

5
Vagrantfile vendored
View file

@ -35,6 +35,8 @@ $forwarded_ports = {}
$subnet = "172.17.8"
$os = "ubuntu1804"
$network_plugin = "flannel"
# Setting multi_networking to true will install Multus: https://github.com/intel/multus-cni
$multi_networking = false
# The first three nodes are etcd servers
$etcd_instances = $num_instances
# The first two nodes are kube masters
@ -140,7 +142,8 @@ Vagrant.configure("2") do |config|
"ip": ip,
"local_release_dir" => $local_release_dir,
"download_run_once": "False",
"kube_network_plugin": $network_plugin
"kube_network_plugin": $network_plugin,
"kube_network_plugin_multus": $multi_networking
}
config.vm.network :private_network, ip: ip

73
docs/multus.md Normal file
View file

@ -0,0 +1,73 @@
Multus
===========
Multus is a meta CNI plugin that provides multiple network interface support to
pods. For each interface, Multus delegates CNI calls to secondary CNI plugins
such as Calico, macvlan, etc.
See [multus documentation](https://github.com/intel/multus-cni).
## Multus installation
Since Multus itself does not implement networking, it requires a master plugin, which is specified through the variable `kube_network_plugin`. To enable Multus an additional variable `kube_network_plugin_multus` must be set to `true`. For example,
```
kube_network_plugin: calico
kube_network_plugin_multus: true
```
will install Multus and Calico and configure Multus to use Calico as the primary network plugin.
## Using Multus
Once Multus is installed, you can create CNI configurations (as a CRD objects) for additional networks, in this case a macvlan CNI configuration is defined. You may replace the config field with any valid CNI configuration where the CNI binary is available on the nodes.
```
cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: macvlan-conf
spec:
config: '{
"cniVersion": "0.3.0",
"type": "macvlan",
"master": "eth0",
"mode": "bridge",
"ipam": {
"type": "host-local",
"subnet": "192.168.1.0/24",
"rangeStart": "192.168.1.200",
"rangeEnd": "192.168.1.216",
"routes": [
{ "dst": "0.0.0.0/0" }
],
"gateway": "192.168.1.1"
}
}'
EOF
```
You may then create a pod with and additional interface that connects to this network using annotations. The annotation correlates to the name in the NetworkAttachmentDefinition above.
```
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: samplepod
annotations:
k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
containers:
- name: samplepod
command: ["/bin/bash", "-c", "sleep 2000000000000"]
image: dougbtv/centos-network
EOF
```
You may now inspect the pod and see that there is an additional interface configured:
```
$ kubectl exec -it samplepod -- ip a
```
For more details on how to use Multus, please visit https://github.com/intel/multus-cni

12
roles/download/defaults/main.yml
View file

@ -59,6 +59,7 @@ pod_infra_version: 3.1
contiv_version: 1.2.1
cilium_version: "v1.3.0"
kube_router_version: "v0.2.1"
multus_version: "v3.1.autoconf"
# Download URLs
kubeadm_download_url: "https://storage.googleapis.com/kubernetes-release/release/{{ kubeadm_version }}/bin/linux/{{ image_arch }}/kubeadm"
@ -160,6 +161,8 @@ cilium_image_repo: "docker.io/cilium/cilium"
cilium_image_tag: "{{ cilium_version }}"
kube_router_image_repo: "cloudnativelabs/kube-router"
kube_router_image_tag: "{{ kube_router_version }}"
multus_image_repo: "docker.io/nfvpe/multus"
multus_image_tag: "{{ multus_version }}"
nginx_image_repo: nginx
nginx_image_tag: 1.13
dnsmasq_version: 2.78
@ -290,6 +293,15 @@ downloads:
groups:
- k8s-cluster
multus:
enabled: "{{ kube_network_plugin_multus }}"
container: true
repo: "{{ multus_image_repo }}"
tag: "{{ multus_image_tag }}"
sha256: "{{ multus_digest_checksum|default(None) }}"
groups:
- k8s-cluster
flannel:
enabled: "{{ kube_network_plugin == 'flannel' or kube_network_plugin == 'canal' }}"
container: true

5
roles/kubernetes-apps/network_plugin/meta/main.yml
View file

@ -34,3 +34,8 @@ dependencies:
when: kube_network_plugin == 'kube-router'
tags:
- kube-router
- role: kubernetes-apps/network_plugin/multus
when: kube_network_plugin_multus
tags:
- multus

11
roles/kubernetes-apps/network_plugin/multus/tasks/main.yml Normal file
View file

@ -0,0 +1,11 @@
---
- name: Multus | Start resources
kube:
name: "{{item.item.name}}"
namespace: "kube-system"
kubectl: "{{bin_dir}}/kubectl"
resource: "{{item.item.type}}"
filename: "{{kube_config_dir}}/{{item.item.file}}"
state: "latest"
with_items: "{{ multus_manifest_1.results }} + {{multus_manifest_2.results }}"
when: inventory_hostname == groups['kube-master'][0] and not item|skipped

1
roles/kubespray-defaults/defaults/main.yaml
View file

@ -103,6 +103,7 @@ kube_users:
# Choose network plugin (cilium, calico, weave or flannel)
# Can also be set to 'cloud', which lets the cloud provider setup appropriate routing
kube_network_plugin: calico
kube_network_plugin_multus: false
# Determines if calico-rr group exists
peer_with_calico_rr: "{{ 'calico-rr' in groups and groups['calico-rr']|length > 0 }}"

2
roles/network_plugin/contiv/defaults/main.yml
View file

@ -4,7 +4,7 @@ contiv_config_dir: "{{ kube_config_dir }}/contiv"
contiv_etcd_conf_dir: "/etc/contiv/etcd"
contiv_etcd_data_dir: "/var/lib/etcd/contiv-data"
contiv_netmaster_port: 9999
contiv_cni_version: 0.1.0
contiv_cni_version: 0.3.1
# No need to download it by default, but must be defined
contiv_etcd_image_repo: "{{ etcd_image_repo }}"

5
roles/network_plugin/meta/main.yml
View file

@ -37,3 +37,8 @@ dependencies:
- role: network_plugin/cloud
when: kube_network_plugin == 'cloud'
- role: network_plugin/multus
when: kube_network_plugin_multus
tags:
- multus

7
roles/network_plugin/multus/defaults/main.yml Normal file
View file

@ -0,0 +1,7 @@
---
multus_conf_file: "auto"
multus_cni_conf_dir_host: "/etc/cni/net.d"
multus_cni_bin_dir_host: "/opt/cni/bin"
multus_cni_conf_dir: "{{ ('/host', multus_cni_conf_dir_host) | join }}"
multus_cni_bin_dir: "{{ ('/host', multus_cni_bin_dir_host) | join }}"
multus_kubeconfig_file_host: "{{ (multus_cni_conf_dir_host, '/multus.d/multus.kubeconfig') | join }}"

16
roles/network_plugin/multus/files/multus-clusterrole.yml Normal file
View file

@ -0,0 +1,16 @@
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: multus
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'

13
roles/network_plugin/multus/files/multus-clusterrolebinding.yml Normal file
View file

@ -0,0 +1,13 @@
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: multus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: multus
subjects:
- kind: ServiceAccount
name: multus
namespace: kube-system

22
roles/network_plugin/multus/files/multus-crd.yml Normal file
View file

@ -0,0 +1,22 @@
---
kind: CustomResourceDefinition
apiVersion: apiextensions.k8s.io/v1beta1
metadata:
name: network-attachment-definitions.k8s.cni.cncf.io
spec:
group: k8s.cni.cncf.io
version: v1
scope: Namespaced
names:
plural: network-attachment-definitions
singular: network-attachment-definition
kind: NetworkAttachmentDefinition
shortNames:
- net-attach-def
validation:
openAPIV3Schema:
properties:
spec:
properties:
config:
type: string

6
roles/network_plugin/multus/files/multus-serviceaccount.yml Normal file
View file

@ -0,0 +1,6 @@
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: multus
namespace: kube-system

19
roles/network_plugin/multus/tasks/main.yml Normal file
View file

@ -0,0 +1,19 @@
---
- name: Multus | Copy manifest files
copy:
src: "{{ item.file }}"
dest: "{{ kube_config_dir }}"
with_items:
- {name: multus-crd, file: multus-crd.yml, type: customresourcedefinition}
- {name: multus-serviceaccount, file: multus-serviceaccount.yml, type: serviceaccount}
- {name: multus-clusterrole, file: multus-clusterrole.yml, type: clusterrole}
- {name: multus-clusterrolebinding, file: multus-clusterrolebinding.yml, type: clusterrolebinding}
register: multus_manifest_1
- name: Multus | Copy manifest templates
template:
src: "{{ item.file }}.j2"
dest: "{{ kube_config_dir }}/{{ item.file }}"
with_items:
- {name: multus-daemonset, file: multus-daemonset.yml, type: daemonset}
register: multus_manifest_2

54
roles/network_plugin/multus/templates/multus-daemonset.yml.j2 Normal file
View file

@ -0,0 +1,54 @@
---
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
name: kube-multus-ds-amd64
namespace: kube-system
labels:
tier: node
app: multus
spec:
template:
metadata:
labels:
tier: node
app: multus
spec:
hostNetwork: true
nodeSelector:
beta.kubernetes.io/arch: amd64
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
serviceAccountName: multus
containers:
- name: kube-multus
image: {{ multus_image_repo }}:{{ multus_image_tag }}
command: ["/entrypoint.sh"]
args:
- "--cni-conf-dir={{ multus_cni_conf_dir }}"
- "--cni-bin-dir={{ multus_cni_bin_dir }}"
- "--multus-conf-file={{ multus_conf_file }}"
- "--multus-kubeconfig-file-host={{ multus_kubeconfig_file_host }}"
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: true
volumeMounts:
- name: cni
mountPath: {{ multus_cni_conf_dir }}
- name: cnibin
mountPath: {{ multus_cni_bin_dir }}
volumes:
- name: cni
hostPath:
path: {{ multus_cni_conf_dir_host }}
- name: cnibin
hostPath:
path: {{ multus_cni_bin_dir_host }}

12
tests/files/gce_centos7-multus-calico.yml Normal file
View file

@ -0,0 +1,12 @@
# Instance settings
cloud_image_family: centos-7
cloud_region: us-central1-c
cloud_machine_type: "n1-standard-1"
mode: default
# Deployment settings
kube_network_plugin_multus: true
kube_network_plugin: calico
deploy_netchecker: true
kubedns_min_replicas: 1
cloud_provider: gce

65
tests/testcases/040_check-network-adv.yml
View file

@ -80,3 +80,68 @@
run_once: true
when:
- agents.content == '{}'
- name: Create macvlan network conf
# We cannot use only shell: below because Ansible will render the text
# with leading spaces, which means the shell will never find the string
# EOF at the beginning of a line. We can avoid Ansible's unhelpful
# heuristics by using the cmd parameter like this:
shell:
cmd: |
cat <<EOF | {{ bin_dir }}/kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: macvlan-conf
spec:
config: '{
"cniVersion": "0.3.0",
"type": "macvlan",
"master": "eth0",
"mode": "bridge",
"ipam": {
"type": "host-local",
"subnet": "192.168.1.0/24",
"rangeStart": "192.168.1.200",
"rangeEnd": "192.168.1.216",
"routes": [
{ "dst": "0.0.0.0/0" }
],
"gateway": "192.168.1.1"
}
}'
EOF
when:
- kube_network_plugin_multus|default(false)
- name: Annotate pod with macvlan network
# We cannot use only shell: below because Ansible will render the text
# with leading spaces, which means the shell will never find the string
# EOF at the beginning of a line. We can avoid Ansible's unhelpful
# heuristics by using the cmd parameter like this:
shell:
cmd: |
cat <<EOF | {{ bin_dir }}/kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: samplepod
annotations:
k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
containers:
- name: samplepod
command: ["/bin/bash", "-c", "sleep 2000000000000"]
image: dougbtv/centos-network
EOF
when:
- kube_network_plugin_multus|default(false)
- name: Check secondary macvlan interface
shell: "{{ bin_dir }}/kubectl exec samplepod -- ip addr show dev net1"
register: output
until: output.rc == 0
retries: 90
changed_when: false
when:
- kube_network_plugin_multus|default(false)