Commit graph

2975 commits

Author SHA1 Message Date
Victor Morales
da089b5fca Update CRI-O in CentOS (#4582)
According to their compatibility matrix[1] the 1.11.5 version seems to
be deprecated. This change updates the CentOS repository reference.

[1] https://github.com/cri-o/cri-o#compatibility-matrix-cri-o---kubernetes-clusters
2019-08-22 01:16:32 -07:00
Sergey
494a6512b8 fix bug: run Copy image to ansible host cache on download_delegate host (#5094)
* run 'task download_container | Copy image to ansible host cache' with synchronize on download_delegate host

* try to run task copy file to ansible host on all inventory, not only on first random host
2019-08-21 23:38:30 -07:00
Tony Fouchard
f6a63d88a7 Allow to configure strict ARP on kube-proxy (#5092) 2019-08-20 18:21:17 -07:00
Andreas Krüger
86cc703c75 Upgrade to Kubernetes 1.15.3 (#5091) 2019-08-20 02:05:32 -07:00
Hugo Blom
4dba34bd02 add cinder max attached volumes (#5089) 2019-08-19 23:45:32 -07:00
Xiaodu
b0437516c1 Kube-router annotate.yml: Use group 'k8s-cluster' instead of 'all' (#5087) (#5088) 2019-08-19 04:53:29 -07:00
Ali Sanhaji
a1ff1de975 fix openstack_cacert conditional (#5078) 2019-08-15 05:50:34 -07:00
Zou Nengren
1bfbc5bbc4 remove resource-container default value for kube-proxy (#4994) 2019-08-15 05:30:33 -07:00
Bort Verwilst
c5b4d3ceaa upgrade Helm to 2.14.3 (#5075)
Signed-off-by: Bart Verwilst <bart@verwilst.be>
2019-08-15 04:34:33 -07:00
w33dw0r7d
8fc9c5d025 Upgrade ingress nginx to 0.25.1 (#5081) 2019-08-15 04:14:34 -07:00
刘旭
53bc80bb59 Ingress nginx (#5066)
* remove svc-default-backend

* update ingress-nginx clusterrole
2019-08-15 02:34:33 -07:00
Matthew Mosesohn
771ce96e6d Set initial kubeadm token if specified in kubeadm init (#5057)
Change-Id: I7fd94ec6d195af60d237b3cfe91668ca1f707d26
2019-08-15 02:26:33 -07:00
Oilbeater
fc456ff0cd move kube-ovn images to dockerhub (#5063) 2019-08-14 04:02:24 -07:00
Sergey Kolekonov
b4f70db878 Fix broken containerd pinning on Ubuntu (#5072) 2019-08-13 19:26:23 -07:00
Matthew Mosesohn
0a2f4edfc6 Always download coredns images with kubeadm (#5071)
Fixes situation when using manual mode because it
tries to download coredns v1.3.1 from the same
image repository where kubernetes images are
downloaded from.

Change-Id: Ibbec8a72c8162ce8befa74e2013a268737ea5f8a
2019-08-13 08:53:43 -07:00
Danilo Riecken P. de Morais
56fa46716e Add missing coredns tag. (#5054) 2019-08-09 02:29:27 -07:00
Simon Lelievre
62aecd1e4a multus | fix use last version (#5041) 2019-08-08 23:05:25 -07:00
Mario
973afef96e Fix variable for rbd_provisioner_user_secret (#5042)
* Update main.yml

* fix dead link 404
2019-08-08 20:03:25 -07:00
Bort Verwilst
a235605d2c go to k8s 1.15.2, update nodelocaldns to latest bugfix release (#5048) 2019-08-08 19:49:25 -07:00
Matthew Mosesohn
023108a733 Refactor calico route reflector to run in k8s cluster (#4975)
* Refactor calico-rr to run in k8s cluster with taint

Change-Id: I75a3169ff5b36ce8302fc7ef1c32d3eb697b5afa

* add preinstall checks

* rework calico/rr role

Change-Id: I2f0a7e6cb77cf91ad4a615923680760d2e5d9ca8

* add empty calico-rr group

Change-Id: I006c0a60db9b72d02245bf8fdfabcf982144a5ad
2019-08-08 07:37:22 -07:00
Matthew Mosesohn
75d1be8272 Fix check for removing etcd member (#5051)
Change-Id: Ib27d051ff111f813097a9b33a86465a2a30a6db0
2019-08-07 08:26:51 -07:00
Matthew Mosesohn
a44235d11b Refactor remove node to allow removing dead nodes and etcd members (#5009)
Change-Id: I1c59249f08f16d0f6fd60df6ab61f17a0a7df189
2019-08-07 04:46:50 -07:00
Matthew Mosesohn
7abf6a6958 Allow etcd member join by checking cluster health only on first etcd (#5032)
Change-Id: I9cc01cef3a437893225e2d9f58495826bbce7be9
2019-08-07 04:44:50 -07:00
Maxim Snezhkov
b710c72f04 Add ability to setup virtual ip for ingress-controller (#5044) 2019-08-06 19:24:50 -07:00
Holger Frydrych
bc6de32faf Upgrade Cilium network plugin to v1.5.5. (#5014)
* Needs an additional cilium-operator deployment.
  * Added option to enable hostPort mappings.
2019-08-06 01:37:55 -07:00
Matthew Mosesohn
7cf8ad4dc7 Optionally refresh kubeadm token every time (#5043)
Change-Id: I278cb14aa93abf20160cc001f69e2f472504e6d8
2019-08-06 00:59:53 -07:00
Remous-Aris Koutsiamanis
02ec72fa40 Fix commands for using experimental kubeadm control plane (#5006) 2019-08-05 07:31:50 -07:00
Johannes Scheuermann
d22634a597 Refactor containerd ubuntu setup and remove redundant tasks (#5015) 2019-08-05 07:29:48 -07:00
Mark Janssen
f3df0d5f4a Always create bash_completion.d folder (#5039) 2019-08-04 18:15:48 -07:00
w33dw0r7d
92bfcf0467 Add CoreDNS endpoint_pod_names option (#5012) 2019-07-31 11:26:15 -07:00
koriukiv
54b1fe83f3 Add an option to reserve resources for OS system daemons (#5007) 2019-07-31 11:24:15 -07:00
Oilbeater
1be788f785 add Kube-OVN cni to kubespray (#5020) 2019-07-30 20:10:20 -07:00
rptaylor
8afbf339f7 fix broken link (#5023) 2019-07-30 19:18:22 -07:00
Andreas Krüger
8c935dfb50 Update CoreDNS to 1.6.0 (#5021) 2019-07-30 18:58:21 -07:00
Johannes Scheuermann
66c5ed8406 Update critools to v1.15.0 (#5016) 2019-07-30 12:04:09 -07:00
Erwan Miran
4087e97505 Additional files and dirs to remove when running reset (#5000) 2019-07-30 12:02:08 -07:00
Jeff Bornemann
da50ed0936 move flexvolume plugin directory creation to preinstall (#4999)
* move flexvolume plugin directory creation to preinstall

* changes per pr feedback
2019-07-30 12:00:10 -07:00
okamototk
fbbfff3795 fix broken ubuntu containerd engine (#5002) 2019-07-30 11:58:11 -07:00
Aleksey Kasatkin
fb9103acd3 Update calico-typha deployment to address v3.7.x changes (#5003)
* Update calico-typha deployment to address v3.7.x changes

So that calico-typha works for Calico v3.7.x.

* Apply changes for v3.7.x only.
2019-07-24 09:12:16 -07:00
nico-netminded
49d921cf91 Restart canal after scale or upgrade. Just like PR#4531, but for canal (#4992) 2019-07-22 00:50:53 -07:00
刘旭
fe29c97ae8 add ansible_hostname and ansible_fqdn to apiserver_sans (#4990) 2019-07-22 00:48:53 -07:00
Hugo Blom
2abb6c8689 update to kubernetes 1.15.1 (#4989)
* update to kubernetes 1.15.1

* Revert to sonobuoy 0.15.0

* update test timeout from 3 to 5 minutes
2019-07-21 12:24:51 -07:00
Andreas Krüger
a3ca441998 Remove unused handlers from Flannel CNI (#4984)
* Only reload docker when is_atomic for Flannel

* Remove unused handlers from Flannel CNI
2019-07-21 00:16:54 -07:00
rptaylor
9cf503acb1 configure docker_options directly with template (#4912) 2019-07-21 00:12:53 -07:00
Matthew Mosesohn
1cbdd7ed5c Fixup etcdctl download for etcd kubeadm mode (#4991)
Change-Id: I8d8e59a97823390f40e8810905ca52430329f040
2019-07-21 00:08:53 -07:00
Sergey Kolekonov
428e52e0d1 Fix calico handler for containerd (#4985)
crictl tool must be used to delete containers in case of containerd
deployment
2019-07-16 08:35:24 -07:00
Matthew Mosesohn
70dc222719 Upgrade local volume provisioner to v2.3.2 (#4983) 2019-07-16 05:27:26 -07:00
Tilman Beitter
69f796f0c7 use the locally deployed kubectl binary within the kubectl.sh helper script (#4311) 2019-07-16 02:23:25 -07:00
Vincent Link
5826f0810c Check all apt config files for configured proxies (#4972) 2019-07-16 01:41:24 -07:00
刘旭
de9443a694 remove unused code (#4981) 2019-07-16 01:39:24 -07:00
Alex Barcelo
99c5f7e013 add k8s_external plugin to CoreDNS configuration (#4704) 2019-07-16 00:53:23 -07:00
Matthew Mosesohn
23ae6027ab remove support for calico v2.x (#4974)
* Remove support for calico below version v3.0.0

Change-Id: If8fe3036b9e054901a8b2c48516eff1e1271970f

* Update main.yml

* fixup node peering

Change-Id: Ifac4d363deba826f0c80e390ce80a28df9827323

* fixups

Change-Id: Ic35417330af6741962003b3930604393c90804d1

* fixups

Change-Id: I0ea82d634bb0c81d9b7dc50569c70988bc8d3a3b
2019-07-15 07:47:09 -07:00
Waret
781b5691c9 prep_kubeadm_images: parse repo and tag (#4976) (#4977) 2019-07-15 03:15:09 -07:00
Matthew Mosesohn
fd9bbcb157 Enable nodes to run calicoctl for calico kdd mode (#4956)
* Enable nodes to run calicoctl

per-node tasks require waiting for calico-node to be applied

Change-Id: Ibe1076b7334a2da0332f2dd766fde0c3f172d1f2

* cleanup tasks that should run on master

Change-Id: I43a837879ef41596f14657ecd7f813899b6865ae

* Switch run_once calico logic to just run on first master

Change-Id: I6893711e354f63c5e1eaf6ac2e23d9a6347a555d
2019-07-15 01:59:06 -07:00
Andreas Krüger
161a8f55fa Update CoreDNS to 1.5.2 (#4970) 2019-07-15 00:59:06 -07:00
Andreas Krüger
7481cc31e1 Update Weave CNI to 2.5.2 (#4969) 2019-07-15 00:57:06 -07:00
Matthew Mosesohn
b15b6e834f fix parsing refresh of kubeadm cert key (#4971)
* fix parsing refresh of kubeadm cert key

Change-Id: I4de2a1df6498790a80351b4bc7d88e6c9e470358

* Update kubeadm-secondary-experimental.yml
2019-07-15 00:45:06 -07:00
Hugo Blom
76640cf1a1 update docker-ce to 18.09.7 (#4973) 2019-07-14 22:59:04 -07:00
Sergey Kolekonov
46bef931e9 Fix images info logic for containerd (#4965)
As crictl tool is used to download images, it must be also used to gather
images info
2019-07-12 03:29:07 -07:00
Stanislav Makar
a36e9ae690 Add checksums for k8s 1.14.4 (#4959) 2019-07-11 23:19:05 -07:00
Jeff Bornemann
728155a2a1 Support for Oracle Linux (#3655)
Fixed Issue #1032

test case for OEL7 AIL with kubeadm

Add packet CI stuff for oracle 7
2019-07-11 23:17:05 -07:00
Matthew Mosesohn
cdf9a9f4fc Generate certificate key before kubeadm control plane config (#4964) 2019-07-11 05:30:54 -07:00
Matthew Mosesohn
29307740dd Enable containerd to deploy vanilla containerd package (#4951)
* Enable containerd to deploy vanilla containerd package

Fixes kubeadm references to CRI socket for containerd
Fixes download role cache feature to work with containerd

Change-Id: I2ab8f0031107e2f0d1a85c39b4beb66f08509a01

* use containerd for flannel-addons job

Change-Id: Ied375c7d65e64a625ffbd995ff16f2374067dee6

* add containerd vars

Change-Id: Ib9a8a04e501c481a86235413cbec63f3672baf91

* fixup vars

Change-Id: Ibea64e4b18405a578b52a13da100384582aa24c2

* more fixes

* fix rh repo

Change-Id: I00575a77cfb7b81d6095db5d918a52023c8f13ba

* Adjust helm host install for containerd
2019-07-10 23:46:54 -07:00
jlacoline
20c7e31ea3 Add calico 3.7.3 support (#4953)
* Add calico 3.7.3 support

* add calico_datastore variable to policy controller role

* add missing clusterrole rules for calico policy controller

* disable calico kube controller when kdd mode is used for versions < 3.6
2019-07-09 12:42:28 -07:00
Matthew Mosesohn
65065e7fdf Default kubeadm_images to empty dict instead of set_fact (#4957)
Change-Id: I9ef52232176fe8e2a99b4f09ae05e1451f7ad1ff
2019-07-09 12:07:30 -07:00
Matthew Mosesohn
352297cf8d
Fixup deploy of kubeadm etcd for Kubernetes v1.15.0 (#4952)
* Fixup deploy of kubeadm etcd for Kubernetes v1.15.0

Change-Id: If42c2c75c4d278ba9475ebf76c243f3e6ee4d02e

* undo renaming cloud config file

Change-Id: Iafbd27c3887d6a2a6d0819c711f150ecf70c515d
2019-07-09 15:41:59 +03:00
champtar
a67a50f9c0 nodelocaldns: allow to set health port, switch to 9254 by default (#4902)
8080 is a pretty common port, using nodelocaldns_ip:8080 still
prevents node processes or hostNetwork=true processes to bind to *:8080
so switch to 9254 by default (prometheus port is 9253)

Signed-off-by: Etienne Champetier <champetier.etienne@gmail.com>
2019-07-09 00:52:01 -07:00
rptaylor
324bc41097 Add support for Docker plugins (#4934)
* Add support for Docker plugins

* support multiple Docker plugins using looped include

* fix yamllint error
2019-07-08 06:44:35 -07:00
andreyshestakov
c81b443d93 Fix order of names in /etc/hosts (#4940)
Configure fqdn properly
2019-07-08 06:08:34 -07:00
Julian Tabel
dc16ab92f4 fix for calico with kdd datastore (#4922)
* fix for calico with kdd datastore

* remove AS number from daemonset

* revert changes to canal

* additionnal fixes for kdd datastore in calico
2019-07-08 12:20:03 +03:00
Konrad Kurdej
d90a5f291b Using also uppercase proxy env variables (#4910) 2019-07-02 18:13:12 -07:00
Evans Tucker
3b7791501e Adding "-F /dev/null" to load null SSH config file. (#4933) 2019-07-02 01:53:08 -07:00
okamototk
f2b8a3614d Use K8s 1.15 (#4905)
* Use K8s 1.15

* Use Kubernetes 1.15 and use kubeadm.k8s.io/v1beta2 for
  InitConfiguration.
* bump to v1.15.0

* Remove k8s 1.13 checksums.

* Update README kubernetes version 1.15.0.

* Update metrics server 0.3.3 for k8s 1.15

* Remove less than k8s 1.14 related code

* Use kubeadm with --upload-certs instead of --experimental-upload-certs due to depricate

* Update dnsautoscaler 1.6.0

* Skip certificateKey if it's not defined

* Add kubeadm-conftolplane.v2beta2 for k8s 1.15 or later

* Support kubeadm control plane for k8s 1.15

* Update sonobuoy version 0.15.0 for k8s 1.15
2019-07-02 01:51:08 -07:00
Matthew Mosesohn
e89b47c7ee Add nginx stub metrics if health check enabled (#4938)
Change-Id: Iac90beef20e63fb4a539f91836231469c573f402
2019-07-01 13:38:37 -07:00
Matthew Mosesohn
2aa66eb12d Default to refreshing kubeadm etcd key (#4931)
Change-Id: Icc0176773b6d581c43647de433214079440d7321
2019-06-30 03:37:22 -07:00
okamototk
4c8b93e5b9 containerd support (#4664)
* Add limited containerd support

Containerd support for Ubuntu + Calico

* Added CRI-O support for ubuntu

* containerd support.

* Reset  containerd support.

* fix lint.

* implemented feedback

* Change task name cri xx instead of cri-o in reset task and timeout condition.

* set crictl to fixed version

* Use docker-ce's container.io package for containerd.

* Add check containerd is installable or not.

* Avoid stop docker when use containerd and optimize retry for reset.

* Add config.toml.

* Fixed containerd for kubelet.env.

* Merge PR #4629

* Remove unused ubuntu variable for containerd

* Polish code for containerd and cri-o

* Refactoring cri socket configuration.

* Configurable conmon.

* Remove unused crictl/runc download

* Now crictl and runc is downloaded by common crictl.yml.

* fixed yamllint error

* Fixed brokenfiles by conflict.

* Remove commented line in config.toml

* Remove readded v1.12.x version

* Fixed broken set_docker_image_facts

* Fix yamllint errors.

* Remove unused apt source

* Fix crictl could not be installed

* Add containerd config from skolekonov's PR #4601
2019-06-29 14:09:20 -07:00
Tony Fouchard
216631bf02 Repair kube_proxy_exclude_cidrs (#4909) 2019-06-28 00:39:37 -07:00
Erwan Miran
c7f3123e28 kubeadm_discovery_address should not contain proto (#4930) 2019-06-28 00:37:37 -07:00
Simon Lelievre
f599c2a691 add macvlan cni to kubespray (#4901)
* add macvlan cni to kubespray

* macvlan: lint yaml files and fix sample config file

* macvlan: add OWNERS file

* add macvlan to README

* macvlan : CI first shoot

* macvlan : CI add full masquerade

* delegate retrive pod cidr to master only

* macvlan: add config for CI

* macvlan: add netchecker deployment
2019-06-28 00:35:38 -07:00
Matthew Mosesohn
465dfd68bc Fix empty kube_override_hostname in apiserver_sans (#4916)
kubernetes/master role defines this value as an empty string
when using a cloud provider, not undefined. The check was updated
accordingly.

Change-Id: I58dc31ef4fd568a717a6753eb89ca687933018ae
2019-06-25 08:00:37 -07:00
Matthew Mosesohn
73f45fbe94
Revert "Filter undefined SANs for apiserver cert (#4913)" (#4914)
This reverts commit d270678bda.
2019-06-25 06:56:00 -07:00
Matthew Mosesohn
d270678bda Filter undefined SANs for apiserver cert (#4913)
Change-Id: I37442fb095fb4217f67f74744ad07c1d5d8229ea
2019-06-25 05:54:36 -07:00
Andreas Krüger
de028814e5 Upgrade to etcd version 3.3.10 per 1.14 release notes. (#4898)
* Upgrade to etcd version 3.3.10 per 1.14 release notes.

* Update etcd binary checksums
2019-06-24 01:27:55 -07:00
andreyshestakov
b5406b752d Add kube_override_hostname to kubeadm certs. (#4903) 2019-06-23 23:19:56 -07:00
Matthew Mosesohn
6025981ceb Allow skip kubeadm image prep but install kubeadm (#4904)
Change-Id: I744e9a192cd863a1ce22fbd16d217c5dfb16750c
2019-06-23 23:17:56 -07:00
Matthew Mosesohn
4348e78b24 Enable kubeadm etcd mode (#4818)
* Enable kubeadm etcd mode

Uses cert commands from kubeadm experimental control plane to
enable non-master nodes to obtain etcd certs.

Related story: PROD-29434

Change-Id: Idafa1d223e5c6ceadf819b6f9c06adf4c4f74178

* Add validation checks and exclude calico kdd mode

Change-Id: Ic234f5e71261d33191376e70d438f9f6d35f358c

* Move etcd mode test to ubuntu flannel HA job

Change-Id: I9af6fd80a1bbb1692ab10d6da095eb368f6bc732

* rename etcd_mode to etcd_kubeadm_enabled

Change-Id: Ib196d6c8a52f48cae370b026f7687ff9ca69c172
2019-06-20 11:12:51 -07:00
Tony Fouchard
f67a24499b Allow to specify feature_control in calico cni config (#4879)
* Allow to specify feature_control in calico cni config

* list length checking

* double check

* remove 2 conditions
2019-06-16 23:14:07 -07:00
Simon Lelievre
5c704552d8 multus | use last version (#4880) 2019-06-16 23:12:07 -07:00
Simon Lelievre
2849191e67 CNI plugins: use last version 0.8.1 (#4878)
* CNI plugins: bump version 0.8.1

* cni plugins : update checksums

* cni : update readme
2019-06-14 02:42:23 -07:00
刘旭
a3a7fe7c8e fix start CoreDNS when init secondary master (#4867) 2019-06-11 04:56:18 -07:00
Andreas Krüger
c8d95a1586 Remove dnsPolicy from PSP (#4864) 2019-06-10 23:34:16 -07:00
Neven Miculinic
27a99e0a3f Added configurable min memory assertions (#4307) 2019-06-10 23:22:15 -07:00
Andreas Krüger
3cc351dff9 Require min version of Kubernetes (#4860)
* Require minimum version of Kubernetes

* Remove checksums for kubernetes version 1.12

* Add kube_version to precheck output and add min required version to README

* Fix merge

* Fix defaults

* Fix typo in precheck
2019-06-10 23:18:15 -07:00
Johnny Halfmoon
23c9071c30 Added file and container image caching (#4828)
* File and container image downloads are now cached localy, so that repeated vagrant up/down runs do not trigger downloading of those files. This is especially useful on laptops with kubernetes runnig locally on vm's. The total size of the cache, after an ansible run, is currently around 800MB, so bandwidth (=time) savings can be quite significant.

* When download_run_once is false, the default is still not to cache, but setting download_force_cache will still enable caching.

* The local cache location can be set with download_cache_dir and defaults to /tmp/kubernetes_cache

* A local docker instance is no longer required to cache docker images; Images are cached to file. A local docker instance is still required, though, if you wish to download images on localhost.

* Fixed a FIXME, wher the argument was that delegate_to doesn't play nice with omit. That is a correct observation and the fix is to use default(inventory_host) instead of default(omit). See ansible/ansible#26009

* Removed "Register docker images info" task from download_container and set_docker_image_facts because it was faulty and unused.

* Removed redundant when:download.{container,enabled,run_once} conditions from {sync,download}_container.yml

* All features of commit d6fd0d2aca by Timoses <timosesu@gmail.com>, merged May 1st 2019, are included in this patch. Not all code was included verbatim, but each feature of that commit was checked to be working in this patch. One notable change: The actual downloading of the kubeadm images was moved to {download,sync)_container, to enable caching.

Note 1: I considered splitting this patch, but most changes that are not directly related to caching, are a pleasant by-product of implementing the caching code, so splitting would be impractical.

Note 2: I have my doubts about the usefulness of the upload, download and upgrade tags in the download role. Must they remain or can they be removed? If anybody knows, then please speak up.
2019-06-10 11:21:07 -07:00
rptaylor
5bec2edaf7 remove namespace from ClusterRole (#4856) 2019-06-10 11:15:12 -07:00
Matthew Mosesohn
f504d0ea99 Remove invalid field dnsPolicy from podSecurityPolicy (#4863)
Change-Id: I02864011bf5fda5dbd35c7513c73875769036f87
2019-06-10 07:11:10 -07:00
Matthew Mosesohn
3b7797b1a1 Ensure haproxy and nginx reload when config changes (#4862)
Change-Id: Ia9a41e7b1cfcb1e6acb2dbae6eecc541dce25a74
2019-06-10 05:59:08 -07:00
Sergey Nuzhdin
4d5c4a13cb Add missing checksums, update default k8s version to 1.14.3 (#4850)
This PR adds missing checksums for kubeadm and hyperkube and changes
default version to 1.14.3

Signed-off-by: Sergey Nuzhdin <ipaq.lw@gmail.com>
2019-06-09 11:49:05 -07:00
AlawnWong
69a8f91512 Update dns-autoscaler.yml.j2 (#4857)
Merge two tolerations.  because the latest tolerations will cover the first tolerations.
2019-06-09 11:39:04 -07:00
Frank Ritchie
ab6f0012cc Make local volume provisioner dir mode a variable (#4821)
* Make local volume provisioner dir mode a variable

I need to change this for Nagios monitoring. Others may
need to as well. Had to close previous commits, sorry for
the spam.

* Make local volume provisioner dir mode a variable

I need to change this for Nagios monitoring. Others may
need to as well. Had to close previous commits, sorry for
the spam.
2019-06-06 04:36:14 -07:00