Commit graph

528 commits

Author SHA1 Message Date
陈谭军
99dbc6d780 clean-up doc,spelling mistakes (#5206) 2019-09-26 04:25:08 -07:00
陈谭军
2fc02ed456 fix-typo (#5199) 2019-09-25 04:04:00 -07:00
Mario
1ce7831f6d Update main.yml (#5166) 2019-09-17 05:36:24 -07:00
andreyshestakov
cb4f797d32 Fix macro on local_volume_provisioner (#5168)
mydict.keys() should be converted to list,
otherwise it causes errors in loop iteration.

Remove extra space after class name, which broke configmap.

Also allow set reclaimPolicy property.
2019-09-13 00:50:33 -07:00
Matthew Mosesohn
27ec548b88 Add support for k8s v1.16.0-beta.2 (#5148)
Cleaned up deprecated APIs:
apps/v1beta1
apps/v1beta2
extensions/v1beta1 for ds,deploy,rs

Add workaround for deploying helm using incompatible
deployment manifest.
Change-Id: I78b36741348f47a999df3841ee63cf4e6f377830
2019-09-10 12:06:54 -07:00
刘旭
53bc80bb59 Ingress nginx (#5066)
* remove svc-default-backend

* update ingress-nginx clusterrole
2019-08-15 02:34:33 -07:00
Danilo Riecken P. de Morais
56fa46716e Add missing coredns tag. (#5054) 2019-08-09 02:29:27 -07:00
Mario
973afef96e Fix variable for rbd_provisioner_user_secret (#5042)
* Update main.yml

* fix dead link 404
2019-08-08 20:03:25 -07:00
Maxim Snezhkov
b710c72f04 Add ability to setup virtual ip for ingress-controller (#5044) 2019-08-06 19:24:50 -07:00
w33dw0r7d
92bfcf0467 Add CoreDNS endpoint_pod_names option (#5012) 2019-07-31 11:26:15 -07:00
Oilbeater
1be788f785 add Kube-OVN cni to kubespray (#5020) 2019-07-30 20:10:20 -07:00
rptaylor
8afbf339f7 fix broken link (#5023) 2019-07-30 19:18:22 -07:00
Matthew Mosesohn
70dc222719 Upgrade local volume provisioner to v2.3.2 (#4983) 2019-07-16 05:27:26 -07:00
刘旭
de9443a694 remove unused code (#4981) 2019-07-16 01:39:24 -07:00
Alex Barcelo
99c5f7e013 add k8s_external plugin to CoreDNS configuration (#4704) 2019-07-16 00:53:23 -07:00
Matthew Mosesohn
fd9bbcb157 Enable nodes to run calicoctl for calico kdd mode (#4956)
* Enable nodes to run calicoctl

per-node tasks require waiting for calico-node to be applied

Change-Id: Ibe1076b7334a2da0332f2dd766fde0c3f172d1f2

* cleanup tasks that should run on master

Change-Id: I43a837879ef41596f14657ecd7f813899b6865ae

* Switch run_once calico logic to just run on first master

Change-Id: I6893711e354f63c5e1eaf6ac2e23d9a6347a555d
2019-07-15 01:59:06 -07:00
Matthew Mosesohn
29307740dd Enable containerd to deploy vanilla containerd package (#4951)
* Enable containerd to deploy vanilla containerd package

Fixes kubeadm references to CRI socket for containerd
Fixes download role cache feature to work with containerd

Change-Id: I2ab8f0031107e2f0d1a85c39b4beb66f08509a01

* use containerd for flannel-addons job

Change-Id: Ied375c7d65e64a625ffbd995ff16f2374067dee6

* add containerd vars

Change-Id: Ib9a8a04e501c481a86235413cbec63f3672baf91

* fixup vars

Change-Id: Ibea64e4b18405a578b52a13da100384582aa24c2

* more fixes

* fix rh repo

Change-Id: I00575a77cfb7b81d6095db5d918a52023c8f13ba

* Adjust helm host install for containerd
2019-07-10 23:46:54 -07:00
jlacoline
20c7e31ea3 Add calico 3.7.3 support (#4953)
* Add calico 3.7.3 support

* add calico_datastore variable to policy controller role

* add missing clusterrole rules for calico policy controller

* disable calico kube controller when kdd mode is used for versions < 3.6
2019-07-09 12:42:28 -07:00
champtar
a67a50f9c0 nodelocaldns: allow to set health port, switch to 9254 by default (#4902)
8080 is a pretty common port, using nodelocaldns_ip:8080 still
prevents node processes or hostNetwork=true processes to bind to *:8080
so switch to 9254 by default (prometheus port is 9253)

Signed-off-by: Etienne Champetier <champetier.etienne@gmail.com>
2019-07-09 00:52:01 -07:00
okamototk
f2b8a3614d Use K8s 1.15 (#4905)
* Use K8s 1.15

* Use Kubernetes 1.15 and use kubeadm.k8s.io/v1beta2 for
  InitConfiguration.
* bump to v1.15.0

* Remove k8s 1.13 checksums.

* Update README kubernetes version 1.15.0.

* Update metrics server 0.3.3 for k8s 1.15

* Remove less than k8s 1.14 related code

* Use kubeadm with --upload-certs instead of --experimental-upload-certs due to depricate

* Update dnsautoscaler 1.6.0

* Skip certificateKey if it's not defined

* Add kubeadm-conftolplane.v2beta2 for k8s 1.15 or later

* Support kubeadm control plane for k8s 1.15

* Update sonobuoy version 0.15.0 for k8s 1.15
2019-07-02 01:51:08 -07:00
Simon Lelievre
2849191e67 CNI plugins: use last version 0.8.1 (#4878)
* CNI plugins: bump version 0.8.1

* cni plugins : update checksums

* cni : update readme
2019-06-14 02:42:23 -07:00
Andreas Krüger
c8d95a1586 Remove dnsPolicy from PSP (#4864) 2019-06-10 23:34:16 -07:00
rptaylor
5bec2edaf7 remove namespace from ClusterRole (#4856) 2019-06-10 11:15:12 -07:00
Matthew Mosesohn
f504d0ea99 Remove invalid field dnsPolicy from podSecurityPolicy (#4863)
Change-Id: I02864011bf5fda5dbd35c7513c73875769036f87
2019-06-10 07:11:10 -07:00
AlawnWong
69a8f91512 Update dns-autoscaler.yml.j2 (#4857)
Merge two tolerations.  because the latest tolerations will cover the first tolerations.
2019-06-09 11:39:04 -07:00
Frank Ritchie
ab6f0012cc Make local volume provisioner dir mode a variable (#4821)
* Make local volume provisioner dir mode a variable

I need to change this for Nagios monitoring. Others may
need to as well. Had to close previous commits, sorry for
the spam.

* Make local volume provisioner dir mode a variable

I need to change this for Nagios monitoring. Others may
need to as well. Had to close previous commits, sorry for
the spam.
2019-06-06 04:36:14 -07:00
mervynzhang
a8dfcbbfc7 Switch /root references to ansible_env.HOME (#4842)
* kube config dir for current/ansible become user

* remove extra /

* fix default value
2019-06-06 02:06:11 -07:00
Andreas Krüger
818aa7aeb1 Set dnsPolicy to ClusterFirstWithHostNet when hostNetwork is true (#4843) 2019-06-05 03:17:55 -07:00
Toni Pokki
54416cabfd prefer_udp for upstream dns servers (#4810) 2019-06-04 23:27:55 -07:00
Andreas Krüger
7423932510 Add ready plugin for CoreDNS (#4817) 2019-05-28 06:47:56 -07:00
André R. de Miranda
0ef3a7914c Added pod psp in Rancher Local Path Provisioner (#4385)
* Added pod psp in Rancher Local Path Provisioner

Added pod security policy (psp) in Rancher Local Path Provisioner.

Signed-off-by: André R. de Miranda <andre@miranda.work>

* Apply psp for Rancher Local Path Provisioner only when local_path_provisioner_namespace is not kube-system and also reorganized the templates
2019-05-22 00:16:08 -07:00
André R. de Miranda
4bc204925a Error in nginx when starting registry-proxy (#4785)
Error starting nginx because in requiredDropCapabilities is dropped all capabilities.

The nginx requires the following capabilities:
- CHOWN
- SETGID
- SETUID

Signed-off-by: André R. de Miranda <andre@miranda.work>
2019-05-20 11:27:15 -07:00
MarkusTeufelberger
73c2ff17dd Fix Ansible-lint error [E502] (#4743) 2019-05-16 00:27:43 -07:00
Aleksey Kasatkin
14749df6f3 Fix "netchecker-server" ClusterRole (#4730)
* Add sha256 hashes for calicoctl v3.6.1

Hashes are added to calicoctl_binary_checksums for both adm and arm platforms.

* Add rules for "network-checker.ext" resource to "netchecker-server" ClusterRole

So that it could access the resource after it is created.

Corresponding issues:
https://github.com/Mirantis/k8s-netchecker-server/issues/125
https://github.com/kubernetes-sigs/kubespray/issues/3281
2019-05-09 01:30:49 -07:00
Andreas Krüger
8a5eae94ea Minor cleanups of CoreDNS issues and CI job (#4719)
* Minor cleanups

* Add comment in docs that nodelocaldns cache is enabled by default
2019-05-07 13:20:36 -07:00
MarkusTeufelberger
e67f848abc ansible-lint: add spaces around variables [E206] (#4699) 2019-05-02 14:24:21 -07:00
Youngchul Bang
fbba259933 ingress-nginx: enable --report-node-internal-ip-address flag (#4114)
Close #4113
2019-04-29 01:44:22 -07:00
Matthew Mosesohn
741de6051c Fix nodeselectors for contiv and nginx-ingress (#4662)
* Fix nodeselectors for contiv and nginx-ingress

Change-Id: Ib3eb6bd87193c69a90ee944c9164a0b6792c79ba

* Set kube proxy mode to iptables for addons task

Change-Id: Iff71a71f672405c74b4708c71db15ddc4391a53a
2019-04-28 23:36:19 -07:00
MarkusTeufelberger
88d919337e ansible-lint: don't compare to empty string [E602] (#4665) 2019-04-28 23:00:20 -07:00
Andreas Krüger
3722acee85 Fix broken metrics-server deployment not starting (#4651)
* Fix metrics-server deployment

* Make metrics server work

* Fix sample inventory
2019-04-26 00:44:26 -07:00
grialeyur
82119ca923 Add support calico kubernetes datastore and typha. (#4498)
* Add support calico kubernetes datastore and typha.

* Add typha_enabled to kubespray-defaults.
2019-04-25 05:00:48 -07:00
Matthew Mosesohn
fc072300ea Purge legacy cleanup tasks from older than 1 year (#4450)
We don't need to support upgrades from 2 year old installs,
just from the last major version.

Also changed most retried tasks to 1s delay instead of longer.
2019-04-24 00:08:05 -07:00
MarkusTeufelberger
424e59805f ansible-lint: Fix commands that are also available as module (#4619) 2019-04-23 22:18:00 -07:00
Andreas Krüger
d588532c9b Update probe timeouts, delays etc. (#4612)
* Fix merge conflict

* Add check delay

* Add more liveness and readiness options to metrics-server
2019-04-23 14:46:02 -07:00
Matthew Mosesohn
d89ecb8308 disable metrics server and fix terraform (#4617)
* disable metrics server in centos7-flannel-addons job

Change-Id: I1d87923547584896f64dda9ea8feb5581ad48cbe

* Fix tf facility->facilities syntax

Change-Id: I434bfe53f47e8e4a546890e0b62d24bde6e6d6a7

* Update Terraform CI for facilities

* Fix undefined variable error
2019-04-23 12:06:03 -07:00
Maxime Guyot
50751bb610 Revert "Optimize kube resources creation (#4572)" (#4621)
This reverts commit f8fdc0cd93.
2019-04-23 20:37:23 +03:00
andreyshestakov
f8fdc0cd93 Optimize kube resources creation (#4572) 2019-04-22 23:34:10 -07:00
Andreas Krüger
656633f784 YAMLLint everything (#4576) 2019-04-18 23:59:54 -07:00
Victor Morales
c6586829de Ensure /etc/bash_completion.d/ folder exists (#4543)
The Stateless ClearLinux feature[1] requires the creation of folders
in /etc folder. This change ensure the existence of the
/etc/bash_completion.d/ folder for ClearLinux Distribution.

[1] https://clearlinux.org/features/stateless
2019-04-18 02:24:10 -07:00
Maxime Guyot
37eac010c8 ansible-lint: Don’t compare to literal True/False (#4499) 2019-04-17 08:42:03 -07:00
Maxime Guyot
ec3daedf9e Revert "Fix for unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels (#4320)" (#4553)
This reverts commit 586ad89d50.
2019-04-17 07:58:06 -07:00
Jugwan Eom
d83181a2be add RBD Provisioner Addon (#3667) (#3668)
Based on the CephFS Provisioner Addon, the following changes have been made:
 - Upstream v2.1.1-k8s1.11
 - Configurable Provisioner replicas
2019-04-16 23:14:02 -07:00
Maxime Guyot
46ba6a4154 ansible-lint: when lines should not include Jinja2 variables (#4496) 2019-04-11 03:06:10 -07:00
Andreas Krüger
15597aa493 Do not force TCP connections to upstreams. (#4492) 2019-04-10 12:40:09 -07:00
Neven Miculinic
a30ad1e5a5 Added generic CNI network plugin (#4322)
* Added generic CNI network plugin

* Added CNI network plugin documentation

* added necessary fix
2019-04-10 04:16:15 -07:00
Robert Neumann
586ad89d50 Fix for unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels (#4320)
* Fix the file path for all.yml and k8s-cluster.yml

* Fix --node-labels namespace error "unknown labels specified"

* Update templates and configs kubelet node-labels
2019-04-10 04:14:12 -07:00
Sidharth Anupkrishnan
6caa639243 Update CoreDNS label as specified in the kubernetes coredns repository (#3920) 2019-04-10 04:12:13 -07:00
Andreas Krüger
4d39c1856e Fix jinja filters (#4470) 2019-04-09 02:19:06 -07:00
Xinghong Fang
d711a0c83f [nodelocaldns] expand tolerations on the daemonset (#4451) 2019-04-08 02:24:26 -07:00
Victor Morales
7e4f4a96fc Replace iteritems() to items() in Jinja2 templates (#4437)
The iteritems() dictionary's method has been removed in Python3. Using
this method in Jinja2 templates limits the execution to Python2 which
will be deprecated in 2020[1]. This change replaces that method for
the items() method as it's suggested in the official website[2].

[1] https://pythonclock.org/
[2] https://docs.ansible.com/ansible/latest/user_guide/playbooks_python_version.html#dict-iteritems
2019-04-08 00:32:26 -07:00
Xavi
20b12751af add Cinder allowVolumeExpansion option (#4415) 2019-04-04 02:36:50 -07:00
Andreas Krüger
7a72e567d5 Update CoreDNS to 1.4.0 (#4422)
* Update CoreDNS to 1.4.0

* Update readme to reflect CoreDNS update
2019-04-04 00:40:50 -07:00
Andreas Krüger
3c050be0b0 Update nodelocaldns cache settings (#4423) 2019-04-04 00:38:51 -07:00
Andreas Krüger
41e684eb5a Update DNS Autoscaler to 1.4.0 (#4425)
* Update DNS Autoscaler

* Update downloads too

* Fix yamllint

* Fix yamllint
2019-04-04 00:36:51 -07:00
Matthew Mosesohn
5f12b7aedf Remove kubedns and dnsmasq. Move dns_late phase after apps (#4406)
Both kubedns and dnsmasq modes are long not maintained.
We should run dns_late steps at the end because sshd
makes DNS lookups during Ansible run and has 2s timeouts
for each failed lookup trying to connect to coredns before
it is ready.
2019-04-01 12:32:34 -07:00
Stefan Prietl
2fb27c8521 Use static files in KubeDNS templating task (#4379)
This commit adapts the "Lay Down KubeDNS Template" task to use the static
files moved by pull request [1]

[1] https://github.com/kubernetes-sigs/kubespray/pull/4341
2019-03-28 06:26:43 -07:00
Matthew Mosesohn
6d7f3c4405 Reduce jinja2 filters in coredns templates (#4390) 2019-03-26 11:09:17 -07:00
Matthew Mosesohn
b7fd462944 Fix support for ansible 2.7.9 (#4375) 2019-03-20 11:29:42 -07:00
Matthew Mosesohn
150a969cf4
Forcefully delete pods when necessary (#4328)
Pods on down/unresponsive nodes can't be deleted without
--force --grace-period=0.

Fixes #4314
2019-03-14 07:45:46 -07:00
Manuel Cintron
3c4cbf133e Adding ability to override dashboard replica count (#4344) 2019-03-13 13:58:25 -07:00
Matthew Mosesohn
fd2c47b56a Move most coredns templates to static files (#4341)
* Move most coredns templates to static files

This should speed up the task slightly

* yaml lint fixes
2019-03-12 21:17:31 -07:00
Matthew Mosesohn
4fe61968cf Set default value for local_path_provisioner_enabled in role (#4309) 2019-02-28 05:36:08 -08:00
Peter Metz
26ca58419f feat(external-provisioner): adds support for local-path-provisioner (#4232)
* feat(external-provisioner/local-path-provisioner): adds support for local path provisioner

Helpful for local development but also in production workloads (once the
permission model is worked out) where you have redundancy built into the
software uses the PVCs (e.g. database cluster with synchronous
replication)

* feat(local-path-provisioner): adds debug flag, image tag group var

* fix(local-path-provisioner): moves image repo/tag to download role

* test(gce_centos7-flannel): enables local-path-provisioner in test case

* fix(addons): add image repo/tag to commented default values

* fix(local-path-provisioner): typo in jinja template for local path provisioner

* style(local-path-provisioner): debug flag condition re-formatted

* fix(local-path-provisioner): adds missing default value for debug flag

* fix(local-path-provisioner): syntax fix for debug if condition end

* fix(local-path-provisioner): jinja template syntax: if condition white space
2019-02-25 22:45:30 -08:00
hikoz
3d25b4dfc1 30MiB for gpu-device-plugin (#4227)
* 30MiB for gpu-device-plugin

* use vars for easier configuration
2019-02-25 20:03:53 -08:00
Vasilis Remmas
81801ce23b Add master toleration flag in dashboard deployment (#4290) 2019-02-25 19:34:47 -08:00
Matthew Mosesohn
4638acfe81 Retry applying podsecurity policies (#4279) 2019-02-24 22:50:55 -08:00
Kaoet
aadef80404 Upgrade to latest version of ubuntu-nvidia-driver-installer. (#4296)
The lastest version of ubuntu-nvidia-driver-installer contains a fix for
https://github.com/GoogleCloudPlatform/container-engine-accelerators/issues/90
which causes the installer pod to crash when driver is already loaded.
2019-02-24 22:22:48 -08:00
Maxime Guyot
323d788f48 Add support for --enable-skip-login in Dashboard (#4265) 2019-02-19 23:24:29 -08:00
Kaoet
23685b4537 Add image tag in "pause" container of nvidia driver installer. (#4247) 2019-02-17 21:02:30 -08:00
Kaoet
192f4c4e96 Allow customizing container image path used in NVIDIA GPU addon. (#4229) 2019-02-14 03:51:38 -08:00
Maxime Guyot
6a33411d65 Add an option for helm init --wait (#4202) 2019-02-11 14:32:26 -08:00
Jeff Bornemann
c41c1e771f OCI Cloud Provider Update (#4186)
* OCI subnet AD 2 is not required for CCM >= 0.7.0

Reorganize OCI provider to generate configuration, rather than pull

Add pull secret option to OCI cloud provider

* Updated oci example to document new parameters
2019-02-11 12:08:53 -08:00
peerapach
69e5deeccc Fix newline issue of priorityClassName when enable tolerations (#4164) 2019-02-04 12:59:01 -08:00
Thomas Nys
68fd7e39da Set cluster DNS correctly in case of nodelocal dns cache (#3879)
* Set cluster DNS correctly in case of nodelocal dns cache

* Pass in cluster_ip based on dns mode

* Disable nodelocaldns by default

* Fix syntax error

* Fix syntax issue

* Add nodelocadns ip to vars of node installation

* Change location of nodelocaldns_ip

* Try to remove newlines from jinja template

* Add debug for config file

* Move parameter logic outside of template

* Adapt templates after feedback

* Remove debugging
2019-01-28 23:39:27 -08:00
wangxf
a096761306 [PR-Calico]Support calico 3.4.0 (#4102)
* Suport calico 3.4.0

Signed-off-by: wangxf1987 <xiaofeix.wang@gmail.com>

* Remove symlink + cni conflist template when 3.3.0+, handle Canal, addition of install-cni: sidecar(3.3.0) or initontainer(3.4.0), KUBECONFIG_FILEPATH, calico_cert_dir, advertise cluster ips

* scheduler.alpha.kubernetes.io/critical-pod deprecated since 1.12
2019-01-28 11:03:49 -08:00
Danny Kulchinsky
595d6427ac [Nodelocal DNS cache] Mount host /run/xtables.lock in nodelocaldns container (#4074)
* Mount host /run/xtables.lock in nodelocaldns container

* fix typo in nodelocaldns daemonset manifest yml

* Add prometheus scrape annotation, updateStrategy and reduce termination grace period

* fix indentation

* actually fix it..

* Bump k8s-dns-node-cache tag to 1.15.1 (fixes https://github.com/kubernetes/dns/issues/282)
2019-01-28 01:57:40 -08:00
Chad Swenson
cc30220f01
Merge pull request #4044 from chadswen/lvp-cm-fix
Fix local-volume-provisioner configmap template
2019-01-15 09:08:08 -06:00
Chad Swenson
4959bfc1b3
Merge pull request #3950 from elementyang/pr-registry
fix registry_storage_class equals empty string
2019-01-14 15:45:09 -06:00
Chad Swenson
3ee5aa0d6b Fix local-volume-provisioner configmap template
Looks like the template is removing the trailing space between storage
class entries, and since CI only has one storage class we never hit this
issue. This change will prevent the yaml from printing on a single line
when multiple storage classes are defined.
2019-01-14 14:28:00 -06:00
Chad Swenson
72802e4d8d Bugfixes for Local Volume Provisioner
- Fixed an issue where storage class host directories were looped
through excessive target hosts
- Fixes examples in the LVP `README.md` to use nested dicts instead of a
list of dicts
2019-01-08 17:45:20 -06:00
Wilmar den Ouden
4fb8adb9e4 More dynamic local-storage-provisioner approach (#3472)
* Makes local volume provisioner more dynamic

* Correct variable name in local storage provisioner defaults

* Updates external-provisioner readme

* Updates variable naming to be more clear, more documentation, fixes sample inventory

* Variable refactor, untangled some jinja2 loops

* Corrects variable name

* No variable substitution in dict keys, replaced with anchor

* Fixes default storage_classes dict, inline docs

* Fixes spelling in inline docs

* Addresses comments in review

* Updates all the defaults

* Fix failing CI task

* Fixes external provisioner daemonset
2019-01-08 12:36:44 -08:00
Chad Swenson
5c52a830d2 Update kubernetes dashboard to latest patch (#3995) 2019-01-08 09:46:20 -08:00
elementyang
e1e13b68b3 fix ingress nodeSelector label 2018-12-29 14:41:23 +08:00
elementyang
90ee5df413 fix registry_storage_class equals empty string 2018-12-29 14:31:47 +08:00
elementyang
532e97c542 fix registry_storage_class equals empty string 2018-12-28 14:23:19 +08:00
Ryler Hockenbury
3e8f4c1545 Use recommended defaults for dns autoscale (#3884) 2018-12-12 05:05:46 -08:00
Ryler Hockenbury
c04e8b57b9 Metrics server resizer addon needs to target metrics server deployment (#3867)
* Metrics server resizer addon should target metrics server deployment

* Target metrics server deployment without version
2018-12-12 00:09:09 -08:00
Thomas Nys
3e3ee0aeb1 Add support for running a nodelocal dns cache (#3861)
* Add support for running a nodelocal dns cache

After encountering dns issues in a cluster I was recently working on I
noticed Kubernetes 1.13 introduced support for running a nodelocal dns
cache.

I believe this can usefull for more people.

73b548db06
https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/0030-nodelocal-dns-cache.md

* Add requested changes

* Add additional requested changes + documentation

* Add requested changes after review

* Replace incorrect variable
2018-12-10 17:28:03 -08:00
Julien C
593a9a262d Add metrics service to kube-dns (#3852)
Metrics port is exposed through a service for CoreDNS but not for kube-dns.
2018-12-10 01:45:00 -08:00
Bjorn Skovlund Ryden
d42b37b77d Added RBAC rights for metrics_server. Fixes #3829 (#3843) 2018-12-07 03:11:35 -08:00