Commit graph

371 commits

Author SHA1 Message Date
Cristian Calin
db470f8529
Update CSI snaphotter and make it independent (#7943)
* CSI: update CSI snapshot CRDs

* CSI: update snapshot controller tag version with kubernetes specific versions

* CSI: allow enabling csi_snapshot_controller independent of Cinder CSI

* CSI: Align csi-snapshot-controller with upstream and use a Deployment instead of a StatefulSet
2021-09-06 04:24:29 -07:00
Florian Ruynat
ed87386d7b Set default k8s version to 1.21.3 2021-07-20 01:29:31 -07:00
Atsushi Nukariya
417180246c
Fix: typos in docs and comments (#7805) 2021-07-16 18:58:50 -07:00
Cristian Calin
bd6d810d0a
nodelocaldns: allow binding metrics address to host IP (#7748) 2021-06-29 05:28:41 -07:00
Cristian Calin
a3e34f589a
Enable Graceful Node Shutdown for Kubernetes >= 1.21.0 (#7746)
* Enable Graceful Node Shutdown for Kubernetes >= 1.21.0

* Add sample graceful shutdown parameters
2021-06-27 23:53:25 -07:00
Cristian Calin
282a27a07c
gVisor: initial support for gVisor container runtime (#7661)
* Docker/Containerd: move downloads urls to containerd-common

* gVisor: initial support for gVisor container runtime
2021-06-21 05:18:51 -07:00
Sergey
d9d29af87f
update containerd to version 1.4.6 (#7674) 2021-06-03 10:55:38 -07:00
Florian Ruynat
54cda80018
Fix debian docker available version (#7668) 2021-06-01 20:58:39 -07:00
Cedric Hnyda
4674b03661
Add cinder_csi_ignore_volume_az (#7624)
Signed-off-by: Cedric Hnyda <cedric.hnyda@itera.io>
2021-06-01 07:10:27 -07:00
Florian Ruynat
7208169db3 Update kubernetes version to 1.21.1 2021-05-27 11:18:24 -07:00
Florian Ruynat
bdf74c6749 Set default version to 1.20.7 2021-05-14 09:48:06 -07:00
Cedric Hnyda
1dd38721b3
Add external_openstack_enable_ingress_hostname option for openstack (#7572)
Signed-off-by: Cedric Hnyda <cedric.hnyda@itera.io>
2021-05-04 00:33:11 -07:00
Samuel Liu
96e6a6ac3f
Add krew support (#7464)
* Add krew support

* Add reset for krew

* Update install krew(local)

* ansible lint

* yamllint

* fix krew default vars

* fix kubectl_localhost mode

* replace include

* fix e206
2021-05-03 07:16:03 -07:00
Cristian Calin
360aff4a57
Rename ansible groups to use _ instead of - (#7552)
* rename ansible groups to use _ instead of -

k8s-cluster -> k8s_cluster
k8s-node -> k8s_node
calico-rr -> calico_rr
no-floating -> no_floating

Note: kube-node,k8s-cluster groups in upgrade CI
      need clean-up after v2.16 is tagged

* ensure old groups are mapped to the new ones
2021-04-29 05:20:50 -07:00
muzi502
69806e0a46
Add nerdctl cli tool for containerd user (#7500)
* Add nerdctl cli tool for containerd user

* Add nerdctl enable option

* Add nerdctl enable option and update nerdctl version to 0.8.0
2021-04-25 23:47:01 -07:00
Florian Ruynat
6001edeecd
Cleanup hashes and 1.18 hooks (#7534) 2021-04-20 15:34:33 -07:00
Florian Ruynat
1c0836946f Update default Kubernetes version to 1.20.6 2021-04-15 22:26:22 -07:00
Florian Ruynat
cd9a03f86c
Update some docker defaults (#7499) 2021-04-14 15:13:07 -07:00
emiran-orange
d56ac216f4
Use kubeadm_feature_gates instead of kube_feature_gates to leverage kubeadm feature gates and not to interfere with k8s components feature gates (#7447) 2021-04-12 01:05:59 -07:00
Daniil Muidinov
2257181ca8
Set containerd version to 1.4.4 (#7398)
* Set containerd version to 1.4.3

* Set containerd version to 1.4.4

Co-authored-by: Barry Melbourne <9964974+bmelbourne@users.noreply.github.com>
2021-04-01 23:20:11 -07:00
Kaleb Elwert
6fa3565dac
Allow connecting to bastion via non-standard SSH port (#7396)
* Allow connecting to bastion via non-standard port

* Fix bastion connection when ansible_port is not provided
2021-03-26 00:48:43 -07:00
Kenichi Omichi
486b223e01
Replace kube-master with kube_control_plane (#7256)
This replaces kube-master with kube_control_plane because of [1]:

  The Kubernetes project is moving away from wording that is
  considered offensive. A new working group WG Naming was created
  to track this work, and the word "master" was declared as offensive.
  A proposal was formalized for replacing the word "master" with
  "control plane". This means it should be removed from source code,
  documentation, and user-facing configuration from Kubernetes and
  its sub-projects.

NOTE: The reason why this changes it to kube_control_plane not
      kube-control-plane is for valid group names on ansible.

[1]: https://github.com/kubernetes/enhancements/blob/master/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint/README.md#motivation
2021-03-23 17:26:05 -07:00
Florian Ruynat
6d9ed398e3 Set default k8s version to 1.20.5 2021-03-19 10:04:34 -07:00
Mikael Johansson
f103ac7640
Change default OCCM internal and public networks variables to empty lists (#7380)
Signed-off-by: Mikael Johansson <mik.json@gmail.com>
2021-03-18 16:52:36 -07:00
stress-t
15f1b19136
Fix: added string to bool conversion for use_localhost_as_kube api load balancer (#7324) 2021-03-01 11:53:36 -08:00
Sergey Mikhaltsov
a4cc416511
use external_openstack_lbaas_use_octavia for template openstack-cloud… (#7298)
* use external_openstack_lbaas_use_octavia for template openstack-cloud-config

* Delete external_openstack_lbaas_use_octavia from default values. Added description and default values of variables to docs

* markdown fix

* make this simple

* set external_openstack_lbaas_use_octavia in default values

* duplicated variable in doc
2021-02-25 11:25:25 -08:00
Florian Ruynat
6334e4bd84 Set Kubernetes default version to 1.20.4 2021-02-22 08:45:42 -08:00
Etienne Champetier
3749729d5a
Remove calico-upgrade leftovers (#7282)
This is dead code since 28073c76ac

Signed-off-by: Etienne Champetier <e.champetier@ateme.com>
2021-02-16 11:24:58 -08:00
Etienne Champetier
1c5391dda7
Ensure kubeadm doesn't use proxy (#7275)
* Move proxy_env to kubespray-defaults/defaults

There is no reasons to use set_facts here

Signed-off-by: Etienne Champetier <e.champetier@ateme.com>

* Ensure kubeadm doesn't use proxy

*_proxy variables might be present in the environment (/etc/environment, bash profile, ...)
When this is the case we end up with those proxy configuration in /etc/kubernetes/manifests/kube-*.yaml manifests

We cannot unset env variables, but kubeadm is nice enough to ignore empty vars
93d288e2a4/cmd/kubeadm/app/util/env.go (L27)

Signed-off-by: Etienne Champetier <e.champetier@ateme.com>
2021-02-16 08:44:58 -08:00
stress-t
796d3fb975
Improving PR 6473 (#7259) 2021-02-16 05:19:05 -08:00
Matt Calvert
4cc065e66d Changes to support Dual Stack networking 2021-02-05 00:04:52 -08:00
Florian Ruynat
7a033a1d55
Add hashes and update default K8S version to 1.20.2 (#7171) 2021-01-15 12:43:09 -08:00
Florian Ruynat
a923f4e7c0
Update kube_version_min_required and cleanup hashes for release (#7160) 2021-01-15 00:33:51 -08:00
Florian Ruynat
09fa99fdc6
Update hashes and set default version to 1.19.7 (#7150) 2021-01-13 14:57:02 -08:00
Florent Monbillard
9a05037352
SHAs for 1.19.6, 1.18.14, 1.18.13, 1.17.16 and 1.17.15 (#7063)
* SHAs for 1.19.6, 1.18.14, 1.18.13, 1.17.16 and 1.17.15

* Fix etcd version in README

* Bump kube_version to 1.19.6
2020-12-18 15:42:24 -08:00
Kenichi Omichi
0aa6d3d4bc
Replace non-ascii with ascii (#7044)
When opening the main.yaml, vi cannot show the string correctly
due to non-ascii string. This replaces it.
2020-12-16 23:43:09 -08:00
Anthony Rabbito
93445b4dbc
Update hashes and set default version to 1.19.5 (#7012)
* Update hashes and set default version to 1.19.5

Signed-off-by: anthr76 <hello@anthonyrabbito.com>

* Reorder hashes

1.19.5 hashes should be near 1.19.x

* Added back blank line
2020-12-16 01:42:20 -08:00
Christoph Stäbler
1a491fc10c
Update hashes and set default to 1.19.4 (#6903) 2020-12-03 06:34:59 -08:00
Emerson Ford
f377d9f057
Set etcd_.*_addresses to use etcd_[events_]access_address instead of access_ip (#6936) 2020-12-02 13:55:00 -08:00
Florian Ruynat
db4e942b0d
Remove hyperkube from codebase (#6965) 2020-12-02 13:50:59 -08:00
Etienne Champetier
68b96bdf1a
Helm v3 only (#6846)
* Fix etcd download dest

Signed-off-by: Etienne Champetier <champetier.etienne@gmail.com>

* Only support Helm v3, cleanup install

Signed-off-by: Etienne Champetier <champetier.etienne@gmail.com>
2020-12-02 00:20:50 -08:00
Victor Morales
4f7a760a94
Add crun support (#6864)
Signed-off-by: Victor Morales <v.morales@samsung.com>
2020-12-01 11:00:50 -08:00
Florian Ruynat
f6eed8091e
Remove contiv related files (#6964) 2020-11-30 06:48:50 -08:00
David Medinets
fefcb8c9f8
Allow the eventRecordQPS setting to be set. (#6880)
* Allow the eventRecordQPS setting to be set.

The eventRecordQPS parameter controls rate limiting for event recording. When zero, unlimited events can cause denial-of-service situations. For my situation, I don't need more than a setting of "5". This change allows me to configure the setting before creating the cluster.

* Allow the eventRecordQPS setting to be set.

The default settings (see types.go) is five. So, this change does not affect the cluster provisioning. However, it does allow for the setting to be changed.
2020-11-03 00:42:15 -08:00
Victor Morales
e03e3c4582
Add Kata Containers support to CRI-O runtime (#6830)
* Enable Kata Containers for CRI-O runtime

Kata Containers is an OCI runtime where containers are run inside
lightweight VMs. This runtime has been enabled for containerd runtime
thru the kata_containers_enabled variable. This change enables Kata
Containers to CRI-O container runtime.

Signed-off-by: Victor Morales <v.morales@samsung.com>

* Set appropiate conmon_cgroup when crio_cgroup_manager is 'cgroupfs'

* Set manage_ns_lifecycle=true when KataContainers is enabed

* Add preinstall check for katacontainers

Signed-off-by: Victor Morales <v.morales@samsung.com>

Co-authored-by: Pasquale Toscano <pasqualetoscano90@gmail.com>
2020-10-23 03:07:46 -07:00
Hans Feldt
04b19359cb
allow non existing etcd group (#6797)
When using kubeadm managed etcd, configuring an etcd group can now
be skipped.
2020-10-21 07:32:20 -07:00
Florian Ruynat
60b0fb3e88
Update hashes and set default version to 1.19.3 (#6841) 2020-10-21 00:58:20 -07:00
Hans Feldt
92b1166dd0
Disable dashboard by default (#6804)
Users should opt in for features and not opt out.
2020-10-11 08:06:47 -07:00
Hans Feldt
6141b98bf8
calico: default to using kdd datastore (#6693)
If already deployed, get current datastore from CNI config file
2020-09-23 08:38:09 -07:00
lukasz bielinski
5f034330c5
properly generate extravolumes in kubeadmconfig for centos (#6708) 2020-09-23 01:20:09 -07:00
Florent Monbillard
5bd937ece0
Remove pypi repo and pip extra flags (#6729) 2020-09-21 13:27:51 -07:00
Florian Ruynat
79226d0870
Add Kubernetes hashes 1.19.2/1.18.9/1.17.12 and set default (#6698) 2020-09-17 11:12:45 -07:00
Hans Feldt
6da385de9d
Use "kubeadm join" to join masters to control plane (#6661)
Remove configuration variable kubeadm_control_plane
2020-09-17 04:34:45 -07:00
Hans Feldt
0cc5e3ef03
Remove workaround with kube_proxy_remove (#6512)
* kube-proxy never gets deployed so need to remove it
2020-09-17 04:30:45 -07:00
Florian Ruynat
a556f8f2bf
Remove deprecated (and removed in 1.19) flag and function --basic-auth-file (#6655) 2020-09-11 00:30:14 -07:00
Florian Ruynat
ad15721677
Add Kubernetes 1.19.1 hashes and set default (#6654) 2020-09-10 10:43:46 -07:00
Mikael Johansson
040dda37ed
Add comment clarifying network allocation and sizes (#6607)
* Add comment from roles/kubespray-defaults/defaults/main.yaml clarifying network allocation and sizes

Signed-off-by: Mikael Johansson <mik.json@gmail.com>

* Rewrite of the comment and added new examples

Signed-off-by: Mikael Johansson <mik.json@gmail.com>
2020-09-10 03:49:44 -07:00
Maxime Guyot
961149b865
Update kube_version_min_required for 2.14 release (#6634) 2020-09-07 23:59:43 -07:00
Florian Ruynat
f1566cb8c2
Add protectKernelDefaults option (default true) to kubelet config file (#6611) 2020-09-03 07:41:41 -07:00
Lovro Seder
c1ba8e1b3a
Rotate kubelet server certificate. (#6453)
* Rotate kubelet server certificate.

* CI test kubelet server cert rotation

* Approve kubelet serving certificates in tests.
2020-09-03 07:25:41 -07:00
Barry Melbourne
058438a25d
Remove support for CoreOS Container Linux (#6576) 2020-08-28 02:28:53 -07:00
Florian Ruynat
142b9e1eff
Update k8s hashes and set default version to 1.18.8 (#6532) 2020-08-21 00:09:39 -07:00
Arthur Outhenin-Chalandre
35682b5228
Fix cilium strict kube proxy replacement in HA (#6473)
* Update the cilium svc proxy test to HA mode

Signed-off-by: Arthur Outhenin-Chalandre <arthur@cri.epita.fr>

* Fix cilium strict kube-proxy in HA

Signed-off-by: Arthur Outhenin-Chalandre <arthur@cri.epita.fr>

* Add a single global endpoint variable

Signed-off-by: Arthur Outhenin-Chalandre <arthur@cri.epita.fr>

* Add cilium docs about kube-proxy replacement

Signed-off-by: Arthur Outhenin-Chalandre <arthur@cri.epita.fr>

* Fix issues in docs

Signed-off-by: Arthur Outhenin-Chalandre <arthur@cri.epita.fr>
2020-08-06 00:14:55 -07:00
Cristian Chiru
94df580674
Moved docker_dns_options to defaults so it can be overridden (#6394)
* Moved docker_dns_options to defaults so it can be overridden

* Fixed yaml indentation and markdown

* Moved docker_dns_search_domains to defaults
2020-08-01 00:29:41 -07:00
Arthur Outhenin-Chalandre
3550e3c145
Adding kube-proxy-replacement support in cilium (#6334)
Signed-off-by: Arthur Outhenin-Chalandre <arthur@cri.epita.fr>
2020-07-30 02:46:31 -07:00
Florent Monbillard
924cc11af6
Upgrade to kubernetes 1.18.6 (#6405)
- Add 1.17.9 and 1.16.13 SHAs
2020-07-29 14:54:09 -07:00
chenguoquan1024
9c48f666ec
change /etc/ssl/etcd to etcd_config_dir param (#6408)
* change /etc/ssl/etcd to etcd_config_dir param

* add use etcd_events_data_dir param
2020-07-21 23:58:05 -07:00
Florian Ruynat
f515898cb5
Update hashes and set default version to 1.18.5 (#6335) 2020-06-30 02:00:05 -07:00
Kenichi Omichi
25bab0e976
Change MetalLB to one of addons (#6238)
This changes MetalLB contrib to one of addons for deploying MetalLB with
Kubernetes cluster deployment. By the default, Kubespray doesn't deploy
MetalLB addon.
2020-06-29 15:11:59 -07:00
Joel Seguillon
4c1e0b188d
Add .editorconfig file (#6307) 2020-06-29 12:39:59 -07:00
Alvaro
80d16e6c91
Support for Ambassador OSS as an Ingress (#6135)
Support for Ambassador OSS as an Ingress Controller when
settings `ingress_ambassador_enabled: true`.

Signed-off-by: Alvaro Saurin <alvaro.saurin@gmail.com>
2020-06-24 07:39:17 -07:00
Pasquale Toscano
8f5c4dcd2e
Add support for Kata Containers (#6256)
* Install Kata Containers as additional container runtime

* Create RuntimeClasses for Kata Containers

* Updated Vagrant to optionally run without Docker as container manager

* Updated Vagrant to optionally use Libvirt nested virtualization

* Add Kata Containers documentation

* Fix lint errors

* Add kata_containers_enabled to kubespray-defaults

* Fixed typo error

* Fixed typo error
2020-06-22 00:28:39 -07:00
Florian Ruynat
eeb77369cb
Update hashes and set default to 1.18.4 (#6285) 2020-06-18 06:30:19 -07:00
Florian Ruynat
45d8797dce
Fix download boolean for local_path_provisioner (#6177) 2020-05-28 06:56:02 -07:00
Florian Ruynat
b2a0b649fd
Add new Kubernetes version hashes and set default to 1.18.3 (#6173) 2020-05-28 01:02:03 -07:00
Paul Rey
b5aaaf864d
Add additional network configuration options to external Openstack CCM (#6083) (#6085)
* Add additional network configuration options to external Openstack CCM (#6083)

* Change the default version of external openstack cloud controller image to v1.18.1 since there was an issue in v1.18.0 where some IPs of the private network were ignored

* Change Network section in external-openstack-cloud-config.j2 to Networking

* Add networking customization information in the openstack documentation
2020-05-18 02:31:36 -07:00
Florian Ruynat
c06f482901
Update default kubernetes version to 1.18.2 (#6064) 2020-05-06 11:17:09 -07:00
Florian Ruynat
299e35ebe4
Cleanup unused/erroneous variables (#6003) 2020-04-24 01:54:07 -07:00
Pierre Lebrun
03c8d0113c
Add vSphere external cloud provider (#5959) 2020-04-20 08:47:39 -07:00
Sergey
6318bb9f96
Return the ability to start control plain from the hyperkube image (#5422) 2020-04-18 05:59:36 -07:00
Florian Ruynat
49bd208026
Update hashes (1.18.2/1.17.5/1.16.9) and set default to 1.17.5 (#5967) 2020-04-17 06:55:07 -07:00
Sergey
35f248dff0
assembly fallback_ips and no_proxy var only one time on localhost and… (#5957)
* assembly fallback_ips and no_proxy var only one time on localhost and populate result on all hosts

* add tag always, fix ansible lint errors

* workaround to mitogen issue dw/mitogen#663

* do not gather fact before install python on coreos like distros

* try to pass docker molecule test
2020-04-16 07:22:47 -07:00
Alexander Kross
910a821d0b
Fix chicken and egg problem with proxy_env not defined on the first … (#5896)
* Fix chicken and egg problem with proxy_env not defined on the first envinronment usage.

* Disable fact gathering for the first proxy_env evaluation.

* Move proxy_env var set up from the role defaults to the root playbooks as fact.
2020-04-08 00:53:43 -07:00
Ali Sanhaji
937adec515
Azure Disk CSI deployment (#5833)
* Azure Disk CSI deployment

* Mention Azure CSI support

* Fix: remove unnecessary file

* Typo in documentation

* Add newline to end of file
2020-04-01 00:53:27 -07:00
Ali Sanhaji
484df62c5a
GCP Persistent Disk CSI Driver deployment (#5857)
* GCP Persistent Disk CSI Driver deployment

* Fix MD lint

* Fix Yaml lint
2020-03-31 00:06:40 -07:00
Florian Ruynat
09308d6125
Upgrade to Kubernetes 1.174 (#5628)
* Upgrade to Kubernetes 1.17.4 - change defaults

* Update ci jobs to previous k8s release (will fix them afterward)
2020-03-27 07:40:23 -07:00
Ali Sanhaji
a8a05a21a4
AWS EBS CSI implementation (#5549)
* AWS EBS CSI implementation

* Fixing image repos

* Add OWNERS file

* Fix expressions

* Add csi-driver tag

* Add AWS EBS prefix to variables

* Add AWS EBS CSI Driver documentation
2020-03-25 13:10:25 -07:00
Xiaodu
63fa406c3c
Move host_architecture to kubespray-defaults (#5811)
The variable is defined in `kubernetes/preinstall` role and used in several roles. Since `kubernetes/preinstall` is not always included when `ansible-playbook` is run with tag selectors (see #5734 for reason), they will fail, or individual roles must copy the same fact definitions (as in #3846). Moving the definition to the always-included `kubespray-defaults` role will resolve the dependency problem.
2020-03-25 12:58:25 -07:00
eddebc
db693d46df
Fixed an issue where without a default GW, ansible_default_ipv4 would return an empty dict which passed as a valid fallback_ip dict item (#5394) 2020-03-20 14:06:09 -07:00
spaced
8ce5a9dd19
remove atomic support because reached end of live (#5783) 2020-03-17 14:31:27 -07:00
spaced
876d4de6be
Fedora CoreOS support (#5657)
* fedora coreos support
- bootstrap and new fact for

* fedora coreos support
- fix bootstrap condition

* fedora coreos support
- allow customize packages for fedora coreos bootstrap

* fedora coreos support
- prevent install ptyhon3 and epel via dnf for fedora coreos

* fedora coreos support
- handle all ostree like os in same way

* fedora coreos support
- handle all ostree like os in same way for crio

* fedora coreos support
- add fcos documentations
2020-03-17 03:12:21 -07:00
Michael Shnit
29128eb316
Add AWS ALB Ingress Controller (#5489)
* Add AWS ALB Ingress Controller Ansible role

* remove trailing spaces

* update owners

* ALB ingress: update rbac clusterrole and remove role

* Move alb-ingress role to roles/kubernetes-apps/ingress_controller folder
2020-03-16 02:58:35 -07:00
Florent Monbillard
0330442c63
Kubernetes 1.16.8 (#5770)
* Kubernetes 1.16.8

* Use 1.16.8 in sample inventory and kubespray-defaults
2020-03-13 13:41:47 -07:00
Sergey
221b429c24
move var preinstall_selinux_state: to roles/kubespray-defaults/defaults/main.yaml (#5715) 2020-03-10 07:45:35 -07:00
Kubernetes Prow Robot
66408a87ee
Refactor download role (#5697)
* download file

* download containers

* fix push image to nodes

* pull if none image on host

* fix

* improve docker image tag checks.
do not pull already cached images

* rebase fix merge conflict

* add support download_run_once when upgrade and scale cluster
add some test with download_run_once

* set default values to temp flag for every download cycle

* add save,load abilty for containerd and crio when download_run_once=true

* return redefine image save/load command to  set_docker_image_facts.yml

* move set command to set_container_facts

* ctr in containerd_bin_dir

* fix order of ctr image export arguments

* temporary disable download_run_once for containerd and crio
due https://github.com/containerd/containerd/issues/4075

* remove unused files

* fix strict yaml linter warning and errors

* refactor logical conditions to pull and cache container images

* remove comment due lint check

* document role

* remove image_load_on_localhost, because cached images are always loaded to docker on remote sites

* remove XXX from debug output
2020-03-05 07:31:39 -08:00
Hugo Blom
0c1a0ab966
implement max-volumes for cinder csi (#5666) 2020-03-02 03:30:43 -08:00
Hugo Blom
171d2ce59c
Implement topology support for Cinder CSI (#5667)
* make cinder csi topology aware

* change feature description do better reflect whats being done

* remove sameas true since it isn't required
2020-02-26 05:12:25 -08:00
Ali Sanhaji
646fd5f47b
External OpenStack Cloud Controller Manager implementation (#5491)
* External OpenStack Cloud Controller Manager implementation

* Adding controller image tag

* Minor fixes

* Restructuring the external cloud controller to work with KubeADM
2020-02-18 04:47:28 -08:00
Sylvain Chateau
0ca7aa126b
added "Flatcar", "Flatcar Container Linux by Kinvolk" for all coreOS role (#5607) 2020-02-18 00:15:29 -08:00
Florian Ruynat
f56171b513
Remove old features gates (#5608) 2020-02-14 02:24:17 -08:00