Commit graph

1256 commits

Author SHA1 Message Date
MarkusTeufelberger 424e59805f ansible-lint: Fix commands that are also available as module (#4619) 2019-04-23 22:18:00 -07:00
Matthew Mosesohn d6d7458d68 Fix control plane setup without a hardcoded key (#4610) 2019-04-23 14:37:59 -07:00
Matthew Mosesohn 09fe95bc60 Avoid creating k8s cert dir on non-k8s nodes (#4602) 2019-04-21 15:27:43 -07:00
Vedran Bartonicek 33ab615072 Wait longer for node to join the cluster (#4549) 2019-04-20 07:05:40 -07:00
Matthew Mosesohn 05dc2b3a09 Use K8s 1.14 and add kubeadm experimental control plane mode (#4514)
* Use K8s 1.14 and add kubeadm experimental control plane mode

This reverts commit d39c273d96.

* Cleanup kubeadm setup run on first master

* pin kubeadm_certificate_key in test

* Remove kubelet autolabel of kube-node, add symlink for pki dir

Change-Id: Id5e74dd667c60675dbfe4193b0bc9fb44380e1ca
2019-04-19 06:01:54 -07:00
Victor Morales c6586829de Ensure /etc/bash_completion.d/ folder exists (#4543)
The Stateless ClearLinux feature[1] requires the creation of folders
in /etc folder. This change ensure the existence of the
/etc/bash_completion.d/ folder for ClearLinux Distribution.

[1] https://clearlinux.org/features/stateless
2019-04-18 02:24:10 -07:00
Maxime Guyot b218e17f44 ansible-lint: E403 Package installs should not use latest (#4500) 2019-04-18 01:34:08 -07:00
Maxime Guyot 37eac010c8 ansible-lint: Don’t compare to literal True/False (#4499) 2019-04-17 08:42:03 -07:00
Maxime Guyot ec3daedf9e Revert "Fix for unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels (#4320)" (#4553)
This reverts commit 586ad89d50.
2019-04-17 07:58:06 -07:00
Matthew Mosesohn c5fb734098 Switch calicoctl from a container to a binary (#4524) 2019-04-15 04:24:04 -07:00
Matthew Mosesohn d39c273d96 Revert "Use K8s 1.14 and add kubeadm experimental control plane mode (#4317)" (#4510)
This reverts commit 316508626d.
2019-04-11 12:52:43 -07:00
Matthew Mosesohn 316508626d Use K8s 1.14 and add kubeadm experimental control plane mode (#4317)
* Use Kubernetes 1.14 and experimental control plane support

* bump to v1.14.0
2019-04-11 05:30:13 -07:00
Qasim Sarfraz 3af90f8772 disable cloud-routes for non-cloud plugin (#4443) 2019-04-10 23:50:09 -07:00
Sergey 3b9d13fda9 Return back bind API server node loadbalancer to 127.0.0.1 for security purposes. (#4489) 2019-04-10 12:20:08 -07:00
Andreas Krüger 5e0249ae7c Add HAProxy as internal loadbalancer (#4480) 2019-04-10 05:56:18 -07:00
Neven Miculinic a30ad1e5a5 Added generic CNI network plugin (#4322)
* Added generic CNI network plugin

* Added CNI network plugin documentation

* added necessary fix
2019-04-10 04:16:15 -07:00
Robert Neumann 586ad89d50 Fix for unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels (#4320)
* Fix the file path for all.yml and k8s-cluster.yml

* Fix --node-labels namespace error "unknown labels specified"

* Update templates and configs kubelet node-labels
2019-04-10 04:14:12 -07:00
André R. de Miranda 097806dfe8 Added tag kube-proxy (#4272)
Signed-off-by: André R. de Miranda <andre@miranda.work>
2019-04-09 05:25:06 -07:00
Abdulaziz AlMalki 7cdf1fd388 quote values for kube_oidc_groups_prefix and kube_oidc_username_prefix values to accept colon, e.g oidc: (#4305)
This will fix error: error converting YAML to JSON: yaml: line 36: mapping values are not allowed in this context

Signed-off-by: Abdulaziz AlMalki <almalki.a@gmail.com>
2019-04-09 05:23:06 -07:00
Maxime Guyot 913fed0089 kubeadmn init: add 'until' to make 'retries' effective (#4464)
an 'until' clause is required or 'retries' is ignored

(see note @ https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html#do-until-loops)
2019-04-09 00:21:04 -07:00
rptaylor f52584a715 robust handling of API server SANs (#4435)
* robust handling of API server SANs

* use apiserver_loadbalancer_domain_name if it is defined, according to PR 3977
2019-04-08 08:10:35 -07:00
Andreas Krüger d18ad63e49 Update nginx to 1.15. Update manifest and performance optimize (#4458) 2019-04-08 02:02:29 -07:00
Maxime Guyot 8ad74404c9 Remove bash-completion (#4431) 2019-04-05 01:23:22 -07:00
Maxime Guyot 1ce2f04f47 allow Suse OS family (#4430) 2019-04-04 03:02:51 -07:00
Xavi 20b12751af add Cinder allowVolumeExpansion option (#4415) 2019-04-04 02:36:50 -07:00
Sergey 55890e1b82 keep compatibility as it was before (#4268) 2019-04-03 01:39:42 -07:00
Sergey 740d8b0a26 enable kubelet client certificate rotation (#4081)
* enable kubelet client certificate rotation

* change to variable kubelet_rotate_certificates
2019-04-03 01:35:44 -07:00
Matthew Mosesohn 5f12b7aedf Remove kubedns and dnsmasq. Move dns_late phase after apps (#4406)
Both kubedns and dnsmasq modes are long not maintained.
We should run dns_late steps at the end because sshd
makes DNS lookups during Ansible run and has 2s timeouts
for each failed lookup trying to connect to coredns before
it is ready.
2019-04-01 12:32:34 -07:00
Dmitry Chepurovskiy 0440e45d65 Fix supplementary_addresses rendering error (#4403) 2019-03-29 00:26:13 -07:00
Dmitry Chepurovskiy 669ab10c17 Added livenessProbe for local nginx apiserver proxy liveness probe (#4222)
* Added configurable local apiserver proxy liveness probe

* Enable API LB healthcheck by default

* Fix template spacing and moved healthz location to nginx http section

* Fix healthcheck listen address to allow kubelet request healthcheck
2019-03-28 06:20:46 -07:00
Etienne d0ae316934 Use proxy_env with kubeadm phase commands (#4325) 2019-03-26 03:03:19 -07:00
Matthew Mosesohn b7fd462944 Fix support for ansible 2.7.9 (#4375) 2019-03-20 11:29:42 -07:00
Matthew Mosesohn ec08303f82 Revert "Fix #4237: update kube cert path (#4354)" (#4369)
This reverts commit ea7a6f1cf1.

This change modified the certs dir for Kubernetes, but did not move the directories for existing clusters.
2019-03-20 05:56:57 -07:00
Dmitry Chepurovskiy ea7a6f1cf1 Fix #4237: update kube cert path (#4354) 2019-03-17 23:55:11 -07:00
Matthew Mosesohn 150a969cf4
Forcefully delete pods when necessary (#4328)
Pods on down/unresponsive nodes can't be deleted without
--force --grace-period=0.

Fixes #4314
2019-03-14 07:45:46 -07:00
Matthew Mosesohn acbf3db233 Remove hard dependence on facts for all nodes (#4304)
* Remove hard dependence on facts for all nodes

* Update main.yaml

* Update main.yaml
2019-03-05 03:04:39 -08:00
Matthew Mosesohn adf6a7121f Reenable set_facts task for dns_late (#4312) 2019-03-01 05:39:30 -08:00
hikoz 67832aada9 changed_when:false (#4189) 2019-02-25 20:09:30 -08:00
Ryler Hockenbury 88249308a0 Add labels to vsphere cloud config (#4275) 2019-02-25 19:58:15 -08:00
Gabor Lekeny b4aaa7b908 Speed up tasks (#4278)
* fact gathering should run only once per node
* eliminate ansible version check, it is at the beginning of each
  playbook
2019-02-25 19:56:23 -08:00
Matthew Mosesohn b07641c3f3 Move kube_proxy_remove out of set_facts and set default (#4180) 2019-02-25 00:08:06 -08:00
Frank Ritchie 9805fb7a34 Add flexvolume plugin dir to kubeadm kubelet (#4168)
This was already approved in #4106 but there are CI issues
with that PR due to references to kubernetes incubator.

After upgrading to Kubespray 2.8.1 with Kubeadm enabled Rook
Ceph volume provision failed due to the flexvolume plugin dir not
being correct. Adding the var fixed the issue
2019-02-20 15:02:02 -08:00
Abdulaziz AlMalki eafab9636f fix wrong indent of oidc-username-prefix and oidc-groups-prefix in kubeadm config template (#4263) 2019-02-19 23:22:32 -08:00
Seungkyu Ahn 107bfb259a This PS is to fix the bug when Workers can't join the cluster (#4276)
because of etc-kubernetes-manifests not empty.
2019-02-19 22:13:59 -08:00
Rong Zhang d4a36aa55b
Merge pull request #4027 from riverzhang/kube-proxy
Add update server field in kube-proxy kubeconfig
2019-02-20 13:41:06 +08:00
Manuel Cintron 07b2894080 Adding ability to maintain existing Encryption Secrets at Rest. (#4255)
* Adding ability to maintain existing Encryption Secrets at Rest.

If secrets_encryption.yaml is present it will not be overriten with a new kube_encrypt_token.

This should allow for it to be set ahead of a playbook running or maintain it if cluster.yml is ran on the same cluster and the ansible host does not have access to the secrets.

* Setting existing kube_encrypt_token across all master nodes in case it was missing in one or more nodes.
2019-02-19 07:31:45 -08:00
hikoz e03588f431 use swapon -s (#4216) 2019-02-14 02:35:17 -08:00
Sorin Sbarnea 22a5a00c49 Improve kubeadm join tasks (#4206)
Fix issue where `kubeadm join` could wait forever for joining.

Fix issue where `kubeadm join` were not reaching the user, making
impossible to find the cause of the failure.

New behaviour is to first attempt to join without bypassing the
verifications checks and to display them if needed.

If this fails it still attempts to join by ignoring the check in
order to make previous behavior.

A timeout of 60 seconds is allocated for a joining.

Related-bug: #3973
2019-02-12 13:42:56 -08:00
Sergey fbce6349c4 check kube_pods_subnet and kube_service_addresses to valid ip network range, not single ip address (#4188) 2019-02-11 14:12:06 -08:00
Chad Swenson 038a2eb862
Merge pull request #3949 from trogeat/patch-fix-missing-ca-cert-apiserver
kubespray: fix missing ca-certificate path in apiserver
2019-02-11 15:40:04 -06:00
Chad Swenson 6878c2af4e Fix kube_hostname_override inconsistencies (#4185) 2019-02-06 22:20:11 -08:00
Earl C. Ruby III 52e0aa7a80 Install the latest filesystem creation packages (#3904)
This PR ensures that the e2fsprogs and xfsprogs packages are
installed on all Kubernetes nodes and that the packages are
the latest versions. It also ensures that the nodes can
create XFS filesystems when necessary, since not all distros
install xfsprogs by default.

e2fsprogs - ext2/ext3/ext4 file system utilities
xfsprogs - Utilities for managing the XFS filesystem
2019-02-04 12:23:33 -08:00
peerapach bd9474bafd fix kubeadm-setup when enable access_ip (#4145) 2019-02-01 20:10:34 -08:00
Sorin Sbarnea 316b73178d Add timeout to Get current version of calico cluster version (#4149)
Avoid waiting forever for this task that should be very quick.

Fixes: #4148
2019-02-01 20:09:04 -08:00
Vasilis Remmas cd7924f8c9 Add oidc prefixes to kubeadm templates (#4159) 2019-01-31 15:31:43 -08:00
Erwan Miran 7f93a5a0f5 Fix deprecation warnings (#4130)
* use not deprecated ansible_play_hosts variable

* Using tests as filters is deprecated

* Fix deprecation warning about pkg list
2019-01-31 14:57:22 -08:00
Thomas Nys 68fd7e39da Set cluster DNS correctly in case of nodelocal dns cache (#3879)
* Set cluster DNS correctly in case of nodelocal dns cache

* Pass in cluster_ip based on dns mode

* Disable nodelocaldns by default

* Fix syntax error

* Fix syntax issue

* Add nodelocadns ip to vars of node installation

* Change location of nodelocaldns_ip

* Try to remove newlines from jinja template

* Add debug for config file

* Move parameter logic outside of template

* Adapt templates after feedback

* Remove debugging
2019-01-28 23:39:27 -08:00
wangxf a096761306 [PR-Calico]Support calico 3.4.0 (#4102)
* Suport calico 3.4.0

Signed-off-by: wangxf1987 <xiaofeix.wang@gmail.com>

* Remove symlink + cni conflist template when 3.3.0+, handle Canal, addition of install-cni: sidecar(3.3.0) or initontainer(3.4.0), KUBECONFIG_FILEPATH, calico_cert_dir, advertise cluster ips

* scheduler.alpha.kubernetes.io/critical-pod deprecated since 1.12
2019-01-28 11:03:49 -08:00
Florent Monbillard 2054a98cf7 Run kubeadm and hyperkube outside of local_release_dir (#4098)
Addressing the discussion started in #4064, this PR moves kubeadm and
hyperkube binaries to /usr/local/bin before running them on the master
nodes.

It is to address the case where local_release_dir points to /tmp
(kubespray default) and /tmp is mounted with noexec mode, preventing
any binaries to be run in that partition.

In role "node", we still move kubeadm to bin_dir only on the worker
nodes.
2019-01-28 02:00:49 -08:00
Sergey ce8ba1f170 create artifacts_dir (#4079) 2019-01-28 01:59:15 -08:00
Danny Kulchinsky 96688269f8 Support both --address and --bind-address for scheduler and controller-manager (#4112) 2019-01-27 23:43:34 -08:00
rongzhang 3ed5f89cf5 Add update server field in kube-proxy kubeconfig
I know this is a bit hack.
If you use cloud LB, you can use kubeadm's controlPlaneEndpoint to configure kube-proxy's server field.
But for nginx-proxy, it didn't start when kubeadm init.
2019-01-28 14:45:43 +08:00
Matthew Mosesohn 77d31e679a
fixup external kube-apiserver port (#4075) 2019-01-21 14:43:27 +03:00
Florent Monbillard decbcdc423 Use external LB IP for external api endpoint (#4060)
* Use external LB IP for external api endpoint

Use loadbalancer_apiserver.address instead of apiserver_loadbalancer_domain_name for kudadm init --apiserver-advertise-address argument

https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#options states apiserver-advertise-address needs to be a IPv4 or IPv6 address

* only use loadbalancer IP if it is defined
2019-01-21 12:27:42 +03:00
Chad Swenson e3ffa21303
Merge pull request #4019 from chadswen/kubeadm-env
Fix PATH for kubeadm init
2019-01-18 11:27:57 -06:00
Matthew Mosesohn eecaba6b84
Generate external admin.conf with kubeadm (#4056)
* Generate external admin.conf with kubeadm

* Fix apiserver sans
2019-01-16 16:30:50 +03:00
Thomas Rogeat 83e11f9ef7 kubespray: fix missing ca-certificate path in apiserver 2019-01-16 11:48:24 +01:00
Chad Swenson 0697ab4b4f
Merge pull request #4048 from chadswen/readonly-writable-fix
Fix kubeadm config extra volumes
2019-01-15 13:02:04 -06:00
Chad Swenson 13e3e867ac Fix kubeadm config extra volumes
I found a potential use case where `writable` could be null and therfore
not treated like a boolean, so this adds an extra default statement to
avoid negating a non-boolean as boolean which would lead to undefined. refs #4020
2019-01-15 12:35:22 -06:00
rongzhang bab2e5ed0d Use --bind-address instead of --address
--address deprecated
2019-01-11 12:22:47 +08:00
Chad Swenson 1d9c0c7d17 Fix readOnly flag in kubeadm-config.v1beta1.yaml.j2
In v1beta1 of `ClusterConfiguration` the extraVolumes `writable` field was changed to `readOnly` and its boolean value must be negated.

Also, the json field for `useHyperKubeImage` was incorrectly capitalized.
2019-01-09 20:43:35 -06:00
Chad Swenson aa1d5b8970 Fix PATH for kubeadm init
Right now we're consistently getting warnings about kubelet not found in
path during `kubeadm init`. We fixed this for `kubeadm join` in #3342, and this brings the change to init
as well.
2019-01-09 18:38:02 -06:00
Chad Swenson 72802e4d8d Bugfixes for Local Volume Provisioner
- Fixed an issue where storage class host directories were looped
through excessive target hosts
- Fixes examples in the LVP `README.md` to use nested dicts instead of a
list of dicts
2019-01-08 17:45:20 -06:00
Wilmar den Ouden 4fb8adb9e4 More dynamic local-storage-provisioner approach (#3472)
* Makes local volume provisioner more dynamic

* Correct variable name in local storage provisioner defaults

* Updates external-provisioner readme

* Updates variable naming to be more clear, more documentation, fixes sample inventory

* Variable refactor, untangled some jinja2 loops

* Corrects variable name

* No variable substitution in dict keys, replaced with anchor

* Fixes default storage_classes dict, inline docs

* Fixes spelling in inline docs

* Addresses comments in review

* Updates all the defaults

* Fix failing CI task

* Fixes external provisioner daemonset
2019-01-08 12:36:44 -08:00
Andreas Holmsten 4d5b41b8db Allow override of bind addr for controller-manager and scheduler (#3968)
* allows to override the bind addresses for controller-manager and scheduler

Useful for Prometheus metrics monitoring

* Add bind addr override support in kubeadm/v1beta1

Adds support for override of bind addresses for controller-manager
and scheduler in kubeadm/v1beta1

* Move location of bind address vars

* Remove double declaration of schedulerExtraArgs
2019-01-07 20:41:54 -08:00
okamototk 8216e821d3 Fix kubeadm v1beta1 configuration taint (#3928)
* Use master node taint same as kubeadm configuration v1alpha3 or before.
2019-01-03 03:42:23 -08:00
Andreas Holmsten a34139e19e (Re)add line break for supplementary addr in SANs (#3952)
The change implemented in #3908 remove line breaks for supplementary
addresses in kubeadm SANs, causing errors in the config file and
failure to bring cluster up. This commit reimplement line breaks in
between supplementary addresses.
2019-01-03 00:12:00 -08:00
Chad Swenson 80379f6cab Fix kube-proxy configuration for kubeadm (#3958)
- Creates and defaults an ansible variable for every configuration option in the `kubeproxy.config.k8s.io/v1alpha1` type spec
  - Fixes vars that were orphaned by removing non-kubeadm
  - Fixes previously harcoded kubeadm values
- Introduces a `main` directory for role default files per component (requires ansible 2.6.0+)
  - Split out just `kube-proxy.yml` in this first effort
- Removes the kube-proxy server field patch task

We should continue to pull out other components from `main.yml` into their own defaults files as I did here for `defaults/main/kube-proxy.yml`. I hope for and will need others to join me in this refactoring across the project until each component config template has a matching role defaults file, with shared defaults in `kubespray-defaults` or `downloads`
2019-01-03 00:04:26 -08:00
Rong Zhang 5834e609a6 Add scale master features (#3946)
* Add scale master features

* Add certificate management with kubeadm

* Add kubeadm kubeconfig

* Fix ymalroles error

* fix upgrade cluster fialed

* force update cert and keys when you reconfigure cluster
2018-12-27 23:27:27 -08:00
Gautam Divgi 320f4d4d7f Added filters for integer conversion of kubelet_max_pods and kube_network_node_prefix (#3857) 2018-12-26 13:58:53 -08:00
Seongjin Cho 16715adfa0 Adds support for webhook token auth. (#3939)
Webhook token auth:
https://kubernetes.io/docs/reference/access-authn-authz/authentication/#webhook-token-authentication

Fixes #3063.
2018-12-26 01:52:53 -08:00
rongzhang 1bb1ba2274 Fix GPU node Scheduling 2018-12-25 21:37:10 +08:00
Zefool 6ebcaab2bb controlPlaneEndpoint set up through load balancer should be possible … (#3888)
* controlPlaneEndpoint set up through load balancer should be possible  even in single master setups

Enable load balancer for single-master setups
Fixes an issue where single-master setups are not reachable using the usual admin.conf from outside the cluster. 

controlPlaneEndpoint set up through load balancer should be possible  even in single master setups

* add fix to other api versions

* remove obsolete check completely

* remove check, pass 2

* removes checks in client configuration

* delete 'and'
2018-12-25 00:03:32 -08:00
Rong Zhang cd42e649a7 Fix reconfigure and upgrade cluster (#3938) 2018-12-24 23:06:27 -08:00
rongzhang dd4159fe65 Delete unused controlPlane for join node
it is used for join master or use --experimental-control-plane argments
2018-12-23 00:31:01 +08:00
Seongjin Cho e7b835eb4c Fix duplicate storage-backend (#3906) 2018-12-20 01:01:39 -08:00
Rong Zhang 925a820b56 Fix skip upgrade first master (#3915) 2018-12-19 05:16:14 -08:00
Matthew Mosesohn 50b884a32d Fixup line breaks for kubeadm SANs (#3908) 2018-12-19 02:47:31 -08:00
Ganesh Maharaj Mahalingam 73aee004ac Enable ClearLinux as a distro in kubespray (#3855)
Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
2018-12-18 01:39:25 -08:00
ihard 30a9149b52 add vars for cilium init container (#3893)
* add vars for cilium init container

* make yamllint happy

* add var cilium_init in downloads
2018-12-18 00:34:19 -08:00
Egor dc8a8011be Load nf_conntrack module if nf_conntrack_ipv4 failed (#3764) 2018-12-12 05:33:54 -08:00
Maxim Snezhkov 5e84dabb46 Fix assertion for alone etcd nodes (#3847) 2018-12-12 05:21:54 -08:00
Maxim Snezhkov 90a7941d56 Fix disabling swap on ubuntu systems (#3864) 2018-12-11 02:42:00 -08:00
Egor 7da9880ff7 Move node-cidr-mask-size to ControllerManagerextraArgs (#3845) 2018-12-07 04:23:17 -08:00
pasqualet ea833a4cd7 Fix apiServerCertSANs in kubeadm config file (#3839) 2018-12-07 00:11:08 -08:00
Andreas Krüger d5ce5874e8 Streamline path to certs dir (#3836)
* Streamline path to certs dir

* More fixes

* Set path to etcd certs in kubernetes defaults instead
2018-12-06 23:11:53 -08:00
Rong Zhang 225f765b56 Upgrade kubernetes to v1.13.0 (#3810)
* Upgrade kubernetes to v1.13.0

* Remove all precense of scheduler.alpha.kubernetes.io/critical-pod in templates

* Fix cert dir

* Use kubespray v2.8 as baseline for gitlab
2018-12-06 12:11:48 -08:00
Andreas Krüger ddffdb63bf Remove non-kubeadm deployment (#3811)
* Remove non-kubeadm deployment

* More cleanup

* More cleanup

* More cleanup

* More cleanup

* Fix gitlab

* Try stop gce first before absent to make the delete process work

* More cleanup

* Fix bug with checking if kubeadm has already run

* Fix bug with checking if kubeadm has already run

* More fixes

* Fix test

* fix

* Fix gitlab checkout untill kubespray 2.8 is on quay

* Fixed

* Add upgrade path from non-kubeadm to kubeadm. Revert ssl path

* Readd secret checking

* Do gitlab checks from v2.7.0 test upgrade path to 2.8.0

* fix typo

* Fix CI jobs to kubeadm again. Fix broken hyperkube path

* Fix gitlab

* Fix rotate tokens

* More fixes

* More fixes

* Fix tokens
2018-12-06 02:33:38 -08:00
Andreas Krüger 432f8e9841 Fix basic auth tokens for kubeadm deployment. (#3801)
* Fix basic auth tokens for kubeadm deployment.

* Tokens should be a dependancy on master, not nodes
2018-12-03 10:44:29 -08:00
karbyshevds b109f52dab Set configure-cloud-routes=false as default if no network plugin is used (#3788)
* Set configure-cloud-routes=false as default if no network plugin is used

As configure-cloud-routes default value is `true`, so it need to be set to `false` when not required to avoid error messages like:
"Couldn't reconcile node routes: error listing routes: unable to find route table for AWS cluster" 
on, for example, AWS installations that don't use cloud native routing.

* Update kube-controller-manager.manifest.j2

remove extra spaces
2018-12-03 05:04:03 -08:00