Commit graph

1230 commits

Author SHA1 Message Date
Sergey
740d8b0a26 enable kubelet client certificate rotation (#4081)
* enable kubelet client certificate rotation

* change to variable kubelet_rotate_certificates
2019-04-03 01:35:44 -07:00
Matthew Mosesohn
5f12b7aedf Remove kubedns and dnsmasq. Move dns_late phase after apps (#4406)
Both kubedns and dnsmasq modes are long not maintained.
We should run dns_late steps at the end because sshd
makes DNS lookups during Ansible run and has 2s timeouts
for each failed lookup trying to connect to coredns before
it is ready.
2019-04-01 12:32:34 -07:00
Dmitry Chepurovskiy
0440e45d65 Fix supplementary_addresses rendering error (#4403) 2019-03-29 00:26:13 -07:00
Dmitry Chepurovskiy
669ab10c17 Added livenessProbe for local nginx apiserver proxy liveness probe (#4222)
* Added configurable local apiserver proxy liveness probe

* Enable API LB healthcheck by default

* Fix template spacing and moved healthz location to nginx http section

* Fix healthcheck listen address to allow kubelet request healthcheck
2019-03-28 06:20:46 -07:00
Etienne
d0ae316934 Use proxy_env with kubeadm phase commands (#4325) 2019-03-26 03:03:19 -07:00
Matthew Mosesohn
b7fd462944 Fix support for ansible 2.7.9 (#4375) 2019-03-20 11:29:42 -07:00
Matthew Mosesohn
ec08303f82 Revert "Fix #4237: update kube cert path (#4354)" (#4369)
This reverts commit ea7a6f1cf1.

This change modified the certs dir for Kubernetes, but did not move the directories for existing clusters.
2019-03-20 05:56:57 -07:00
Dmitry Chepurovskiy
ea7a6f1cf1 Fix #4237: update kube cert path (#4354) 2019-03-17 23:55:11 -07:00
Matthew Mosesohn
150a969cf4
Forcefully delete pods when necessary (#4328)
Pods on down/unresponsive nodes can't be deleted without
--force --grace-period=0.

Fixes #4314
2019-03-14 07:45:46 -07:00
Matthew Mosesohn
acbf3db233 Remove hard dependence on facts for all nodes (#4304)
* Remove hard dependence on facts for all nodes

* Update main.yaml

* Update main.yaml
2019-03-05 03:04:39 -08:00
Matthew Mosesohn
adf6a7121f Reenable set_facts task for dns_late (#4312) 2019-03-01 05:39:30 -08:00
hikoz
67832aada9 changed_when:false (#4189) 2019-02-25 20:09:30 -08:00
Ryler Hockenbury
88249308a0 Add labels to vsphere cloud config (#4275) 2019-02-25 19:58:15 -08:00
Gabor Lekeny
b4aaa7b908 Speed up tasks (#4278)
* fact gathering should run only once per node
* eliminate ansible version check, it is at the beginning of each
  playbook
2019-02-25 19:56:23 -08:00
Matthew Mosesohn
b07641c3f3 Move kube_proxy_remove out of set_facts and set default (#4180) 2019-02-25 00:08:06 -08:00
Frank Ritchie
9805fb7a34 Add flexvolume plugin dir to kubeadm kubelet (#4168)
This was already approved in #4106 but there are CI issues
with that PR due to references to kubernetes incubator.

After upgrading to Kubespray 2.8.1 with Kubeadm enabled Rook
Ceph volume provision failed due to the flexvolume plugin dir not
being correct. Adding the var fixed the issue
2019-02-20 15:02:02 -08:00
Abdulaziz AlMalki
eafab9636f fix wrong indent of oidc-username-prefix and oidc-groups-prefix in kubeadm config template (#4263) 2019-02-19 23:22:32 -08:00
Seungkyu Ahn
107bfb259a This PS is to fix the bug when Workers can't join the cluster (#4276)
because of etc-kubernetes-manifests not empty.
2019-02-19 22:13:59 -08:00
Rong Zhang
d4a36aa55b
Merge pull request #4027 from riverzhang/kube-proxy
Add update server field in kube-proxy kubeconfig
2019-02-20 13:41:06 +08:00
Manuel Cintron
07b2894080 Adding ability to maintain existing Encryption Secrets at Rest. (#4255)
* Adding ability to maintain existing Encryption Secrets at Rest.

If secrets_encryption.yaml is present it will not be overriten with a new kube_encrypt_token.

This should allow for it to be set ahead of a playbook running or maintain it if cluster.yml is ran on the same cluster and the ansible host does not have access to the secrets.

* Setting existing kube_encrypt_token across all master nodes in case it was missing in one or more nodes.
2019-02-19 07:31:45 -08:00
hikoz
e03588f431 use swapon -s (#4216) 2019-02-14 02:35:17 -08:00
Sorin Sbarnea
22a5a00c49 Improve kubeadm join tasks (#4206)
Fix issue where `kubeadm join` could wait forever for joining.

Fix issue where `kubeadm join` were not reaching the user, making
impossible to find the cause of the failure.

New behaviour is to first attempt to join without bypassing the
verifications checks and to display them if needed.

If this fails it still attempts to join by ignoring the check in
order to make previous behavior.

A timeout of 60 seconds is allocated for a joining.

Related-bug: #3973
2019-02-12 13:42:56 -08:00
Sergey
fbce6349c4 check kube_pods_subnet and kube_service_addresses to valid ip network range, not single ip address (#4188) 2019-02-11 14:12:06 -08:00
Chad Swenson
038a2eb862
Merge pull request #3949 from trogeat/patch-fix-missing-ca-cert-apiserver
kubespray: fix missing ca-certificate path in apiserver
2019-02-11 15:40:04 -06:00
Chad Swenson
6878c2af4e Fix kube_hostname_override inconsistencies (#4185) 2019-02-06 22:20:11 -08:00
Earl C. Ruby III
52e0aa7a80 Install the latest filesystem creation packages (#3904)
This PR ensures that the e2fsprogs and xfsprogs packages are
installed on all Kubernetes nodes and that the packages are
the latest versions. It also ensures that the nodes can
create XFS filesystems when necessary, since not all distros
install xfsprogs by default.

e2fsprogs - ext2/ext3/ext4 file system utilities
xfsprogs - Utilities for managing the XFS filesystem
2019-02-04 12:23:33 -08:00
peerapach
bd9474bafd fix kubeadm-setup when enable access_ip (#4145) 2019-02-01 20:10:34 -08:00
Sorin Sbarnea
316b73178d Add timeout to Get current version of calico cluster version (#4149)
Avoid waiting forever for this task that should be very quick.

Fixes: #4148
2019-02-01 20:09:04 -08:00
Vasilis Remmas
cd7924f8c9 Add oidc prefixes to kubeadm templates (#4159) 2019-01-31 15:31:43 -08:00
Erwan Miran
7f93a5a0f5 Fix deprecation warnings (#4130)
* use not deprecated ansible_play_hosts variable

* Using tests as filters is deprecated

* Fix deprecation warning about pkg list
2019-01-31 14:57:22 -08:00
Thomas Nys
68fd7e39da Set cluster DNS correctly in case of nodelocal dns cache (#3879)
* Set cluster DNS correctly in case of nodelocal dns cache

* Pass in cluster_ip based on dns mode

* Disable nodelocaldns by default

* Fix syntax error

* Fix syntax issue

* Add nodelocadns ip to vars of node installation

* Change location of nodelocaldns_ip

* Try to remove newlines from jinja template

* Add debug for config file

* Move parameter logic outside of template

* Adapt templates after feedback

* Remove debugging
2019-01-28 23:39:27 -08:00
wangxf
a096761306 [PR-Calico]Support calico 3.4.0 (#4102)
* Suport calico 3.4.0

Signed-off-by: wangxf1987 <xiaofeix.wang@gmail.com>

* Remove symlink + cni conflist template when 3.3.0+, handle Canal, addition of install-cni: sidecar(3.3.0) or initontainer(3.4.0), KUBECONFIG_FILEPATH, calico_cert_dir, advertise cluster ips

* scheduler.alpha.kubernetes.io/critical-pod deprecated since 1.12
2019-01-28 11:03:49 -08:00
Florent Monbillard
2054a98cf7 Run kubeadm and hyperkube outside of local_release_dir (#4098)
Addressing the discussion started in #4064, this PR moves kubeadm and
hyperkube binaries to /usr/local/bin before running them on the master
nodes.

It is to address the case where local_release_dir points to /tmp
(kubespray default) and /tmp is mounted with noexec mode, preventing
any binaries to be run in that partition.

In role "node", we still move kubeadm to bin_dir only on the worker
nodes.
2019-01-28 02:00:49 -08:00
Sergey
ce8ba1f170 create artifacts_dir (#4079) 2019-01-28 01:59:15 -08:00
Danny Kulchinsky
96688269f8 Support both --address and --bind-address for scheduler and controller-manager (#4112) 2019-01-27 23:43:34 -08:00
rongzhang
3ed5f89cf5 Add update server field in kube-proxy kubeconfig
I know this is a bit hack.
If you use cloud LB, you can use kubeadm's controlPlaneEndpoint to configure kube-proxy's server field.
But for nginx-proxy, it didn't start when kubeadm init.
2019-01-28 14:45:43 +08:00
Matthew Mosesohn
77d31e679a
fixup external kube-apiserver port (#4075) 2019-01-21 14:43:27 +03:00
Florent Monbillard
decbcdc423 Use external LB IP for external api endpoint (#4060)
* Use external LB IP for external api endpoint

Use loadbalancer_apiserver.address instead of apiserver_loadbalancer_domain_name for kudadm init --apiserver-advertise-address argument

https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#options states apiserver-advertise-address needs to be a IPv4 or IPv6 address

* only use loadbalancer IP if it is defined
2019-01-21 12:27:42 +03:00
Chad Swenson
e3ffa21303
Merge pull request #4019 from chadswen/kubeadm-env
Fix PATH for kubeadm init
2019-01-18 11:27:57 -06:00
Matthew Mosesohn
eecaba6b84
Generate external admin.conf with kubeadm (#4056)
* Generate external admin.conf with kubeadm

* Fix apiserver sans
2019-01-16 16:30:50 +03:00
Thomas Rogeat
83e11f9ef7 kubespray: fix missing ca-certificate path in apiserver 2019-01-16 11:48:24 +01:00
Chad Swenson
0697ab4b4f
Merge pull request #4048 from chadswen/readonly-writable-fix
Fix kubeadm config extra volumes
2019-01-15 13:02:04 -06:00
Chad Swenson
13e3e867ac Fix kubeadm config extra volumes
I found a potential use case where `writable` could be null and therfore
not treated like a boolean, so this adds an extra default statement to
avoid negating a non-boolean as boolean which would lead to undefined. refs #4020
2019-01-15 12:35:22 -06:00
rongzhang
bab2e5ed0d Use --bind-address instead of --address
--address deprecated
2019-01-11 12:22:47 +08:00
Chad Swenson
1d9c0c7d17 Fix readOnly flag in kubeadm-config.v1beta1.yaml.j2
In v1beta1 of `ClusterConfiguration` the extraVolumes `writable` field was changed to `readOnly` and its boolean value must be negated.

Also, the json field for `useHyperKubeImage` was incorrectly capitalized.
2019-01-09 20:43:35 -06:00
Chad Swenson
aa1d5b8970 Fix PATH for kubeadm init
Right now we're consistently getting warnings about kubelet not found in
path during `kubeadm init`. We fixed this for `kubeadm join` in #3342, and this brings the change to init
as well.
2019-01-09 18:38:02 -06:00
Chad Swenson
72802e4d8d Bugfixes for Local Volume Provisioner
- Fixed an issue where storage class host directories were looped
through excessive target hosts
- Fixes examples in the LVP `README.md` to use nested dicts instead of a
list of dicts
2019-01-08 17:45:20 -06:00
Wilmar den Ouden
4fb8adb9e4 More dynamic local-storage-provisioner approach (#3472)
* Makes local volume provisioner more dynamic

* Correct variable name in local storage provisioner defaults

* Updates external-provisioner readme

* Updates variable naming to be more clear, more documentation, fixes sample inventory

* Variable refactor, untangled some jinja2 loops

* Corrects variable name

* No variable substitution in dict keys, replaced with anchor

* Fixes default storage_classes dict, inline docs

* Fixes spelling in inline docs

* Addresses comments in review

* Updates all the defaults

* Fix failing CI task

* Fixes external provisioner daemonset
2019-01-08 12:36:44 -08:00
Andreas Holmsten
4d5b41b8db Allow override of bind addr for controller-manager and scheduler (#3968)
* allows to override the bind addresses for controller-manager and scheduler

Useful for Prometheus metrics monitoring

* Add bind addr override support in kubeadm/v1beta1

Adds support for override of bind addresses for controller-manager
and scheduler in kubeadm/v1beta1

* Move location of bind address vars

* Remove double declaration of schedulerExtraArgs
2019-01-07 20:41:54 -08:00
okamototk
8216e821d3 Fix kubeadm v1beta1 configuration taint (#3928)
* Use master node taint same as kubeadm configuration v1alpha3 or before.
2019-01-03 03:42:23 -08:00
Andreas Holmsten
a34139e19e (Re)add line break for supplementary addr in SANs (#3952)
The change implemented in #3908 remove line breaks for supplementary
addresses in kubeadm SANs, causing errors in the config file and
failure to bring cluster up. This commit reimplement line breaks in
between supplementary addresses.
2019-01-03 00:12:00 -08:00
Chad Swenson
80379f6cab Fix kube-proxy configuration for kubeadm (#3958)
- Creates and defaults an ansible variable for every configuration option in the `kubeproxy.config.k8s.io/v1alpha1` type spec
  - Fixes vars that were orphaned by removing non-kubeadm
  - Fixes previously harcoded kubeadm values
- Introduces a `main` directory for role default files per component (requires ansible 2.6.0+)
  - Split out just `kube-proxy.yml` in this first effort
- Removes the kube-proxy server field patch task

We should continue to pull out other components from `main.yml` into their own defaults files as I did here for `defaults/main/kube-proxy.yml`. I hope for and will need others to join me in this refactoring across the project until each component config template has a matching role defaults file, with shared defaults in `kubespray-defaults` or `downloads`
2019-01-03 00:04:26 -08:00
Rong Zhang
5834e609a6 Add scale master features (#3946)
* Add scale master features

* Add certificate management with kubeadm

* Add kubeadm kubeconfig

* Fix ymalroles error

* fix upgrade cluster fialed

* force update cert and keys when you reconfigure cluster
2018-12-27 23:27:27 -08:00
Gautam Divgi
320f4d4d7f Added filters for integer conversion of kubelet_max_pods and kube_network_node_prefix (#3857) 2018-12-26 13:58:53 -08:00
Seongjin Cho
16715adfa0 Adds support for webhook token auth. (#3939)
Webhook token auth:
https://kubernetes.io/docs/reference/access-authn-authz/authentication/#webhook-token-authentication

Fixes #3063.
2018-12-26 01:52:53 -08:00
rongzhang
1bb1ba2274 Fix GPU node Scheduling 2018-12-25 21:37:10 +08:00
Zefool
6ebcaab2bb controlPlaneEndpoint set up through load balancer should be possible … (#3888)
* controlPlaneEndpoint set up through load balancer should be possible  even in single master setups

Enable load balancer for single-master setups
Fixes an issue where single-master setups are not reachable using the usual admin.conf from outside the cluster. 

controlPlaneEndpoint set up through load balancer should be possible  even in single master setups

* add fix to other api versions

* remove obsolete check completely

* remove check, pass 2

* removes checks in client configuration

* delete 'and'
2018-12-25 00:03:32 -08:00
Rong Zhang
cd42e649a7 Fix reconfigure and upgrade cluster (#3938) 2018-12-24 23:06:27 -08:00
rongzhang
dd4159fe65 Delete unused controlPlane for join node
it is used for join master or use --experimental-control-plane argments
2018-12-23 00:31:01 +08:00
Seongjin Cho
e7b835eb4c Fix duplicate storage-backend (#3906) 2018-12-20 01:01:39 -08:00
Rong Zhang
925a820b56 Fix skip upgrade first master (#3915) 2018-12-19 05:16:14 -08:00
Matthew Mosesohn
50b884a32d Fixup line breaks for kubeadm SANs (#3908) 2018-12-19 02:47:31 -08:00
Ganesh Maharaj Mahalingam
73aee004ac Enable ClearLinux as a distro in kubespray (#3855)
Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
2018-12-18 01:39:25 -08:00
ihard
30a9149b52 add vars for cilium init container (#3893)
* add vars for cilium init container

* make yamllint happy

* add var cilium_init in downloads
2018-12-18 00:34:19 -08:00
Egor
dc8a8011be Load nf_conntrack module if nf_conntrack_ipv4 failed (#3764) 2018-12-12 05:33:54 -08:00
Maxim Snezhkov
5e84dabb46 Fix assertion for alone etcd nodes (#3847) 2018-12-12 05:21:54 -08:00
Maxim Snezhkov
90a7941d56 Fix disabling swap on ubuntu systems (#3864) 2018-12-11 02:42:00 -08:00
Egor
7da9880ff7 Move node-cidr-mask-size to ControllerManagerextraArgs (#3845) 2018-12-07 04:23:17 -08:00
pasqualet
ea833a4cd7 Fix apiServerCertSANs in kubeadm config file (#3839) 2018-12-07 00:11:08 -08:00
Andreas Krüger
d5ce5874e8 Streamline path to certs dir (#3836)
* Streamline path to certs dir

* More fixes

* Set path to etcd certs in kubernetes defaults instead
2018-12-06 23:11:53 -08:00
Rong Zhang
225f765b56 Upgrade kubernetes to v1.13.0 (#3810)
* Upgrade kubernetes to v1.13.0

* Remove all precense of scheduler.alpha.kubernetes.io/critical-pod in templates

* Fix cert dir

* Use kubespray v2.8 as baseline for gitlab
2018-12-06 12:11:48 -08:00
Andreas Krüger
ddffdb63bf Remove non-kubeadm deployment (#3811)
* Remove non-kubeadm deployment

* More cleanup

* More cleanup

* More cleanup

* More cleanup

* Fix gitlab

* Try stop gce first before absent to make the delete process work

* More cleanup

* Fix bug with checking if kubeadm has already run

* Fix bug with checking if kubeadm has already run

* More fixes

* Fix test

* fix

* Fix gitlab checkout untill kubespray 2.8 is on quay

* Fixed

* Add upgrade path from non-kubeadm to kubeadm. Revert ssl path

* Readd secret checking

* Do gitlab checks from v2.7.0 test upgrade path to 2.8.0

* fix typo

* Fix CI jobs to kubeadm again. Fix broken hyperkube path

* Fix gitlab

* Fix rotate tokens

* More fixes

* More fixes

* Fix tokens
2018-12-06 02:33:38 -08:00
Andreas Krüger
432f8e9841 Fix basic auth tokens for kubeadm deployment. (#3801)
* Fix basic auth tokens for kubeadm deployment.

* Tokens should be a dependancy on master, not nodes
2018-12-03 10:44:29 -08:00
karbyshevds
b109f52dab Set configure-cloud-routes=false as default if no network plugin is used (#3788)
* Set configure-cloud-routes=false as default if no network plugin is used

As configure-cloud-routes default value is `true`, so it need to be set to `false` when not required to avoid error messages like:
"Couldn't reconcile node routes: error listing routes: unable to find route table for AWS cluster" 
on, for example, AWS installations that don't use cloud native routing.

* Update kube-controller-manager.manifest.j2

remove extra spaces
2018-12-03 05:04:03 -08:00
Chad Swenson
487cfa5e6c Add options for configuring control plane component extra volumes (#3779)
This takes care of a few arbitrary use cases that may require custom mounts
inside of apiserver, controller manager, or scheduler.
2018-11-28 23:16:55 -08:00
Chad Swenson
b59d5c35bc Fix kubeadm_controller_extra_args (#3778) 2018-11-27 19:30:43 -08:00
Michal Belica
8331f7b056 Add support for setting custom node taints (#3774)
Introduced variable node_taints which can be set in inventory for
specific hosts or in group_vars, which generates --register-with-taints
command line argument for kubelet.
2018-11-27 15:56:49 -08:00
Rong Zhang
ddc19f43ba Add cloud provider config to kubeadm deployments (#3766) 2018-11-27 05:03:03 -08:00
Egor
9a5438ce2f Fix kubeadm-config: add kube_network_node_prefix (#3761) 2018-11-27 00:12:16 -08:00
Aivars Sterns
b07e93e08b
Merge pull request #3754 from MiaoZhou/fix-aws-node-label-error
Fix AWS Node Labels Error
2018-11-27 09:09:54 +02:00
Andreas Krüger
bad886ca9b Update defaults to match k8s 1.12 suggestions (#3760)
* Update defaults to match k8s 1.12 suggestions

* Test if Netchecker works with node ip instead of localhost

* Update defaults to ipvs and coredns

* Update defaults for kube_apiserver_insecure_port

* Update main.yaml
2018-11-26 15:36:39 -08:00
Miao Zhou
885c6cff71 Fix AWS Node Labels Error
Now the `kubespray-aws-inventory.py` script always set a node_labels key
to ansible_host.

When AWS instance did not set property labels, it would be an empty
string.

The TASK `Write kubelet config file (kubeadm or non-kubeadm)` will
failed with a msg:

`AnsibleUndefinedVariable: 'unicode object' has no attribute 'items'`.
2018-11-23 17:37:41 +08:00
Rong Zhang
0cfcd39d55 Switch to kubeadm deployment mode (#3461)
* Switch to kubeadm deployment mode

Discuss:https://github.com/kubernetes-incubator/kubespray/issues/3301

* Add non-kubeadm upgrage to kubeadm cluster
2018-11-21 01:35:40 -08:00
Matthew Mosesohn
daa290100c Fix helper script to refer to admin.conf as relative path (#3738) 2018-11-19 18:28:51 -08:00
Matthew Mosesohn
ac00d23b80 Skip etcd upgrade steps in kubeadm because it is not used (#3737) 2018-11-19 06:29:58 -08:00
Danny Kulchinsky
9ae2eefb9a Add resource-container flag to kube-proxy manifest (#3519)
* Add resource-container flag to kube-proxy manifest

* add resourceContainer: "" to kubeadm kube-proxy configs
2018-11-19 00:39:29 -08:00
Andreas Krüger
6e01c1e377 Fix missing run_once (#3733) 2018-11-18 21:39:29 -08:00
Matthew Mosesohn
ff09141a14 Retry kubeadm proxy and secondary master init tasks (#3715)
Due to suboptimal external loadbalancer configs, the LoadBalancer
might point to a downed kube-apiserver that is not set up yet.
2018-11-15 10:03:23 -08:00
Arslanbekov Denis
d188876a91 Added feature-gates flags in kubelet.env (for kubeadm) (#3713) 2018-11-15 10:01:53 -08:00
Andreas Krüger
931c76e58f Add DNS entries to node certs (#3710) 2018-11-14 13:58:17 -08:00
Erwan Miran
3fafa583d1 hostnameOverride on a per-node basis (#3708) 2018-11-14 09:37:53 -08:00
Dann
98d766c68e Moves apiserver port to bindPort when using controlPlaneEndpoint (#3449) 2018-11-14 00:23:30 -08:00
Arnaud MAZIN
633bfa7ebc Bring static tokens and user back to 1.12 (#3593) 2018-11-13 10:25:59 -08:00
Andreas Krüger
afc3f7dce4 Create certificates for each node too (#3698) 2018-11-13 07:10:59 -08:00
Ryler Hockenbury
e8901a2422 Apply linux node selector to coreDNS deployment (#3688)
* Apply linux node selector to coreDNS deployment

* Remove comment before linux node selector on manifests

* mend
2018-11-13 04:54:15 -08:00
Miao Zhou
fefa1670a6 fix calico_version wrong get (#3694)
the ':' makes wrong return of calico_version after the calicoctl downloaded && before the cluster is up
2018-11-12 07:35:21 -08:00
Antoine Legrand
3dcb914607 Remove Vault (#3684)
* Remove Vault

* Remove reference to 'kargo' in the doc

* change check order
2018-11-10 08:51:24 -08:00
Bily Zhang
b2b421840c Fix some typos (#3690)
Signed-off-by: mooncake <xcoder@tenxcloud.com>
2018-11-10 15:53:58 +01:00
RuriRyan
c2710899ed Fixes network restart for Ubuntu Bionic Beaver (#3600)
As Ubuntu Bionic Beaver uses systemd-networkd the step fails
if it tries to restart networking, as it is nonexistent.
2018-11-09 08:13:57 -08:00
Erwan Miran
a6932b6b81 Install ipvsadm when kube_proxy_mode is ipvs (#3548) 2018-11-07 14:04:11 -08:00