Commit graph

3119 commits

Author SHA1 Message Date
Hoat Le
4c803d579b
@ #5008 | Local path provisioner boolean annotation is rendered incorrectly and not applied (#5669) 2020-02-22 07:08:47 -08:00
Javeria Khan
6368c626c5
Ignore assertion comparison for kube_network_node_prefix when using calico (#5632)
* Fix incorrect assertion comparison for kube_network_node_prefix

* Ignore assertion comparison for kube_network_node_prefix when using calico

* Adding more var docs description for kube_network_node_prefix

* Fixing trailing whitespaces
2020-02-20 00:39:03 -08:00
Erwan Miran
a5445d9c5c
Add stable repo on all masters with helm 3.x.x (#5659) 2020-02-19 14:05:46 -08:00
Adrien Gooris
da86457cda
remove unused var 'kube_apiserver_admission_control' (#5648) (#5651) 2020-02-19 05:08:25 -08:00
Chad Swenson
a15a0b5eb9
Make calico iptables lock timeout configurable (#5658)
Adds `calico_iptables_lock_timeout_secs` variable to calico DS yaml.
2020-02-19 02:28:25 -08:00
Ali Sanhaji
646fd5f47b
External OpenStack Cloud Controller Manager implementation (#5491)
* External OpenStack Cloud Controller Manager implementation

* Adding controller image tag

* Minor fixes

* Restructuring the external cloud controller to work with KubeADM
2020-02-18 04:47:28 -08:00
Sergey
12bc634ec3
helm default version 3.1.0 (#5634)
* helm default version 3.1.0

* fix newline
try to retest2
2020-02-18 03:21:29 -08:00
Sylvain Chateau
0ca7aa126b
added "Flatcar", "Flatcar Container Linux by Kinvolk" for all coreOS role (#5607) 2020-02-18 00:15:29 -08:00
Manuel Cintron
b51b52ac0e
Fixing and issue where if the pids in the orphan list no longer exist then all systemd child processes would be killed. (#5636) 2020-02-17 09:33:29 -08:00
Sergey
36c1f32ef9
remove legacy docker repo in kubernetes/preinstall before any packages installed (#5640) 2020-02-17 08:59:28 -08:00
Steven Reitsma
fa245ffdd5
Fix some minor issues with the Cinder CSI plugin (#5561)
Add Cinder images to download role
2020-02-17 03:47:28 -08:00
Erwan Miran
f7c5f45833
Ability to define plugins.cri.containerd params (#5624)
* Ability to define plugins.cri.containerd params

* addition of containerd field commented as an example

* documentation of containerd_config
2020-02-17 02:15:29 -08:00
lcooper40
579976260f
Added in code to allow control over pull policy for local path provis… (#5334)
* Added in code to allow control over pull policy for local path provisioner

* change to imagePullPolicy to use globally used variable k8s_image_pull_policy

* removed unusued variable from defaults

* updated contiv-etcd and cinder-csi-controllerplugin to use k8s_image_pull_policy variable
2020-02-17 02:13:30 -08:00
Ali Sanhaji
d56e9f6b80
Fix Cinder CSI bugs (#5492) 2020-02-17 01:49:28 -08:00
Erwan Miran
26700e7882
kubelet_config_extra_args and kubelet_node_config_extra_args (#5623)
* Introduce kubelet_config_extra_args and kubelet_node_config_extra_args to pass params to kubelet via YAML config

* kubelet_config_extra_args is not the alternative
2020-02-14 16:05:28 -08:00
Florian Ruynat
d86229dc2b
Upgrade cri-tools (crictl) to 1.17.0 (#5629) 2020-02-14 02:50:17 -08:00
Florian Ruynat
f56171b513
Remove old features gates (#5608) 2020-02-14 02:24:17 -08:00
Bort Verwilst
287421e21e
Set helm 3.0 as default (#5503)
* set helm 3.0 as default

* remove trainling space in vars.yml

* switched to helm 3.0.3
2020-02-13 02:18:35 -08:00
Erwan Miran
339e36fbe6
Files to archive can be passed directly (#5571) 2020-02-12 07:50:51 -08:00
Arthur Outhenin-Chalandre
5e648b96e8
Fix default value of kube_api_server_endpoint (#5529)
Signed-off-by: Arthur Outhenin-Chalandre <arthur@cri.epita.fr>
2020-02-11 01:40:01 -08:00
qvicksilver
ac2135e450
Fix recover-control-plane to work with etcd 3.3.x and add CI (#5500)
* Fix recover-control-plane to work with etcd 3.3.x and add CI

* Set default values for testcase

* Add actual test jobs

* Attempt to satisty gitlab ci linter

* Fix ansible targets

* Set etcd_member_name as stated in the docs...

* Recovering from 0 masters is not supported yet

* Add other master to broken_kube-master group as well

* Increase number of retries to see if etcd needs more time to heal

* Make number of retries for ETCD loops configurable, increase it for recovery CI and document it
2020-02-11 01:38:01 -08:00
Sergey
14b1cab5d2
force rotate control plane certifcate on master node when upgrade cluster (#5596) 2020-02-10 06:09:54 -08:00
Florian Ruynat
e570e2e736
Remove last rkt references (#5606) 2020-02-07 02:19:43 -08:00
Preslav Draganov
422b25ab1f
Bind Docker service to containerd.service on versions >=18.09.1 (#5477) 2020-02-07 02:15:44 -08:00
aca
9d32e2c3b0
fix duplicates when scheduler_extra_volumes defined (#5566) 2020-02-07 02:09:44 -08:00
Florian Ruynat
099341582a
Update nginx image to latest (#5590) 2020-02-07 02:07:44 -08:00
andreyshestakov
2ab5cc73cd
Fix typo in Multus plugin. (#5568) 2020-01-29 01:28:13 -08:00
Etienne Champetier
9f2dd09628
Add proxy support to containerd, improves no_proxy (#5583)
* containerd: add proxy support

Signed-off-by: Etienne Champetier <champetier.etienne@gmail.com>

* kubespray-defaults: add kube_service_addresses / kube_pods_subnet to no_proxy

CIDR notation in no_proxy is supported by a lot of programs/languages,
including go: https://github.com/golang/go/issues/16704
Without that containerd cannot talk the the API server (kube_apiserver_ip),
but it should not go through an external proxy for the nodes/pods/services

Signed-off-by: Etienne Champetier <champetier.etienne@gmail.com>
2020-01-29 01:24:14 -08:00
Sergey
2798adc837 Remove stale legacy yum docker repo /etc/yum.repos.d/docker.repo (#5569)
* Remove stale legacy yum docker repo /etc/yum.repos.d/docker.repo

* move task 'Remove legacy docker repo file' to pre-upgrade.yml
2020-01-28 02:31:40 -08:00
Florian Ruynat
54d9404c0e Fix hashes... kubernetes 1.17.2 (#5581) 2020-01-24 06:44:31 -08:00
Florian Ruynat
f1025dce4e Update to hashes and default version (1.15.8 / 1.16.5 / 1.17.1) (#5564) 2020-01-23 03:54:49 -08:00
Matthew Mosesohn
38688a4486 Remove dockerproject org (#5548)
* Change dockerproject.org to download.docker.com

dockerproject.org was deprecated in 2017 and has gone down.

* Restore yum repo for containerd

Change-Id: I883bb512a2164a85865b1bd4fb569af0358c8c2b

Co-authored-by: Craig Rodrigues <rodrigc@crodrigues.org>
2020-01-17 00:38:55 -08:00
Florian Ruynat
d640a57f9b update api-version for PriorityClass following removal in 1.17 (#5450) 2020-01-16 01:52:22 -08:00
Etienne Champetier
5e9479cded Ensure we always fixup kube-proxy kubeconfig (#5524)
When running with serial != 100%, like upgrade_cluster.yml, we need to apply this fixup each time
Problem was introduced in 05dc2b3a09

Signed-off-by: Etienne Champetier <champetier.etienne@gmail.com>
2020-01-14 02:45:09 -08:00
Matthew Mosesohn
06ffe44f1f Remove downloading deprecated calico-rr image (#5528)
Change-Id: I7354d33c7db513e0ee27c9a4cc40e8501c0e1061
2020-01-14 02:41:08 -08:00
Matthew Mosesohn
b35b816287 Raise typha max connections to 300 (#5527)
Raises limit from 100 to 300 because the default is far too low
and the pod can handle 300 with the given resources.

Change-Id: Ib1eec10da3d09d198933fcfe87291587e58d7cdb
2020-01-10 00:24:33 -08:00
Florian Ruynat
bf15d06568 Update to Kubernetes 1.15.7 (#5518) 2020-01-08 17:35:40 -08:00
Etienne Champetier
2c2ffa846c Calico: update to 3.11.1, allow to configure calico_iptables_backend (#5514)
I've tested this update by deploying a containerd / etcd cluster on top CentOS7,
MetalLB + NGINX Ingress. Upgrade using upgrade-cluster.yml

Signed-off-by: Etienne Champetier <champetier.etienne@gmail.com>
2020-01-08 02:27:40 -08:00
Damon Wang
48c41bcbe7 kube-proxy need conntrack (#5478) 2020-01-06 02:31:35 -08:00
zhanwang
beb47e1c63 update ingress_nginx install guide (#5502) 2020-01-06 02:27:35 -08:00
Erwan Miran
303c3654a1 Set pipefail in case tar fails (#5506) 2020-01-06 02:25:34 -08:00
Matthew Mosesohn
5fab610fab Clean kubectl cache after upgrade on first master (#5479)
Resolves issue where kubectl cache of <v1.16 api schema
interferes with interacting with daemonsets and deployments.

Change-Id: I63b7046958f2008eb144b6da0004c598f945e0ae
2020-01-06 02:23:35 -08:00
Fredrik Lönnegren
ccbcad9741 Ubuntu CRI-O (#5426)
* Fix crictl

* Reload systemd daemon before enabling service

* Typo

* Add crictl template

* Remove seccomp.json for ubuntu

* Set runtime path of runc for ubuntu

* Change path to conmon
2019-12-19 04:37:57 -08:00
Florent Monbillard
109078c5e0 Update CNI plugins to v0.8.3 (#5453) 2019-12-16 04:53:36 -08:00
bozzo
c0b262a22a Add kube-router configuration to enable metrics exposure (#5416) 2019-12-16 04:35:36 -08:00
Douglas Schilling Landgraf
538f1f1a68 cri-o: redhat.yml - remove package cri-tools (#5444)
There is no cri-tools package in CentOS/EPEL/Red Hat.
Additionally, cri-tools is provided into the installation via
roles/download/defaults/main.yml:104:crictl_download_url.
2019-12-16 02:53:36 -08:00
Andreas Krüger
370a0635fa Bump nodelocaldns version to 1.15.8 (#5447)
* Bump nodelocaldns version

* Add missing upstreamsvc
2019-12-13 02:22:55 -08:00
Bort Verwilst
db2ca014cb Add Helm 3.x support (#5441)
* Add Helm 3.x support

* tiller enabled when helm < 3.0.0
2019-12-12 09:24:32 -08:00
Maxime Guyot
815eebf1d7 Add wait for kubectl get ds after upgrades (#5433) 2019-12-11 11:23:55 -08:00
Matthew Mosesohn
696fcaf391 Ensure 0644 mode for ca.crt on nodes (#5428)
Change-Id: I5e018dfaeffe314300b373aeb7ed5f59929cf4f9
2019-12-11 00:54:04 -08:00
Etienne Champetier
5e0140d62c Add k8s 1.15.6 hashes (#5342)
Signed-off-by: Etienne Champetier <champetier.etienne@gmail.com>
2019-12-10 00:45:30 -08:00
Craig Rodrigues
717fe3cf3a Add checksums for v1.17.0 (#5423) 2019-12-09 21:15:28 -08:00
Yujun Zhang
32d80ca438 Add default value for bin_dir in recover control plane (#5396) 2019-12-09 02:54:02 -08:00
Sergey
9fda84b1c9 set node label via kubectl label command (#5257)
* set varios node label via kubectl label command, not kubelet options

* remove node_labels from KUBELET_ARGS
2019-12-09 01:43:09 -08:00
Etienne Champetier
42702dc1a3 Fixes for CentOS 8 (#5213)
* Fix python3-libselinux installation for RHEL/CentOS 8

In bootstrap-centos.yml we haven't gathered the facts,
so #5127 couldn't work

Minimum ansible version to run kubespray is 2.7.8,
so ansible_distribution_major_version is defined an there is no need to default it

Signed-off-by: Etienne Champetier <champetier.etienne@gmail.com>

* Restart NetworkManager for RHEL/CentOS 8

network.service doesn't exist anymore
 # systemctl status network
 Unit network.service could not be found.

Signed-off-by: Etienne Champetier <champetier.etienne@gmail.com>

* Add module_hotfixes=True to docker / containerd yum repo config

https://bugzilla.redhat.com/show_bug.cgi?id=1734081
https://bugzilla.redhat.com/show_bug.cgi?id=1756473
Without this setting you end up with the following error:
 # yum install docker-ce
 Failed to set locale, defaulting to C
 Last metadata expiration check: 0:03:21 ago on Thu Sep 26 22:00:05 2019.
 Error:
  Problem: package docker-ce-3:19.03.2-3.el7.x86_64 requires containerd.io >= 1.2.2-3, but none of the providers can be installed
   - cannot install the best candidate for the job
   - package containerd.io-1.2.2-3.3.el7.x86_64 is excluded
   - package containerd.io-1.2.2-3.el7.x86_64 is excluded
   - package containerd.io-1.2.4-3.1.el7.x86_64 is excluded
   - package containerd.io-1.2.5-3.1.el7.x86_64 is excluded
   - package containerd.io-1.2.6-3.3.el7.x86_64 is excluded
 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Signed-off-by: Etienne Champetier <champetier.etienne@gmail.com>
2019-12-09 01:37:10 -08:00
Maxime Guyot
b15d41a96a Add support to Ansible 2.9 (#5361) 2019-12-05 07:24:32 -08:00
Matthew Mosesohn
7da2083986 Add toleration for calico-typha on master (#5405)
Change-Id: Iea9a366cf6ccc4d491bfc49c5d2dba6d98f81b69
2019-12-05 06:24:32 -08:00
Hugo Blom
f7aea8ed89 update oidc to contain quotes (#5406) 2019-12-05 00:24:32 -08:00
Matthew Mosesohn
57fef8f75e Allow customizing kubelet healthz port and bind addr (#5403)
Change-Id: I1634ba2d2d3337243ffcdea86750003a559f2576
2019-12-03 11:56:58 -08:00
Matthew Mosesohn
f599a4a859 force other resolvers to be secondary when using systemd-resolved (#5391)
Change-Id: I33d46c7e0c5374467e22c5a652b282d1703dea85
2019-12-02 08:41:04 -08:00
Matthew Mosesohn
18cee65c4b Add support for k8s v1.17.0-rc.1, remove hyperkube (#5378)
Change-Id: I3fff04f0211cd9c2e8235acaf51c3aa98abc8bb7
2019-11-28 05:41:03 -08:00
Yujun Zhang
aec5080a47 kubernetes/masters: fix task name in kubeadm setup (#5377) 2019-11-27 06:05:20 -08:00
Anton Fayzrahmanov
80418a44d5 CoreDNS deployment extra tolerations (#5364)
* Add extra tolerations for coredns

* dns_extra_tolerations option

* dns_extra_tolerations

* missing starting space in comment
2019-11-27 05:49:21 -08:00
Florian Ruynat
257c20f39e add 1.16.3 checksums and set new version as default (#5384) 2019-11-27 01:29:20 -08:00
Aaron Crickenberger
f1498d4b53 fix OWNERS file (#5359)
Initially this was to fix a mis-indented approvers key. However, it turns
out that 'oilbeater' is not a member of kubernetes-sigs nor
kubernetes-incubator (the org this repo was migrated from). Thus this
OWNERS file is failing prow's validation check.

As a workaround I've opted to move them to emeritus_approver, which
isn't valiated and can be used as a hint for other approvers in this
repo
2019-11-25 17:59:11 -08:00
Etienne Champetier
18d19d9ed4 containerd: update to 1.2.10 (#5341)
Lot's of bugs and security fixes:
https://github.com/containerd/containerd/releases/tag/v1.2.10
CVE-2019-16884 / CVE-2019-16276
https://github.com/containerd/containerd/releases/tag/v1.2.9
CVE-2019-9512 / CVE-2019-9514 / CVE-2019-9515
https://github.com/containerd/containerd/releases/tag/v1.2.8
CVE-2019-9512 / CVE-2019-9514
https://github.com/containerd/containerd/releases/tag/v1.2.7

Signed-off-by: Etienne Champetier <champetier.etienne@gmail.com>
2019-11-22 00:09:29 -08:00
Michael Shen
6924c6e5a3 [FIX] fix match because trim removes leading/trailing whitespace (#5356) 2019-11-19 22:35:18 -08:00
Matthew Mosesohn
85c851f519 scale down coredns on each master during graceful upgrade (#5344)
This fixes the scenario where masters are upgraded one at a time
and coredns gets improperly scaled back up to 2 replicas.

Change-Id: I7cc9283f40efcfd61b5813c89a5805c95d901567
2019-11-18 00:13:41 -08:00
Matthew Mosesohn
8b67159239 Do not run kubeadm upgrade on first deploy (#5339)
Change-Id: I68a962a9dd28c83ef07eaeaf53eb98287f38bca9
2019-11-14 02:05:34 -08:00
LuciferInLove
4f70da2731 Added Amazon Linux 2 support for deploying with docker (#5301) 2019-11-11 07:05:41 -08:00
Matthew Mosesohn
db5040e6ea Set certs and files with kubeadm token to mode 0640 (#5325)
Change-Id: I298496e55a6889c158b2085fcadeda5e679a873e
2019-11-11 05:41:41 -08:00
Jacopo Secchiero
97764921ed Fix calico name resolution (#5291) 2019-11-11 04:01:41 -08:00
Bjoern Teipel
8c15db53b2 Fix helm for Kubernetes 1.16.2 (#5332)
Since upgrading k8s beyond 1.16.0 version, helm init does
no longer work with helm < 2.16.0 due to
https://github.com/helm/helm/issues/6374

This PR closes issue #5331
2019-11-11 03:53:41 -08:00
Julien Pervillé
0200138a5d Pass ingress_nginx_extra_args when deploying the nginx-ingress addon (#5321) 2019-11-11 03:51:40 -08:00
Florent Monbillard
14af98ebdc Respect cri-tool supported version matrix (#5241)
| Kubernetes Version | cri-tools Version |
|--------------------|-------------------|
| 1.16.x             | v1.16.0           |
| 1.15.X             | v1.15.0           |
| 1.14.X             | v1.14.0           |
| 1.13.X             | v1.13.0           |
| 1.12.X             | v1.12.0           |
| 1.11.X             | v1.11.1           |

- Upgrade to cri-tools 1.16.1
- Add checksums for cri-tools 1.16.1
2019-11-11 03:45:42 -08:00
YichenWong
8a5434419b fix useradd etcd (#5281) 2019-11-11 03:27:41 -08:00
Quentin Gliech
8a406be48a Fix indentation in cilium-ds.yml template (#5305) 2019-11-11 03:25:41 -08:00
Junho Suh
076f254a67 Add cilium_tunnel_mode variable to the cilium config (#5295) 2019-11-11 03:19:42 -08:00
Dmitry Chusovitin
45d151a69d containerd installation on Debian (#5326) 2019-11-11 02:41:41 -08:00
Matthew Mosesohn
bd014c409b Skip coredns image when evaluating kubeadm images (#5327)
It will be enabled correctly in downloads

Change-Id: Ief0b7aa2a8ee2ba6a6849820802f8542584b2c04
Related-story: PRODX-1171
2019-11-09 00:51:39 -08:00
Matthew Mosesohn
1c25ed669c Remove unnecessary and risky reload network for resolvconf propagation (#5322)
Change-Id: I54d706f7941b4b86c4c6cd45340295577155b884
2019-11-06 10:11:52 -08:00
Matthew Mosesohn
a005d19f6f Enable systemd-resolved DNS resolution mode (#5318)
Change-Id: If3e253a40782e03cde7fc4a91493517ae31fda17
2019-11-06 03:33:52 -08:00
Matthew Mosesohn
471589f1f4 Scale down coredns created by kubeadm upgrade to 0 replicas (#5308)
Change-Id: I128b0f9c1acbb956d9a6c4e5510b45a36e296af7
2019-11-05 03:34:38 -08:00
Ali Sanhaji
b0ee1f6cc6 Deploy Cinder CSI driver to provision volumes over OpenStack (#5184)
* Deploy Cinder CSI driver to provision volumes over OpenStack

* Deploy Cinder CSI StorageClass

* Cinder CSI doc
2019-11-01 00:59:24 -07:00
Matthew Mosesohn
186ec13579 Fix incorrect suggestion to enable old k8s apis (#5292)
Change-Id: If965cc6aa0daaca232dcf2ca0efd649aa097497f
2019-10-30 01:58:53 -07:00
Matthew Mosesohn
2c4e6b65d7 Raise delay and retry for rotate tokens (#5304)
Change-Id: I87844b43b9a18064e7a99567ce57c1ca1ffcc4a8
2019-10-30 01:56:52 -07:00
Matthew Mosesohn
94d4ce5a6f Retry cleaning up calico-node container (#5302)
Change-Id: Iad27b107860213759c7ae51f0891d7e5e7c6d96b
2019-10-28 05:11:25 -07:00
Matthew Mosesohn
81da231b1e Set cluster DNS in kubeadm config for kubelet dynamic config (#5293)
Change-Id: I23116efefe8626d361d1904fc6fb8448f66cf3c5
2019-10-25 02:23:40 -07:00
Matthew Mosesohn
a1fff30bd9 Generate TLS certs for calico typha (#5258)
* Generate TLS certs for calico typha

Change-Id: I3883f49c124c52d0fc5b900ca2b44e4e2ed0d707

* Add group vars note

Change-Id: I63550dfef616e884efdbd42010a90b2c04c5eb69
2019-10-17 07:02:38 -07:00
Sergey
81d57fe658 set calico_datastore default value in role kubespray-default (#5259) 2019-10-17 05:58:38 -07:00
Sergey
3118437e10 check on all cluster node - kubelet_max_pods <= (2 ** (32 - kube_network_node_prefix | int)) - 2 (#5279) 2019-10-17 05:48:38 -07:00
Sergey
65e461a7c0 download container always been on download_delegate host (#5177)
* download container always been on download_delegate host

* fix also check pull required
2019-10-17 05:38:38 -07:00
Michael Oglesby
c672681ce5 Revert Pull Request #5084 (#5120)
Kubespray Pull Request #5084 (https://github.com/kubernetes-sigs/kubespray/pull/5084) caused more problems than it solved due to limitations with the synchronize module. See comments on Kubespray Issues #5059 (https://github.com/kubernetes-sigs/kubespray/issues/5059) and #5116 (https://github.com/kubernetes-sigs/kubespray/issues/5116). Details from Ansible documentation: "Currently, synchronize is limited to elevating permissions via passwordless sudo. This is because rsync itself is connecting to the remote machine and rsync doesn’t give us a way to pass sudo credentials in. ... Currently there are only a few connection types which support synchronize (ssh, paramiko, local, and docker) because a sync strategy has been determined for those connection types. Note that the connection for these must not need a password as rsync itself is making the connection and rsync does not provide us a way to pass a password to the connection. ..." Thus, reverting Pull Request #5084.
2019-10-17 05:26:37 -07:00
yelhouti
d332a254ee install python3 instead of python2 for fedora >= 30 fixes 5056, fixes 4802 (#5111) 2019-10-17 05:04:38 -07:00
Matthew Rapa
3debb8aab5 add KUBELET_VOLUME_PLUGIN to kubelet.env (#5128) 2019-10-16 20:08:38 -07:00
YichenWong
aada6e7e40 Add etcd_data_dir variable to the kubeadm config (#5263) 2019-10-16 19:50:39 -07:00
Matthew Mosesohn
ac60786c6f Add support for restart handlers for control plane on crio/containerd (#5250)
* Add support for restart handlers for control plane on crio/containerd

Change-Id: I8343cc4e9df7f55b732628ed01cc6e7ea5dcee85

* Update main.yml
2019-10-16 18:58:39 -07:00
Hugo Blom
db33dc6938 Add support for Kubernetes 1.16.2 (#5272)
* Add support for Kubernetes 1.16.1

* Defaults to 1.16.1

* add 1.16.2 checksums and set new version as default

* correct 1.16.2 checksums and add 1.15.5 checksums
2019-10-16 18:34:38 -07:00
Hugo Blom
9dfb25cafd fix typo (#5275) 2019-10-16 18:26:38 -07:00
Maxime Guyot
df8d2285b6 Update ingress-nginx to v0.26.1 (#5268) 2019-10-16 18:22:39 -07:00