Commit graph

400 commits

Author SHA1 Message Date
Matthew Mosesohn
ec08303f82 Revert "Fix #4237: update kube cert path (#4354)" (#4369)
This reverts commit ea7a6f1cf1.

This change modified the certs dir for Kubernetes, but did not move the directories for existing clusters.
2019-03-20 05:56:57 -07:00
Dmitry Chepurovskiy
ea7a6f1cf1 Fix #4237: update kube cert path (#4354) 2019-03-17 23:55:11 -07:00
Bort Verwilst
33024731e4 Upgrade to k8s 1.13.4 (#4319) 2019-03-06 23:16:56 -08:00
Peter Metz
26ca58419f feat(external-provisioner): adds support for local-path-provisioner (#4232)
* feat(external-provisioner/local-path-provisioner): adds support for local path provisioner

Helpful for local development but also in production workloads (once the
permission model is worked out) where you have redundancy built into the
software uses the PVCs (e.g. database cluster with synchronous
replication)

* feat(local-path-provisioner): adds debug flag, image tag group var

* fix(local-path-provisioner): moves image repo/tag to download role

* test(gce_centos7-flannel): enables local-path-provisioner in test case

* fix(addons): add image repo/tag to commented default values

* fix(local-path-provisioner): typo in jinja template for local path provisioner

* style(local-path-provisioner): debug flag condition re-formatted

* fix(local-path-provisioner): adds missing default value for debug flag

* fix(local-path-provisioner): syntax fix for debug if condition end

* fix(local-path-provisioner): jinja template syntax: if condition white space
2019-02-25 22:45:30 -08:00
Christian Berendt
44ee4b507c terraform: use openstackclient instead of novaclient (#4280)
The openstackclient is the preferred CLI for OpenStack
environments and should be used instead of novaclient.
2019-02-25 20:13:16 -08:00
Kaoet
aadef80404 Upgrade to latest version of ubuntu-nvidia-driver-installer. (#4296)
The lastest version of ubuntu-nvidia-driver-installer contains a fix for
https://github.com/GoogleCloudPlatform/container-engine-accelerators/issues/90
which causes the installer pod to crash when driver is already loaded.
2019-02-24 22:22:48 -08:00
Rong Zhang
d4a36aa55b
Merge pull request #4027 from riverzhang/kube-proxy
Add update server field in kube-proxy kubeconfig
2019-02-20 13:41:06 +08:00
Kaoet
192f4c4e96 Allow customizing container image path used in NVIDIA GPU addon. (#4229) 2019-02-14 03:51:38 -08:00
Jeff Bornemann
c41c1e771f OCI Cloud Provider Update (#4186)
* OCI subnet AD 2 is not required for CCM >= 0.7.0

Reorganize OCI provider to generate configuration, rather than pull

Add pull secret option to OCI cloud provider

* Updated oci example to document new parameters
2019-02-11 12:08:53 -08:00
Chad Swenson
6878c2af4e Fix kube_hostname_override inconsistencies (#4185) 2019-02-06 22:20:11 -08:00
Bort Verwilst
db2b76a22a update k8s to 1.13.3 (#4192)
* update k8s to 1.13.3

* update README as well
2019-02-06 10:48:05 -08:00
Samina Fu
58c71d8ea6 Add Setting Multi on group_vars (#4054) 2019-01-31 23:48:13 -08:00
Erwan Miran
f6d60a7e89 Calico: Ability to define the default IPPool CIDR (instead of kube_pods_subnet) (#4131)
* Calico: Ability to define the default IPPool CIDR (instead of kube_pods_subnet)

* Documentation for calico_pool_cidr (and calico_advertise_cluster_ips which has been forgotten...)
2019-01-31 13:39:13 -08:00
Thomas Nys
68fd7e39da Set cluster DNS correctly in case of nodelocal dns cache (#3879)
* Set cluster DNS correctly in case of nodelocal dns cache

* Pass in cluster_ip based on dns mode

* Disable nodelocaldns by default

* Fix syntax error

* Fix syntax issue

* Add nodelocadns ip to vars of node installation

* Change location of nodelocaldns_ip

* Try to remove newlines from jinja template

* Add debug for config file

* Move parameter logic outside of template

* Adapt templates after feedback

* Remove debugging
2019-01-28 23:39:27 -08:00
wangxf
a096761306 [PR-Calico]Support calico 3.4.0 (#4102)
* Suport calico 3.4.0

Signed-off-by: wangxf1987 <xiaofeix.wang@gmail.com>

* Remove symlink + cni conflist template when 3.3.0+, handle Canal, addition of install-cni: sidecar(3.3.0) or initontainer(3.4.0), KUBECONFIG_FILEPATH, calico_cert_dir, advertise cluster ips

* scheduler.alpha.kubernetes.io/critical-pod deprecated since 1.12
2019-01-28 11:03:49 -08:00
rongzhang
3ed5f89cf5 Add update server field in kube-proxy kubeconfig
I know this is a bit hack.
If you use cloud LB, you can use kubeadm's controlPlaneEndpoint to configure kube-proxy's server field.
But for nginx-proxy, it didn't start when kubeadm init.
2019-01-28 14:45:43 +08:00
rongzhang
0b09c8154a Upgrade kubernetes to v1.13.2 2019-01-11 14:32:42 +08:00
Chad Swenson
72802e4d8d Bugfixes for Local Volume Provisioner
- Fixed an issue where storage class host directories were looped
through excessive target hosts
- Fixes examples in the LVP `README.md` to use nested dicts instead of a
list of dicts
2019-01-08 17:45:20 -06:00
Wilmar den Ouden
4fb8adb9e4 More dynamic local-storage-provisioner approach (#3472)
* Makes local volume provisioner more dynamic

* Correct variable name in local storage provisioner defaults

* Updates external-provisioner readme

* Updates variable naming to be more clear, more documentation, fixes sample inventory

* Variable refactor, untangled some jinja2 loops

* Corrects variable name

* No variable substitution in dict keys, replaced with anchor

* Fixes default storage_classes dict, inline docs

* Fixes spelling in inline docs

* Addresses comments in review

* Updates all the defaults

* Fix failing CI task

* Fixes external provisioner daemonset
2019-01-08 12:36:44 -08:00
Aivars Sterns
39d7503069
Merge pull request #3959 from elementyang/pr-ingress
fix ingress nodeSelector label
2019-01-04 08:58:16 +00:00
okamototk
8216e821d3 Fix kubeadm v1beta1 configuration taint (#3928)
* Use master node taint same as kubeadm configuration v1alpha3 or before.
2019-01-03 03:42:23 -08:00
Chad Swenson
80379f6cab Fix kube-proxy configuration for kubeadm (#3958)
- Creates and defaults an ansible variable for every configuration option in the `kubeproxy.config.k8s.io/v1alpha1` type spec
  - Fixes vars that were orphaned by removing non-kubeadm
  - Fixes previously harcoded kubeadm values
- Introduces a `main` directory for role default files per component (requires ansible 2.6.0+)
  - Split out just `kube-proxy.yml` in this first effort
- Removes the kube-proxy server field patch task

We should continue to pull out other components from `main.yml` into their own defaults files as I did here for `defaults/main/kube-proxy.yml`. I hope for and will need others to join me in this refactoring across the project until each component config template has a matching role defaults file, with shared defaults in `kubespray-defaults` or `downloads`
2019-01-03 00:04:26 -08:00
elementyang
e1e13b68b3 fix ingress nodeSelector label 2018-12-29 14:41:23 +08:00
Hedayat Vatankhah (هدایت)
fbe9e0ac1a Fix docker_options definition when docker_version is 'latest' rather than a number (#3919)
- NOTE: it assumes that the 'latest' version is newer than 17.05
2018-12-20 00:58:21 -08:00
rongzhang
435ef14379 Upgrade kubernetes to v1.13.1 2018-12-19 15:13:43 +08:00
Thomas Nys
3e3ee0aeb1 Add support for running a nodelocal dns cache (#3861)
* Add support for running a nodelocal dns cache

After encountering dns issues in a cluster I was recently working on I
noticed Kubernetes 1.13 introduced support for running a nodelocal dns
cache.

I believe this can usefull for more people.

73b548db06
https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/0030-nodelocal-dns-cache.md

* Add requested changes

* Add additional requested changes + documentation

* Add requested changes after review

* Replace incorrect variable
2018-12-10 17:28:03 -08:00
Zohar Mamedov
456596710e kube-router manifest DSR adjustments (#3828) 2018-12-10 00:40:39 -08:00
Rong Zhang
225f765b56 Upgrade kubernetes to v1.13.0 (#3810)
* Upgrade kubernetes to v1.13.0

* Remove all precense of scheduler.alpha.kubernetes.io/critical-pod in templates

* Fix cert dir

* Use kubespray v2.8 as baseline for gitlab
2018-12-06 12:11:48 -08:00
Andreas Krüger
ddffdb63bf Remove non-kubeadm deployment (#3811)
* Remove non-kubeadm deployment

* More cleanup

* More cleanup

* More cleanup

* More cleanup

* Fix gitlab

* Try stop gce first before absent to make the delete process work

* More cleanup

* Fix bug with checking if kubeadm has already run

* Fix bug with checking if kubeadm has already run

* More fixes

* Fix test

* fix

* Fix gitlab checkout untill kubespray 2.8 is on quay

* Fixed

* Add upgrade path from non-kubeadm to kubeadm. Revert ssl path

* Readd secret checking

* Do gitlab checks from v2.7.0 test upgrade path to 2.8.0

* fix typo

* Fix CI jobs to kubeadm again. Fix broken hyperkube path

* Fix gitlab

* Fix rotate tokens

* More fixes

* More fixes

* Fix tokens
2018-12-06 02:33:38 -08:00
Aivars Sterns
d30dbdde23 Update all kubernetes-incubator/kubespray refs to kubernetes-sigs/kubespray (#3780) 2018-11-28 09:15:25 +01:00
Erwan Miran
551317f1cd Fix docker_options jinja syntax (#3770) 2018-11-27 07:13:15 -08:00
Rong Zhang
ddc19f43ba Add cloud provider config to kubeadm deployments (#3766) 2018-11-27 05:03:03 -08:00
Michal Belica
993b8e2791 Add support to set tolerations for ingress-nginx (#3742)
Introduced variable `ingress_nginx_tolerations` to set custom
tolerations for Ingress nginx daemonset, to be able to schedule
ingress-nginx on dedicated nodes with taints.
2018-11-27 03:30:16 -08:00
Rong Zhang
02169e8f85 Upgrade kubernetes to 1.12.3 (#3767) 2018-11-26 23:22:15 -08:00
Andreas Krüger
bad886ca9b Update defaults to match k8s 1.12 suggestions (#3760)
* Update defaults to match k8s 1.12 suggestions

* Test if Netchecker works with node ip instead of localhost

* Update defaults to ipvs and coredns

* Update defaults for kube_apiserver_insecure_port

* Update main.yaml
2018-11-26 15:36:39 -08:00
okamototk
967a042321 Add flag to deploy container engine manually. (#3753)
This feature was removed by PR#3061. But change flag manage_docker to deploy_container_engine.
2018-11-26 07:26:40 -08:00
Rong Zhang
07d2f1aa36 Add some warning information about deprecating non-kubeadm code (#3759) 2018-11-26 01:17:31 -08:00
okamototk
c5e425b02b Support Metrics Server as addon (#3560). (#3563)
* Support Metrics Server as addon (#3560).

* Update metrics server v0.3.1.

* Add metrics server test.

* Replace metrics server manifests with kubernetes/cluster/addons's.

* Modify metrics server manifests for kubespray.

* Follow PR#3558 node label node-role.kubernetes.io/master change

* Fix metrics server parameters base_metrics_server_... to metrics_server_...

* Fix too hard corded metrics_server_memory_per_node

* Add configurable insecure tls for metrics-apiservice

* Downloadable addon-resizer and extract parameter as variables

* Remove metrics server version from deployment name

* Metrics Server work when all masters has node role

* Download metrics-server and add-resizer container only on master

* ServiceAccount and ConfigMap is separated and fix application name

* Remove old metrics server clusterrole template

* Fix addon-resizer image specify

* Make InternalIP default for metrics_server_kubelet_preferred_address_types

Make InternalIP default because multiple preferrred address types does not work.
2018-11-23 00:36:21 -08:00
Rong Zhang
0cfcd39d55 Switch to kubeadm deployment mode (#3461)
* Switch to kubeadm deployment mode

Discuss:https://github.com/kubernetes-incubator/kubespray/issues/3301

* Add non-kubeadm upgrage to kubeadm cluster
2018-11-21 01:35:40 -08:00
elementyang
1ebb670141 fix modify deprecated --graph flag 2018-11-17 14:22:14 +08:00
Johnny Halfmoon
618ab93b42 added rpm caching for to docker repo (#3718) 2018-11-16 02:33:23 -08:00
Antoine Legrand
3dcb914607 Remove Vault (#3684)
* Remove Vault

* Remove reference to 'kargo' in the doc

* change check order
2018-11-10 08:51:24 -08:00
Bily Zhang
6c14f35f00 Fix some typos (#3636)
Signed-off-by: mooncake <xcoder@tenxcloud.com>
2018-11-05 15:22:16 -08:00
xichengliudui
4a4a3f759c Fix typo 2018-10-29 06:10:33 -04:00
Rong Zhang
dbe99b59a7 Upgrade kubernetes to v1.12.2 (#3597) 2018-10-29 00:58:24 -07:00
Erwan Miran
79bf74e90f Offline deployment: PyPi repo (#3542) 2018-10-23 22:22:09 -07:00
Egor
ccc3f89060 Add kube-router annotations (#3533) 2018-10-21 00:35:52 -07:00
Matthew Mosesohn
127969d65f Align node-role value for kubeadm compatibility (#3558)
kubeadm sets node label node-role.kubernetes.io/master=''
and this is not configurable. We should use it everywhere.
2018-10-20 07:12:54 -07:00
Rong Zhang
76fe84fe93 Use imageRepository instead of the unifiedControlPlaneImage (#3484) 2018-10-16 07:26:04 -07:00
JuanJo Ciarlante
a5edd0d709 [jjo] add kube-router support (#3339)
* [jjo] add kube-router support

Fixes cloudnativelabs/kube-router#147.

* add kube-router as another network_plugin choice
* support most used kube-router flags via
  `kube_router_foo` vars as other plugins
* implement replacing kube-proxy (--run-service-proxy=true) via
  `kube_proxy_mode: none`, verified in a _non kubeadm_enabled_
  install, should also work for recent kubeadm releases via
  `skipKubeProxyInstall: true` config

* [jjo] address PR#3339 review from @woopstar

* add busybox image used by kube-router to downloads

* fix busybox download groups key

* rework kubeadm_enabled + kube_router_run_service_proxy

- verify it working ok w/the kubeadm_enabled and
  kube_router_run_service_proxy true or false

- introduce `kube_proxy_remove` fact, to decouple logic
  from kube_proxy_mode (which affects kubeadm configmap
  settings, thus no-good to ab-use it to 'none')

* improve kube-router.md re: kubeadm_enabled and kube_router_run_service_proxy

* address @woopstar latest review

* add inventory/sample/group_vars/k8s-cluster/k8s-net-kube-router.yml

* fix kube_router_run_service_proxy conditional for kube-proxy removal

* fix kube_proxy_remove fact (w/ |bool), add some needed kube-proxy tags on my and existing changes

* update kube-router tolerations for 1.12 compatibility

* add PriorityClass to kube-router DaemonSet
2018-10-16 07:15:05 -07:00
Aivars Sterns
9b773185c3
Merge pull request #3184 from oracle/new_oci_controls
Add new OCI cloud controls
2018-10-16 11:29:13 +03:00
Giacomo Longo
3f786542d3 Automatically infer bootstrap_os (#3498)
* Automatically infer bootstrap_os

* Rename bootstrap os to os_family
2018-10-10 23:32:10 -07:00
Pierluigi Lenoci
abe711dcb5 Missing [all] sections inside the sample (#3500)
* Missing [all] sections inside the sample

* Update hosts.ini
2018-10-10 21:37:47 +02:00
Rong Zhang
4f51607145 Upgrade kubernetes to v1.12.1 (#3463)
https://github.com/kubernetes/kubernetes/issues/69214
2018-10-07 13:33:13 -07:00
Chad Swenson
6602760a48 Support multiple local volume provisioner StorageClasses (#3450)
- Local Volume StorageClass configuration is now manged by `local_volume_provisioner_storage_classes`, a list of maps that specifies local storage classes with `name` `host_dir` and `mount_dir` keys per entry
- Tasks and templates updated to loop through local volume storage classes
- Previous defaults for path/class names were not changed
- Fixed an issue where a `kubernetes/preinstall` was creating directories inconsistently with the `kubernetes-apps/external_provisioner/local_volume_provisioner` task
2018-10-05 05:52:25 -07:00
Rong Zhang
af97febb04 Upgrade kubernetes to v1.12.0 (#3410)
* Upgrade kubernetes to v1.12.0

Use kubeadm v1alpha3 config

* Upgrade coredns and etcd

* Upgrage docker to 18.06
2018-10-04 02:05:55 -07:00
sangwook
0536125f75 Better fix for openstack cinder zone issue using ignore-volume-az option (#2980)
* Better fix for openstack cinder zone issue[1][2]
using ignore-volume-az option[3].
[1]: https://github.com/kubernetes-incubator/kubespray/pull/2155
[2]: https://github.com/kubernetes-incubator/kubespray/pull/2346
[3]: https://github.com/kubernetes/kubernetes/pull/53523

* Remove kube-scheduler-policy.yaml
2018-09-27 22:15:47 -07:00
Hoat Le
c7c3effd6f Ansible var should be quoted (#3393)
to fix the follow problem in case quote is not used:

PLAY [k8s-cluster:etcd:calico-rr] **********************************************
ERROR! Syntax Error while loading YAML.
  expected <block end>, but found '<scalar>'

The error appears to have been in '/tmp/vagrant-ansible/inventory/group_vars/k8s-cluster.yml': line 59, column 39, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

kube_oidc_ca_file: {{ kube_cert_dir }}/openid-ca.pem
                                      ^ here
We could be wrong, but this one looks like it might be an issue with
missing quotes.  Always quote template expression brackets when they
start a value. For instance:

    with_items:
      - {{ foo }}

Should be written as:

    with_items:
      - "{{ foo }}"

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.
2018-09-25 23:35:35 -07:00
Wilmar den Ouden
8526c30b63 Replaces nonexisting system_namespace variable (#3389) 2018-09-25 01:39:02 -07:00
Anatoly Rugalev
8f85ea89fa Added download_validate_certs option which allows to disables SSL validation for file downloads 2018-09-21 11:51:17 +02:00
k8s-ci-robot
68acdd71f1
Merge pull request #3172 from Atoms/additional-proxy
Add additional no proxy parameter for more customization
2018-09-20 03:26:29 -07:00
k8s-ci-robot
9fa23ffa21
Merge pull request #3364 from SataQiu/fix-20180920
Remove duplicate persistent_volumes_enabled element in k8s-cluster.yml
2018-09-20 02:21:41 -07:00
SataQiu
2a1f77efc6 remove duplicate persistent_volumes_enabled parameter in k8s-cluster.yml 2018-09-20 17:02:26 +08:00
Andreas Kruger
09b67c1ad5 Remove EFK from Kubespray 2018-09-20 10:44:17 +02:00
Antoine Legrand
08179018d4
Merge branch 'master' into gpu2 2018-09-19 15:02:51 +02:00
Sergey Bondarev
e766dd5582 move calico options from all.yml to k8s-cluster/k8s-net-calico.yml 2018-09-18 21:30:49 +03:00
Sergey Bondarev
93429bc661 create separate options files for network plugins
remove plugin options from common files
2018-09-18 14:29:53 +03:00
k8s-ci-robot
2730c90dcd
Merge pull request #3320 from riverzhang/kubelet
Support dynamic kubelet config
2018-09-18 00:16:04 -07:00
rongzhang
77e08ba204 Support dynamic kubelet config
https://kubernetes.io/blog/2018/07/11/dynamic-kubelet-configuration/
2018-09-18 08:44:39 +08:00
Erwan Miran
bcaf2f9ea3 contiv 1.2.1 2018-09-17 16:45:05 +02:00
AtzeDeVries
91b02c057e Add support for GPU accelerator 2018-09-13 11:53:11 +02:00
k8s-ci-robot
0a720b35af
Merge pull request #3270 from riverzhang/fix-registry
Add insecure_registry config to docker options
2018-09-10 04:28:52 -07:00
rongzhang
f557b54489 Add docker_ to values 2018-09-10 18:05:49 +08:00
rongzhang
0140cf71c8 Upgrade kubernetes to v1.11.3 2018-09-10 15:52:49 +08:00
rongzhang
b249b06036 Move docker options to kubespray-defaults 2018-09-09 10:21:18 +08:00
rongzhang
cb133cba68 Add registry_mirrors config to docker options 2018-09-09 01:21:32 +08:00
rongzhang
c41ca22a78 Planning the configuration of docker parameters 2018-09-09 00:59:59 +08:00
rongzhang
009d2ffc6c Add insecure_registry config to docker options 2018-09-08 23:24:35 +08:00
Jeff Bornemann
83838b7fbc Add new OCI cloud controls 2018-09-04 14:03:17 -04:00
Erwan Miran
a644b7c267 Introducing credentials_dir in order to be able to override it 2018-09-03 18:04:50 +02:00
Atoms
8c9588ab59 Add additional no proxy parameter for more customization 2018-09-03 17:09:58 +03:00
Antoine Legrand
ccda9664e7 remove duplicated var 2018-09-03 12:09:31 +02:00
Antoine Legrand
e98ba9e839 Split group-variables 2018-09-03 12:09:31 +02:00
k8s-ci-robot
64b32146ca
Merge pull request #3217 from mirwan/fix_3215
Fix docker_options definition to remove newlines
2018-09-01 07:29:47 -07:00
Erwan Miran
36a7bdfac1 Fix docker_options definition to remove newlines 2018-09-01 09:55:04 +02:00
Erwan Miran
82a28d6bb3 Add documentation about having HA for etcd 2018-08-31 14:40:25 +02:00
Takashi Okamoto
d407a590a6 container_manager variable to specify runtime. 2018-08-28 06:23:38 +00:00
Takashi Okamoto
659cccc507 Update sample. 2018-08-28 02:25:33 +00:00
k8s-ci-robot
f4feb17629
Merge pull request #2958 from elementyang/etcd-pr
change the way that getting etcd_member_name
2018-08-26 23:55:04 -07:00
k8s-ci-robot
f97515352b
Merge pull request #3161 from nutellinoit/kube_proxy_nodeport_addresses
--nodeport-addresses added on kube-proxy.manifest.j2 and on k8s-cluster.yml
2018-08-25 02:00:19 -07:00
Vasilis Remmas
b61eb7d7f3 Add ETCD_QUOTA_BACKEND_BYTES environment variable 2018-08-24 12:17:34 +02:00
Antoine Legrand
4882531c29
Merge pull request #3115 from oracle/oracle_oci_controller
Cloud provider support for OCI (Oracle Cloud Infrastructure)
2018-08-23 18:22:45 +02:00
Aivars Sterns
23fd3461bc calico upgrade to v3 (#3086)
* calico upgrade to v3

* update calico_rr version

* add missing file

* change contents of main.yml as it was left old version

* enable network policy by default

* remove unneeded task

* Fix kubelet calico settings

* fix when statement

* switch back to node-kubeconfig.yaml
2018-08-23 17:17:18 +03:00
Erwan Miran
80cfeea957 psp, roles and rbs for PodSecurityPolicy when podsecuritypolicy_enabled is true 2018-08-22 18:16:13 +02:00
Samuele Chiocca
5d9908c2c3 --nodeport-addresses added on kube-proxy.manifest.j2
Changed author
2018-08-22 15:32:07 +02:00
Jeff Bornemann
94df70be98 Cloud provider support for OCI (Oracle Cloud Infrastructure)
Signed-off-by: Jeff Bornemann <jeff.bornemann@oracle.com>
2018-08-21 17:36:42 -04:00
rongzhang
59176ebbb9 Add kubeadm controlplaneEndpoint
Nginx LB(default)
Other LB by kubeadm controlplane
2018-08-20 00:57:13 +08:00
Antoine Legrand
ac49bbb336
Merge pull request #2168 from xd007/docker_arm64
fix docker opts incompatible running on aarch64 Redhat/Centos
2018-08-17 16:24:07 +02:00
Rong Zhang
364ab2a6b7
Merge pull request #3113 from riverzhang/support-audit
Support audit
2018-08-16 15:33:43 +08:00
rongzhang
2ffc1afe40 Support audit 2018-08-16 14:38:07 +08:00