Commit graph

404 commits

Author SHA1 Message Date
Rong Zhang
4f51607145 Upgrade kubernetes to v1.12.1 (#3463)
https://github.com/kubernetes/kubernetes/issues/69214
2018-10-07 13:33:13 -07:00
Chad Swenson
6602760a48 Support multiple local volume provisioner StorageClasses (#3450)
- Local Volume StorageClass configuration is now manged by `local_volume_provisioner_storage_classes`, a list of maps that specifies local storage classes with `name` `host_dir` and `mount_dir` keys per entry
- Tasks and templates updated to loop through local volume storage classes
- Previous defaults for path/class names were not changed
- Fixed an issue where a `kubernetes/preinstall` was creating directories inconsistently with the `kubernetes-apps/external_provisioner/local_volume_provisioner` task
2018-10-05 05:52:25 -07:00
Rong Zhang
af97febb04 Upgrade kubernetes to v1.12.0 (#3410)
* Upgrade kubernetes to v1.12.0

Use kubeadm v1alpha3 config

* Upgrade coredns and etcd

* Upgrage docker to 18.06
2018-10-04 02:05:55 -07:00
sangwook
0536125f75 Better fix for openstack cinder zone issue using ignore-volume-az option (#2980)
* Better fix for openstack cinder zone issue[1][2]
using ignore-volume-az option[3].
[1]: https://github.com/kubernetes-incubator/kubespray/pull/2155
[2]: https://github.com/kubernetes-incubator/kubespray/pull/2346
[3]: https://github.com/kubernetes/kubernetes/pull/53523

* Remove kube-scheduler-policy.yaml
2018-09-27 22:15:47 -07:00
Hoat Le
c7c3effd6f Ansible var should be quoted (#3393)
to fix the follow problem in case quote is not used:

PLAY [k8s-cluster:etcd:calico-rr] **********************************************
ERROR! Syntax Error while loading YAML.
  expected <block end>, but found '<scalar>'

The error appears to have been in '/tmp/vagrant-ansible/inventory/group_vars/k8s-cluster.yml': line 59, column 39, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

kube_oidc_ca_file: {{ kube_cert_dir }}/openid-ca.pem
                                      ^ here
We could be wrong, but this one looks like it might be an issue with
missing quotes.  Always quote template expression brackets when they
start a value. For instance:

    with_items:
      - {{ foo }}

Should be written as:

    with_items:
      - "{{ foo }}"

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.
2018-09-25 23:35:35 -07:00
Wilmar den Ouden
8526c30b63 Replaces nonexisting system_namespace variable (#3389) 2018-09-25 01:39:02 -07:00
Anatoly Rugalev
8f85ea89fa Added download_validate_certs option which allows to disables SSL validation for file downloads 2018-09-21 11:51:17 +02:00
k8s-ci-robot
68acdd71f1
Merge pull request #3172 from Atoms/additional-proxy
Add additional no proxy parameter for more customization
2018-09-20 03:26:29 -07:00
k8s-ci-robot
9fa23ffa21
Merge pull request #3364 from SataQiu/fix-20180920
Remove duplicate persistent_volumes_enabled element in k8s-cluster.yml
2018-09-20 02:21:41 -07:00
SataQiu
2a1f77efc6 remove duplicate persistent_volumes_enabled parameter in k8s-cluster.yml 2018-09-20 17:02:26 +08:00
Andreas Kruger
09b67c1ad5 Remove EFK from Kubespray 2018-09-20 10:44:17 +02:00
Antoine Legrand
08179018d4
Merge branch 'master' into gpu2 2018-09-19 15:02:51 +02:00
Sergey Bondarev
e766dd5582 move calico options from all.yml to k8s-cluster/k8s-net-calico.yml 2018-09-18 21:30:49 +03:00
Sergey Bondarev
93429bc661 create separate options files for network plugins
remove plugin options from common files
2018-09-18 14:29:53 +03:00
k8s-ci-robot
2730c90dcd
Merge pull request #3320 from riverzhang/kubelet
Support dynamic kubelet config
2018-09-18 00:16:04 -07:00
rongzhang
77e08ba204 Support dynamic kubelet config
https://kubernetes.io/blog/2018/07/11/dynamic-kubelet-configuration/
2018-09-18 08:44:39 +08:00
Erwan Miran
bcaf2f9ea3 contiv 1.2.1 2018-09-17 16:45:05 +02:00
AtzeDeVries
91b02c057e Add support for GPU accelerator 2018-09-13 11:53:11 +02:00
k8s-ci-robot
0a720b35af
Merge pull request #3270 from riverzhang/fix-registry
Add insecure_registry config to docker options
2018-09-10 04:28:52 -07:00
rongzhang
f557b54489 Add docker_ to values 2018-09-10 18:05:49 +08:00
rongzhang
0140cf71c8 Upgrade kubernetes to v1.11.3 2018-09-10 15:52:49 +08:00
rongzhang
b249b06036 Move docker options to kubespray-defaults 2018-09-09 10:21:18 +08:00
rongzhang
cb133cba68 Add registry_mirrors config to docker options 2018-09-09 01:21:32 +08:00
rongzhang
c41ca22a78 Planning the configuration of docker parameters 2018-09-09 00:59:59 +08:00
rongzhang
009d2ffc6c Add insecure_registry config to docker options 2018-09-08 23:24:35 +08:00
Jeff Bornemann
83838b7fbc Add new OCI cloud controls 2018-09-04 14:03:17 -04:00
Erwan Miran
a644b7c267 Introducing credentials_dir in order to be able to override it 2018-09-03 18:04:50 +02:00
Atoms
8c9588ab59 Add additional no proxy parameter for more customization 2018-09-03 17:09:58 +03:00
Antoine Legrand
ccda9664e7 remove duplicated var 2018-09-03 12:09:31 +02:00
Antoine Legrand
e98ba9e839 Split group-variables 2018-09-03 12:09:31 +02:00
k8s-ci-robot
64b32146ca
Merge pull request #3217 from mirwan/fix_3215
Fix docker_options definition to remove newlines
2018-09-01 07:29:47 -07:00
Erwan Miran
36a7bdfac1 Fix docker_options definition to remove newlines 2018-09-01 09:55:04 +02:00
Erwan Miran
82a28d6bb3 Add documentation about having HA for etcd 2018-08-31 14:40:25 +02:00
Takashi Okamoto
d407a590a6 container_manager variable to specify runtime. 2018-08-28 06:23:38 +00:00
Takashi Okamoto
659cccc507 Update sample. 2018-08-28 02:25:33 +00:00
k8s-ci-robot
f4feb17629
Merge pull request #2958 from elementyang/etcd-pr
change the way that getting etcd_member_name
2018-08-26 23:55:04 -07:00
k8s-ci-robot
f97515352b
Merge pull request #3161 from nutellinoit/kube_proxy_nodeport_addresses
--nodeport-addresses added on kube-proxy.manifest.j2 and on k8s-cluster.yml
2018-08-25 02:00:19 -07:00
Vasilis Remmas
b61eb7d7f3 Add ETCD_QUOTA_BACKEND_BYTES environment variable 2018-08-24 12:17:34 +02:00
Antoine Legrand
4882531c29
Merge pull request #3115 from oracle/oracle_oci_controller
Cloud provider support for OCI (Oracle Cloud Infrastructure)
2018-08-23 18:22:45 +02:00
Aivars Sterns
23fd3461bc calico upgrade to v3 (#3086)
* calico upgrade to v3

* update calico_rr version

* add missing file

* change contents of main.yml as it was left old version

* enable network policy by default

* remove unneeded task

* Fix kubelet calico settings

* fix when statement

* switch back to node-kubeconfig.yaml
2018-08-23 17:17:18 +03:00
Erwan Miran
80cfeea957 psp, roles and rbs for PodSecurityPolicy when podsecuritypolicy_enabled is true 2018-08-22 18:16:13 +02:00
Samuele Chiocca
5d9908c2c3 --nodeport-addresses added on kube-proxy.manifest.j2
Changed author
2018-08-22 15:32:07 +02:00
Jeff Bornemann
94df70be98 Cloud provider support for OCI (Oracle Cloud Infrastructure)
Signed-off-by: Jeff Bornemann <jeff.bornemann@oracle.com>
2018-08-21 17:36:42 -04:00
rongzhang
59176ebbb9 Add kubeadm controlplaneEndpoint
Nginx LB(default)
Other LB by kubeadm controlplane
2018-08-20 00:57:13 +08:00
Antoine Legrand
ac49bbb336
Merge pull request #2168 from xd007/docker_arm64
fix docker opts incompatible running on aarch64 Redhat/Centos
2018-08-17 16:24:07 +02:00
Rong Zhang
364ab2a6b7
Merge pull request #3113 from riverzhang/support-audit
Support audit
2018-08-16 15:33:43 +08:00
rongzhang
2ffc1afe40 Support audit 2018-08-16 14:38:07 +08:00
JohnZheng
b50b3430be Disable locksmithd on CoreOS if coreos_auto_upgrade set to false (#3088)
* Disable locksmithd on CoreOS if coreos_auto_upgrade set to false

* change when format to support multiple-condition
2018-08-14 13:42:16 -05:00
Rong Zhang
a11e1eba9e Upgrade kubernetes to V1.11.x (#3078)
Upgrade Kubernetes to V1.11.2
The kubeadm configuration file version has been upgraded from v1alpha1 to v1alpha2
Add bootstrap kubeadm-config.yaml with external etcd
2018-08-14 15:13:44 +03:00
Rong Zhang
039180b2ca
Merge pull request #3022 from alvistack/weave-2.4.0
weave: Upgrade to 2.4.0
2018-08-09 15:01:05 +08:00
rongzhang
ea6af449a8 Remove istio support
Use helm install or support in future
2018-08-08 11:10:09 +08:00
Wong Hoi Sing Edison
538cb3b1bd weave: Upgrade to 2.4.0
Upstream Changes:

-   weave 2.4.0 (https://github.com/weaveworks/weave/releases/tag/v2.4.0)
-   Support `externalTrafficPolicy: Local` (https://github.com/weaveworks/weave/issues/2924)
-   Make the ipset list size bigger (https://github.com/weaveworks/weave/pull/3305)
-   Break out of kube rm-peers loop if nothing changes (https://github.com/weaveworks/weave/pull/3317)

Our Changes:

-   Revamp weave-net.yml.j2 with upstream changes
-   Add more variables for customization
-   Replace WEAVE_PASSWORD with k8s secret
-   Remove hard-corded seed mode support, in favor of variables customization
2018-08-07 18:34:51 +08:00
Wong Hoi Sing Edison
a0defefb3f ingress-nginx: Upgrade to 0.16.2
ingress-nginx 0.16.2 (https://github.com/kubernetes/ingress-nginx/releases/tag/nginx-0.16.2)

This patch simplify ingress-nginx deployment by default deploy on
master, with customizable options; on the other hand, remove the
additional Ansible group "kube-ingress" and its k8s node label
injection.

Reference to https://kubernetes.io/docs/concepts/services-networking/ingress/#prerequisites:

    GCE/Google Kubernetes Engine deploys an ingress controller on the master.

By changing `ingress_nginx_nodeselector` plus custom k8s node
label, user could customize the DaemonSet deployment target.

If `ingress_nginx_nodeselector` is empty, will deploy DaemonSet on
every k8s node.
2018-07-10 12:26:06 +08:00
elementyang
effd27a5f6 change the way that getting etcd_member_name 2018-07-03 22:02:44 +08:00
Wong Hoi Sing Edison
728024e8ff cephfs-provisioner: Upgrade to 06fddbe2
-   cephfs-provisioner 06fddbe2 (https://github.com/kubernetes-incubator/external-storage/tree/06fddbe2/ceph/cephfs)

Noteable changes from upstream:

-   Added storage class parameters to specify a root path within the backing cephfs and, optionally, use deterministic directory and user names (https://github.com/kubernetes-incubator/external-storage/pull/696)
-   Support capacity (https://github.com/kubernetes-incubator/external-storage/pull/770)
-   Enable metrics server (https://github.com/kubernetes-incubator/external-storage/pull/797)

Other noteable changes:

-   Clean up legacy manifests file naming
-   Remove legacy manifests, namespace and storageclass before upgrade
-   `cephfs_provisioner_monitors` simplified as string
-   Default to new deterministic naming
-   Add `reclaimPolicy` support in StorageClass

With legacy non-deterministic naming style (where $UUID are generated ramdonly):

-   cephfs_provisioner_claim_root: /volumes/kubernetes
-   cephfs_provisioner_deterministic_names: false
-   Generated CephFS volume: /volumes/kubernetes/kubernetes-dynamic-pvc-$UUID
-   Generated CephFS user: kubernetes-dynamic-user-$UUID

With new default deterministic naming style (where $NAMESPACE and $PVC are predictable):

-   cephfs_provisioner_claim_root: /volumes
-   cephfs_provisioner_deterministic_names: true
-   Generated CephFS volume: /volumes/$NAMESPACE/$PVC
-   Generated CephFS user: k8s.$NAMESPACE.$PVC
2018-07-03 10:15:24 +08:00
Andreas Krüger
6ac601fd2d
Merge pull request #2876 from neith00/docker_iptables
parametrized iptables options for docker daemon
2018-06-14 22:23:27 +02:00
neith00
f2f1e7f9d1 parametrized iptables options for docker daemon 2018-06-14 12:16:16 +02:00
Wong Hoi Sing Edison
291dd1aca8 Fixup #2545, cephfs-provisioner: Individual Namespace for Add-on 2018-06-13 21:52:58 +08:00
Wong Hoi Sing Edison
0ad0202e8f Upgrade Kubernetes to 10.0.4 and etcd to 3.2.18 2018-06-07 16:20:29 +08:00
Di Xu
f4d762bb95 fix docker opts incompatible running on aarch64 Redhat/Centos
On Aarch64, the default cgroup driver for docker is systemd
instead of cgroupfs. Should conform kubelet to use systemd
as cgroup driver as well to keep it consistent with docker.

Without this change, below exception will be raised.
/usr/bin/docker-current: Error response from daemon: shim
error: docker-runc not installed on system.

Change-Id: Id496ec9eaac6580e4da2f3ef1a386c9abc2a5129
2018-06-05 16:17:16 +08:00
Miouge1
095d33bc51 Remove KPM support 2018-05-21 22:28:08 +02:00
Andreas Krüger
4ac79993e2
Merge pull request #2666 from AnatolyRugalev/master
Added MountFlags variable to docker options
2018-05-16 09:34:34 +02:00
Matthew Mosesohn
7c93e71801
Upgrade k8s to 1.10.2 (#2748)
* Upgrade k8s to 1.10.2

Bumped etcd version to 3.2.16 as recommended

* Add ipvs fix for v1.10

* change flannel addons test to ha
2018-05-15 16:00:29 +03:00
Anatoly Rugalev
eae4fa040a Added docker_mount_flags option (fixes #2624) 2018-05-15 11:57:18 +02:00
Suzuka Asagiri
f81e6d2ccf
Add oidc-user-prefix and oidc-group-prefix args 2018-04-23 12:23:59 +09:00
Aivars Sterns
4b4786f75d
Merge pull request #2381 from vikas027/inventory_fixes
Replaced ansible_ssh_host with ansible_host in sample inventory file and fixed usage of bastion
2018-04-16 10:06:19 +03:00
Matthew Mosesohn
49e3665d96
Remove prometheus operator from Kubespray (#2658)
Kubespray should not install any helm charts. This is a task
that a user should do on his/her own through ansible or another
tool. It opens the door to wrapping installation of any helm
chart.
2018-04-13 18:53:39 +03:00
Vikas Kumar
94eb18b3d9 Replaced ansible_ssh_host with ansible_host in sample inventory file as the former is deprecated since Ansible v2.0
Fixed the reference of ansible_user in kubespray-defaults role

References:
 - http://docs.ansible.com/ansible/latest/intro_inventory.html
2018-04-10 15:21:40 +10:00
Daniel Hoherd
ca40d51bc6 Fix typos (no logic changes) 2018-04-05 15:54:58 -07:00
Andreas Krüger
deac627dc7
Merge pull request #2571 from hswong3i/ingress-nginx-download
ingress-nginx: container download related things should defined in the download role
2018-03-31 20:51:50 +02:00
bobahspb
16961f69f2
Merge branch 'master' into master 2018-03-31 21:48:39 +03:00
Wong Hoi Sing Edison
5fe144aa0f ingress-nginx: container download related things should defined in the download role 2018-04-01 00:22:33 +08:00
Wong Hoi Sing Edison
195d6d791a Integrate jetstack/cert-manager 0.2.3 to Kubespray 2018-03-31 19:29:11 +08:00
Matthew Mosesohn
03bcfa7ff5
Stop templating kube-system namespace and creating it (#2545)
Kubernetes makes this namespace automatically, so there is
no need for kubespray to manage it.
2018-03-30 14:29:13 +03:00
Vladimir Vasilkin
19e1b11d98 prometheus operator, metrics for k8s cluster
install using Helm:
- Prometheus Operator
- metrics for k8s cluster including: grafana dashboard, alertmanager, node exporters

base project:
https://github.com/coreos/prometheus-operator

the issue:
https://github.com/kubernetes-incubator/kubespray/issues/2042

Previous PR, raw ansible without Helm:
https://github.com/kubernetes-incubator/kubespray/pull/2499
2018-03-28 21:23:30 +03:00
Andreas Krüger
03117d9572
Merge pull request #2488 from LuckySB/ingress-nginx-node-role
Dedicated node for ingress nginx controller
2018-03-28 14:07:40 +02:00
Wong Hoi Sing Edison
f8ebd08e75 Registry Addon Fixup 2018-03-22 21:33:32 +08:00
Chad Swenson
bc68188209
Merge pull request #2498 from zmsp/master
Upgraded kubernetes from 1.9.3 to 1.9.5
2018-03-21 20:25:05 -05:00
mirwan
ee8f678010 Addition of the .creds extension to the credentials files generated by password lookup in order for Ansible not to consider them as inventory files with inventory_ignore_extensions set accordingly (#2446) 2018-03-21 10:50:32 +03:00
Zobair Shahadat
ebfee51aca Upgraded kubernetes from 1.9.3 to 1.9.5 2018-03-19 15:42:24 -04:00
Andreas Krüger
f253691a68
Merge pull request #2347 from hswong3i/multiple_artifacts_dir
Support multiple artifacts under individual inventory directory
2018-03-19 12:45:55 +01:00
Sergey Bondarev
038da7255f check if group kube-ingress is not empty
fix spelling mistaker ingress_nginx_host_network
set default value for ingress_nginx_host_network: false
2018-03-19 12:59:38 +03:00
Sergey Bondarev
1481f7d64b Dedicated node for ingress nginx controller
The ability to create dedicated node for ingress nginx controller
host type network for nginx controller

and add from example https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/static-ip/nginx-ingress-controller.yaml
terminationGracePeriodSeconds: 60
2018-03-17 02:54:46 +03:00
Chad Swenson
7d33650019
Merge pull request #2462 from woopstar/coredns-patch
Add CoreDNS support
2018-03-16 18:33:36 -05:00
woopstar
e40368ae2b Add CoreDNS support with various fixes
Added CoreDNS to downloads

Updated with labels. Should now work without RBAC too

Fix DNS settings on hosts

Rename CoreDNS service from kube-dns to coredns

Add rotate based on http://edgeofsanity.net/rant/2017/12/20/systemd-resolved-is-broken.html

Updated docs with CoreDNS info

Added labels and fixed minor settings from official yaml file: https://github.com/kubernetes/kubernetes/blob/release-1.9/cluster/addons/dns/coredns.yaml.sed

Added a secondary deployment and secondary service ip. This is to mitigate dns timeouts and create high resitency for failures. See discussion at 'https://github.com/coreos/coreos-kubernetes/issues/641#issuecomment-281174806'

Set dns list correct. Thanks to @whereismyjetpack

Only download KubeDNS or CoreDNS if selected

Move dns cleanup to its own file and import tasks based on dns mode

Fix install of KubeDNS when dnsmask_kubedns mode is selected

Add new dns option coredns_dual for dual stack deployment. Added variable to configure replicas deployed. Updated docs for dual stack deployment. Removed rotate option in resolv.conf.

Run DNS manifests for CoreDNS and KubeDNS

Set skydns servers on dual stack deployment

Use only one template for CoreDNS dual deployment

Set correct cluster ip for the dns server
2018-03-16 21:51:37 +01:00
Andreas Krüger
3d6fd49179 Added option for encrypting secrets to etcd v.2 (#2428)
* Added option for encrypting secrets to etcd

* Fix keylength to 32

* Forgot the default

* Rename secrets.yaml to secrets_encryption.yaml

* Fix static path for secrets file to use ansible variable

* Rename secrets.yaml.j2 to secrets_encryption.yaml.j2

* Base64 encode the token

* Fixed merge error

* Changed path to credentials dir

* Update path to secrets file which is now readable inside the apiserver container. Set better file permissions

* Add encryption option to k8s-cluster.yml
2018-03-15 22:20:05 +03:00
Aivars Sterns
710295bd2f
Merge pull request #2434 from protomech/feature/azure-vnet-resource-group
add support for azure vnetResourceGroup
2018-03-13 17:42:09 +02:00
Wong Hoi Sing Edison
a086686e9f Support multiple artifacts under individual inventory directory 2018-03-08 11:57:53 +08:00
Wong Hoi Sing Edison
6402004018 FIXUP #2424: local_provisioner directory should be created only if enabled 2018-03-08 11:57:46 +08:00
Wong Hoi Sing Edison
3f96b2da7a Add Custom ConfigMap Support for ingress-nginx 2018-03-07 21:37:45 +08:00
Michael Beatty
07657aecf4 add support for azure vnetResourceGroup 2018-03-05 13:40:25 -06:00
Wong Hoi Sing Edison
fd46442188 Integrate kubernetes/ingress-nginx 0.11.0 to Kubespray 2018-03-02 23:33:19 +08:00
Aivars Sterns
8b21034b31
Merge pull request #2344 from hswong3i/local_volume_provisioner_fixup
Upgrade Local Volume Provisioner Addon to v2.0.0
2018-03-01 13:12:44 +02:00
Andrew Greenwood
a40d9f3c72 Document a silent killer... (#2373)
Adding this into the default example inventory so it has less of a chance of biting others after weeks of random failures (as etcd does not  express that it has run out of RAM it just stalls).. 512MB was not  enough for us to run one of our products.
2018-02-28 15:36:51 +03:00
Brad Beam
810c10a0e9
Merge pull request #2382 from chechiachang/replace-tab-with-space-and-remove-redundant-spaces
Remove redundant spaces
2018-02-27 10:39:57 -06:00
RongZhang
b7e06085c7 Upgrade to Kubernetes v1.9.3 (#2323)
Upgrade to Kubernetes v1.9.3
2018-02-27 14:31:59 +03:00
David Chang
8875e25fe9 Replace tab with space. Remove redundant spaces 2018-02-27 14:34:58 +08:00
Wong Hoi Sing Edison
d4c61d2628 Fixup for gce_centos7-flannel-addons 2018-02-21 13:41:25 +08:00
Wong Hoi Sing Edison
deef47c923 Upgrade Local Volume Provisioner Addon to v2.0.0 2018-02-21 13:41:25 +08:00
melkosoft
f13e76d022 Added cilium support (#2236)
* Added cilium support

* Fix typo in debian test config

* Remove empty lines

* Changed cilium version from <latest> to <v1.0.0-rc3>

* Add missing changes for cilium

* Add cilium to CI pipeline

* Fix wrong file name

* Check kernel version for cilium

* fixed ci error

* fixed cilium-ds.j2 template

* added waiting for cilium pods to run

* Fixed missing EOF

* Fixed trailing spaces

* Fixed trailing spaces

* Fixed trailing spaces

* Fixed too many blank lines

* Updated tolerations,annotations in cilium DS template

* Set cilium_version to iptables-1.9 to see if bug is fixed in CI

* Update cilium image tag to v1.0.0-rc4

* Update Cilium test case CI vars filenames

* Add optional prometheus flag, adjust initial readiness delay

* Update README.md with cilium info
2018-02-16 21:37:47 -06:00
Wong Hoi Sing Edison
07075add3d Add optional StorageClass name with cephfs_provisioner_storage_class 2018-02-10 20:31:34 +08:00
mlushpenko
a37c642127 Remove obsolete token variables
Tokens are generated automatically during init process and on-demand for nodes joining process
2018-02-09 15:53:12 +01:00
Wong Hoi Sing Edison
b25e0f82b1 Add cephfs_provisioner Support for Kubespray 2018-02-08 22:27:54 +08:00
Wong Hoi Sing Edison
1a1d154e14 Support multiple inventory files under individual inventory directory 2018-02-08 08:08:15 +08:00