Commit graph

232 commits

Author SHA1 Message Date
mervynzhang
a8dfcbbfc7 Switch /root references to ansible_env.HOME (#4842)
* kube config dir for current/ansible become user

* remove extra /

* fix default value
2019-06-06 02:06:11 -07:00
Andreas Krüger
1e470b0473 Fix certificate-key param for kubeadm init (#4789)
* Fix certificate-key param for kubeadm init

* Fix yamllint error
2019-05-22 02:06:11 -07:00
Florent Monbillard
8e28ba38d2 Add Load Balancer IP to API servers SANs (#4775)
- Add loadbalancer_apiserver.address to apiserver_sans
2019-05-16 01:23:42 -07:00
MarkusTeufelberger
e67f848abc ansible-lint: add spaces around variables [E206] (#4699) 2019-05-02 14:24:21 -07:00
Timoses
d6fd0d2aca Enable delegating all downloads (binaries, images, kubeadm images) (#4420)
* Download to delegate and sync files when download_run_once

* Fail on error after saving container image

* Do not set changed status when downloaded container was up to date

* Only sync containers when they are actually required

Previously, non-required images (pull_required=false as
image existed on target host) were synced to the target
hosts. This failed as the image was not downloaded to
the download_delegate and hence was not available for
syncing.

* Sync containers when only missing on some hosts

* Consider images with multiple repo tags

* Enable kubeadm images pull/syncing with download_delegate

* Use kubeadm images list to pull/sync

'kubeadm config images pull' is replaced by collecting the images
list with 'kubeadm config images list' and using the commonly
used method of pull/syncing the images.

* Ensure containers are downloaded and synced for all hosts

* Fix download/syncing when download_delegate is a kubernetes host
2019-05-01 01:10:56 -07:00
Matthew Mosesohn
15eb7db36d Fix k8s api endpoint for secondary nodes in control plane mode (#4675)
Change-Id: I1588458b54c52443ad8d0afbd266f77ac0afea67
2019-04-29 07:50:24 -07:00
Matthew Mosesohn
338eb4ce65 Fix kubeadm upload certs with when condition (#4659)
* Fix kubeadm upload certs with when condition

Change-Id: I916dd2375b71eea2386047c7f185a2f8361f7a61

* Update kubeadm-secondary-experimental.yml
2019-04-27 01:14:20 -07:00
Sergey Kolekonov
4a10dca7d4 Add an ability to provide oidc cert in base64 (#4618) 2019-04-24 09:40:01 -07:00
Matthew Mosesohn
4d57ed314d Clean up check for setting kubeadm certificate key (#4634)
Change-Id: I2c97c4753089eb3ec2e6b01b2681a8be98ecbb57
2019-04-24 07:14:12 -07:00
Matthew Mosesohn
fc072300ea Purge legacy cleanup tasks from older than 1 year (#4450)
We don't need to support upgrades from 2 year old installs,
just from the last major version.

Also changed most retried tasks to 1s delay instead of longer.
2019-04-24 00:08:05 -07:00
Matthew Mosesohn
d6d7458d68 Fix control plane setup without a hardcoded key (#4610) 2019-04-23 14:37:59 -07:00
Matthew Mosesohn
05dc2b3a09 Use K8s 1.14 and add kubeadm experimental control plane mode (#4514)
* Use K8s 1.14 and add kubeadm experimental control plane mode

This reverts commit d39c273d96.

* Cleanup kubeadm setup run on first master

* pin kubeadm_certificate_key in test

* Remove kubelet autolabel of kube-node, add symlink for pki dir

Change-Id: Id5e74dd667c60675dbfe4193b0bc9fb44380e1ca
2019-04-19 06:01:54 -07:00
Victor Morales
c6586829de Ensure /etc/bash_completion.d/ folder exists (#4543)
The Stateless ClearLinux feature[1] requires the creation of folders
in /etc folder. This change ensure the existence of the
/etc/bash_completion.d/ folder for ClearLinux Distribution.

[1] https://clearlinux.org/features/stateless
2019-04-18 02:24:10 -07:00
Maxime Guyot
ec3daedf9e Revert "Fix for unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels (#4320)" (#4553)
This reverts commit 586ad89d50.
2019-04-17 07:58:06 -07:00
Matthew Mosesohn
d39c273d96 Revert "Use K8s 1.14 and add kubeadm experimental control plane mode (#4317)" (#4510)
This reverts commit 316508626d.
2019-04-11 12:52:43 -07:00
Matthew Mosesohn
316508626d Use K8s 1.14 and add kubeadm experimental control plane mode (#4317)
* Use Kubernetes 1.14 and experimental control plane support

* bump to v1.14.0
2019-04-11 05:30:13 -07:00
Robert Neumann
586ad89d50 Fix for unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels (#4320)
* Fix the file path for all.yml and k8s-cluster.yml

* Fix --node-labels namespace error "unknown labels specified"

* Update templates and configs kubelet node-labels
2019-04-10 04:14:12 -07:00
Maxime Guyot
913fed0089 kubeadmn init: add 'until' to make 'retries' effective (#4464)
an 'until' clause is required or 'retries' is ignored

(see note @ https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html#do-until-loops)
2019-04-09 00:21:04 -07:00
rptaylor
f52584a715 robust handling of API server SANs (#4435)
* robust handling of API server SANs

* use apiserver_loadbalancer_domain_name if it is defined, according to PR 3977
2019-04-08 08:10:35 -07:00
Dmitry Chepurovskiy
0440e45d65 Fix supplementary_addresses rendering error (#4403) 2019-03-29 00:26:13 -07:00
Etienne
d0ae316934 Use proxy_env with kubeadm phase commands (#4325) 2019-03-26 03:03:19 -07:00
Matthew Mosesohn
ec08303f82 Revert "Fix #4237: update kube cert path (#4354)" (#4369)
This reverts commit ea7a6f1cf1.

This change modified the certs dir for Kubernetes, but did not move the directories for existing clusters.
2019-03-20 05:56:57 -07:00
Dmitry Chepurovskiy
ea7a6f1cf1 Fix #4237: update kube cert path (#4354) 2019-03-17 23:55:11 -07:00
Matthew Mosesohn
acbf3db233 Remove hard dependence on facts for all nodes (#4304)
* Remove hard dependence on facts for all nodes

* Update main.yaml

* Update main.yaml
2019-03-05 03:04:39 -08:00
Manuel Cintron
07b2894080 Adding ability to maintain existing Encryption Secrets at Rest. (#4255)
* Adding ability to maintain existing Encryption Secrets at Rest.

If secrets_encryption.yaml is present it will not be overriten with a new kube_encrypt_token.

This should allow for it to be set ahead of a playbook running or maintain it if cluster.yml is ran on the same cluster and the ansible host does not have access to the secrets.

* Setting existing kube_encrypt_token across all master nodes in case it was missing in one or more nodes.
2019-02-19 07:31:45 -08:00
peerapach
bd9474bafd fix kubeadm-setup when enable access_ip (#4145) 2019-02-01 20:10:34 -08:00
Chad Swenson
e3ffa21303
Merge pull request #4019 from chadswen/kubeadm-env
Fix PATH for kubeadm init
2019-01-18 11:27:57 -06:00
Matthew Mosesohn
eecaba6b84
Generate external admin.conf with kubeadm (#4056)
* Generate external admin.conf with kubeadm

* Fix apiserver sans
2019-01-16 16:30:50 +03:00
Chad Swenson
aa1d5b8970 Fix PATH for kubeadm init
Right now we're consistently getting warnings about kubelet not found in
path during `kubeadm init`. We fixed this for `kubeadm join` in #3342, and this brings the change to init
as well.
2019-01-09 18:38:02 -06:00
Andreas Holmsten
a34139e19e (Re)add line break for supplementary addr in SANs (#3952)
The change implemented in #3908 remove line breaks for supplementary
addresses in kubeadm SANs, causing errors in the config file and
failure to bring cluster up. This commit reimplement line breaks in
between supplementary addresses.
2019-01-03 00:12:00 -08:00
Rong Zhang
5834e609a6 Add scale master features (#3946)
* Add scale master features

* Add certificate management with kubeadm

* Add kubeadm kubeconfig

* Fix ymalroles error

* fix upgrade cluster fialed

* force update cert and keys when you reconfigure cluster
2018-12-27 23:27:27 -08:00
Seongjin Cho
16715adfa0 Adds support for webhook token auth. (#3939)
Webhook token auth:
https://kubernetes.io/docs/reference/access-authn-authz/authentication/#webhook-token-authentication

Fixes #3063.
2018-12-26 01:52:53 -08:00
Rong Zhang
cd42e649a7 Fix reconfigure and upgrade cluster (#3938) 2018-12-24 23:06:27 -08:00
Rong Zhang
925a820b56 Fix skip upgrade first master (#3915) 2018-12-19 05:16:14 -08:00
Matthew Mosesohn
50b884a32d Fixup line breaks for kubeadm SANs (#3908) 2018-12-19 02:47:31 -08:00
pasqualet
ea833a4cd7 Fix apiServerCertSANs in kubeadm config file (#3839) 2018-12-07 00:11:08 -08:00
Andreas Krüger
d5ce5874e8 Streamline path to certs dir (#3836)
* Streamline path to certs dir

* More fixes

* Set path to etcd certs in kubernetes defaults instead
2018-12-06 23:11:53 -08:00
Rong Zhang
225f765b56 Upgrade kubernetes to v1.13.0 (#3810)
* Upgrade kubernetes to v1.13.0

* Remove all precense of scheduler.alpha.kubernetes.io/critical-pod in templates

* Fix cert dir

* Use kubespray v2.8 as baseline for gitlab
2018-12-06 12:11:48 -08:00
Andreas Krüger
ddffdb63bf Remove non-kubeadm deployment (#3811)
* Remove non-kubeadm deployment

* More cleanup

* More cleanup

* More cleanup

* More cleanup

* Fix gitlab

* Try stop gce first before absent to make the delete process work

* More cleanup

* Fix bug with checking if kubeadm has already run

* Fix bug with checking if kubeadm has already run

* More fixes

* Fix test

* fix

* Fix gitlab checkout untill kubespray 2.8 is on quay

* Fixed

* Add upgrade path from non-kubeadm to kubeadm. Revert ssl path

* Readd secret checking

* Do gitlab checks from v2.7.0 test upgrade path to 2.8.0

* fix typo

* Fix CI jobs to kubeadm again. Fix broken hyperkube path

* Fix gitlab

* Fix rotate tokens

* More fixes

* More fixes

* Fix tokens
2018-12-06 02:33:38 -08:00
Rong Zhang
0cfcd39d55 Switch to kubeadm deployment mode (#3461)
* Switch to kubeadm deployment mode

Discuss:https://github.com/kubernetes-incubator/kubespray/issues/3301

* Add non-kubeadm upgrage to kubeadm cluster
2018-11-21 01:35:40 -08:00
Matthew Mosesohn
ac00d23b80 Skip etcd upgrade steps in kubeadm because it is not used (#3737) 2018-11-19 06:29:58 -08:00
Matthew Mosesohn
ff09141a14 Retry kubeadm proxy and secondary master init tasks (#3715)
Due to suboptimal external loadbalancer configs, the LoadBalancer
might point to a downed kube-apiserver that is not set up yet.
2018-11-15 10:03:23 -08:00
Erwan Miran
1ad1e80ae3 Checking new CA key presence is not relevant to determine if kubeadm has already run (#3653) 2018-11-07 11:46:11 -08:00
Matthew Mosesohn
7e84de2ae1 Purge /root/.kube/config when migrating to kubeadm (#3566) 2018-10-23 05:09:11 -07:00
Matthew Mosesohn
4bdd0ce417 Allow kubeadm master untaint to fail (#3549) 2018-10-19 00:38:12 -07:00
Erwan Miran
87193fd270 Fix ansible syntax to avoid ansible warnings (one more) (#3536)
* warning on meta flush_handlers

* avoid rm

* avoid "Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually" warning on subsequent tasks using blockinfile

* is match
2018-10-17 12:27:11 -07:00
Erwan Miran
7bec169d58 Fix ansible syntax to avoid ansible deprecation warnings (#3512)
* failed

* version_compare

* succeeded

* skipped

* success

* version_compare becomes version since ansible 2.5

* ansible minimal version updated in doc and spec

* last version_compare
2018-10-16 15:33:30 -07:00
okamototk
c825f4d180 Untaint master when it has node role (#3466) 2018-10-09 01:40:43 -07:00
Rong Zhang
af97febb04 Upgrade kubernetes to v1.12.0 (#3410)
* Upgrade kubernetes to v1.12.0

Use kubeadm v1alpha3 config

* Upgrade coredns and etcd

* Upgrage docker to 18.06
2018-10-04 02:05:55 -07:00
sangwook
0536125f75 Better fix for openstack cinder zone issue using ignore-volume-az option (#2980)
* Better fix for openstack cinder zone issue[1][2]
using ignore-volume-az option[3].
[1]: https://github.com/kubernetes-incubator/kubespray/pull/2155
[2]: https://github.com/kubernetes-incubator/kubespray/pull/2346
[3]: https://github.com/kubernetes/kubernetes/pull/53523

* Remove kube-scheduler-policy.yaml
2018-09-27 22:15:47 -07:00
rongzhang
84c4c7dc82 Use synchronize module 2018-09-16 20:36:44 +08:00
k8s-ci-robot
db11394711
Merge pull request #3200 from pablodav/feature/k8s_win_v1.11
Required support to start working on windows node support
2018-09-03 04:51:23 -07:00
Pablo Estigarribia
7cbe3c2171 ensure there is pin priority for docker package to avoid upgrade of docker to incompatible version
ensure there is pin priority for docker package to avoid upgrade of docker to incompatible version

remove empty when line

ensure there is pin priority for docker package to avoid upgrade of docker to incompatible version

force kubeadm upgrade due to failure without --force flag

ensure there is pin priority for docker package to avoid upgrade of docker to incompatible version

added nodeSelector to have compatibility with hybrid cluster with win nodes, also fix for download with missing container type

fixes in syntax and LF for newline in files

fix on yamllint check

ensure there is pin priority for docker package to avoid upgrade of docker to incompatible version

some cleanup for innecesary lines

remove conditions for nodeselector
2018-09-02 12:47:06 -03:00
rongzhang
2609ec0dc3 Fix copy etcd-ssl-ca failed 2018-08-31 15:06:03 +08:00
Takashi Okamoto
5eb805f098 Change timeout for kubeadm 600s.
* kubeadm timeout is too short and it may interrupt by timeout.
2018-08-28 04:51:38 +00:00
Takashi Okamoto
359009bb05 Download etcd and hyperkube binary. 2018-08-28 01:24:26 +00:00
Takashi Okamoto
6849788ebc Fix copy ca cert and ca key for kubeadm. 2018-08-28 01:24:25 +00:00
Erwan Miran
80cfeea957 psp, roles and rbs for PodSecurityPolicy when podsecuritypolicy_enabled is true 2018-08-22 18:16:13 +02:00
rongzhang
59176ebbb9 Add kubeadm controlplaneEndpoint
Nginx LB(default)
Other LB by kubeadm controlplane
2018-08-20 00:57:13 +08:00
Erwan Miran
58d4d65fab minor variable fix and reuse + handle auditlog redirected to stdout 2018-08-16 12:51:09 +02:00
rongzhang
2ffc1afe40 Support audit 2018-08-16 14:38:07 +08:00
Rong Zhang
a11e1eba9e Upgrade kubernetes to V1.11.x (#3078)
Upgrade Kubernetes to V1.11.2
The kubeadm configuration file version has been upgraded from v1alpha1 to v1alpha2
Add bootstrap kubeadm-config.yaml with external etcd
2018-08-14 15:13:44 +03:00
Aivars Sterns
72f053d9bb
Merge pull request #2972 from mattymo/force_cni_cp
Force copy cni files
2018-07-10 09:40:10 +03:00
Dao Hoang Son
d306c9708c Remove step that force disable kube_basic_auth.
The referenced issue (https://github.com/kubernetes/kubeadm/issues/441) has already been fixed.
2018-07-08 16:57:43 +07:00
Matthew Mosesohn
1a3b9dd864 Force copy cni files 2018-07-06 16:39:42 +03:00
Christopher J. Ruwe
73800ef111 make certificates non-executable 2018-05-15 07:54:32 +00:00
Miouge1
ad48606e4e Restart scheduler when policy changes 2018-05-14 10:09:30 +02:00
Chad Swenson
d87b6fd9f3 Use dedicated front-proxy-ca for front-proxy-client 2018-04-12 11:03:22 -05:00
Andreas Krüger
d9418b1dc4
Merge pull request #2554 from georgejdli/fix-sa-token-signing
Fix kubespray's ServiceAccount token signing keys
2018-03-31 09:59:22 +02:00
georgejdli
572ab650db copy dedicated service account token signing key for kubeadm migration 2018-03-30 13:03:32 -05:00
Kuldip Madnani
daeeae1a91 Added retries in pre-upgrade.yml and retries while applying kube-dns.yml (#2553)
* Added retries in pre-upgrade.yml and retries while applying kube-dns.yml

* Removed trailing spaces
2018-03-29 11:37:32 -05:00
Andreas Krüger
3d6fd49179 Added option for encrypting secrets to etcd v.2 (#2428)
* Added option for encrypting secrets to etcd

* Fix keylength to 32

* Forgot the default

* Rename secrets.yaml to secrets_encryption.yaml

* Fix static path for secrets file to use ansible variable

* Rename secrets.yaml.j2 to secrets_encryption.yaml.j2

* Base64 encode the token

* Fixed merge error

* Changed path to credentials dir

* Update path to secrets file which is now readable inside the apiserver container. Set better file permissions

* Add encryption option to k8s-cluster.yml
2018-03-15 22:20:05 +03:00
Andreas Krüger
d84ff06f73 Set filemode to 0640 (#2315)
* Set filemode to 0640

weave-net.yml file is readable by all users on the host. It however contains the weave_password to encrypt all pod communication. It should only be readable by root.

* Set mode 0640 on users_file with basic auth
2018-02-21 23:13:46 +03:00
Matthew Mosesohn
dc6a17e092
Use include/import tasks (#2192)
import_tasks will consume far less memory, so it should be
used whenever it is compatible.
2018-01-29 14:37:48 +03:00
Chad Swenson
c6e0fcea31
Merge pull request #1948 from sgmitchell/secured-etcd
Enable etcd secure client to prevent etcdctl access without cert and key
2018-01-25 09:35:51 -06:00
Brad Beam
0c8bed21ee
Merge pull request #2019 from chadswen/disable-api-insecure-port
Support for disabling apiserver insecure port (the sequel)
2018-01-24 19:58:53 -06:00
Brad Beam
98300e3165
Merge pull request #2155 from brutus333/fix/pvc
Fix for Issue #2141
2018-01-24 16:15:33 -06:00
Virgil Chereches
a4d142368b Renamed variable from disable_volume_zone_conflict to volume_cross_zone_attachment and removed cloud provider condition; fix identation 2018-01-23 13:14:00 +00:00
Virgil Chereches
c87bb2f239 Fix for Issue #2141 2018-01-12 07:07:02 +00:00
Steve Mitchell
e45b30d033 Add etcd key and cert environment variables for use with client auth 2018-01-02 13:52:17 -05:00
Matthew Mosesohn
ad6fecefa8
Update Kubernetes to v1.9.0 (#2100)
Update checksum for kubeadm
Use v1.9.0 kubeadm params
Include hash of ca.crt for kubeadm join
Update tag for testing upgrades
Add workaround for testing upgrades
Remove scale CI scenarios because of slow inventory parsing
in ansible 2.4.x.

Change region for tests to us-central1 to
improve ansible performance
2017-12-25 08:57:45 +00:00
Chad Swenson
b8788421d5 Support for disabling apiserver insecure port
This allows `kube_apiserver_insecure_port` to be set to 0 (disabled).

Rework of #1937 with kubeadm support

Also, fixed an issue in `kubeadm-migrate-certs` where the old apiserver cert was copied as the kubeadm key
2017-12-05 09:13:45 -06:00
Spencer Smith
43600ffcf8
Merge pull request #1972 from chadswen/master-static-pod-flush
Additional flush for static pod master upgrade
2017-11-28 11:36:38 -05:00
Bogdan Dobrelya
8aafe64397
Defaults for apiserver_loadbalancer_domain_name (#1993)
* Defaults for apiserver_loadbalancer_domain_name

When loadbalancer_apiserver is defined, use the
apiserver_loadbalancer_domain_name with a given default value.

Fix unconsistencies for checking if apiserver_loadbalancer_domain_name
is defined AND using it with a default value provided at once.

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>

* Define defaults for LB modes in common defaults

Adjust the defaults for apiserver_loadbalancer_domain_name and
loadbalancer_apiserver_localhost to come from a single source, which is
kubespray-defaults. Removes some confusion and simplefies the code.

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
2017-11-23 16:15:48 +00:00
Chad Swenson
1f99710b21 Additional flush for static pod master upgrade
Thought this wasn't required at first but I forgot there's no auto flush at the end of these tasks since the `kubernetes/master` role is not the end of the play.
2017-11-13 18:11:57 -06:00
Chad Swenson
e9f795c5ce Master component and kubelet container upgrade fixes
* Fixes an issue where apiserver and friends (controller manager, scheduler) were prevented from restarting after manifests/secrets are changed. This occurred when a replaced kubelet doesn't reconcile new master manifests, which caused old master component versions to linger during deployment. In my case this was causing upgrades from k8s 1.6/1.7 -> k8s 1.8 to fail
* Improves transitions from kubelet container to host kubelet by preventing issues where kubelet container reappeared during the deployment
2017-11-08 01:40:33 -06:00
Matthew Mosesohn
ec53b8b66a Move cluster roles and system namespace to new role
This should be done after kubeconfig is set for admin and
before network plugins are up.
2017-10-26 14:36:05 +01:00
Matthew Mosesohn
86fb669fd3 Idempotency fixes (#1838) 2017-10-25 21:19:40 +01:00
Matthew Mosesohn
0b4fcc83bd Fix up warnings and deprecations (#1848) 2017-10-20 08:25:57 +01:00
Matthew Mosesohn
16462292e1 Properly skip extra SANs when not specified for kubeadm (#1831) 2017-10-18 12:04:13 +01:00
Rémi de Passmoilesel
356515222a Add possibility to insert more ip adresses in certificates (#1678)
* Add possibility to insert more ip adresses in certificates

* Add newline at end of files

* Move supp ip parameters to k8s-cluster group file

* Add supplementary addresses in kubeadm master role

* Improve openssl indexes
2017-10-17 11:06:07 +01:00
Matthew Mosesohn
7e4668859b Change file used to check kubeadm upgrade method (#1784)
* Change file used to check kubeadm upgrade method

Test for ca.crt instead of admin.conf because admin.conf
is created during normal deployment.

* more fixes for upgrade
2017-10-15 10:33:22 +01:00
Aivars Sterns
9c86da1403 Normalize tags in all places to prepare for tag fixing in future (#1739) 2017-10-05 08:43:04 +01:00
Matthew Mosesohn
3ff5f40bdb fix graceful upgrade (#1704)
Fix system namespace creation
Only rotate tokens when necessary
2017-09-27 14:49:20 +01:00
Matthew Mosesohn
689ded0413 Enable kubeadm upgrades to any version (#1709) 2017-09-27 14:48:18 +01:00
Matthew Mosesohn
327ed157ef Verify valid settings before deploy (#1705)
Also fix yaml lint issues

Fixes #1703
2017-09-27 14:47:47 +01:00
tanshanshan
477afa8711 when and run_once are reduplicative (#1694) 2017-09-26 14:48:05 +01:00
Matthew Mosesohn
bd272e0b3c Upgrade to kubeadm (#1667)
* Enable upgrade to kubeadm

* fix kubedns upgrade

* try upgrade route

* use init/upgrade strategy for kubeadm and ignore kubedns svc

* Use bin_dir for kubeadm

* delete more secrets

* fix waiting for terminating pods

* Manually enforce kube-proxy for kubeadm deploy

* remove proxy. update to kubeadm 1.8.0rc1
2017-09-26 10:38:58 +01:00
Matthew Mosesohn
a1cde03b20 Correct master manifest cleanup logic (#1693)
Fixes #1666
2017-09-25 12:19:04 +01:00
Matthew Mosesohn
188bae142b Fix wait for hosts in CI (#1679)
Also fix usage of failed_when and handling exit code.
2017-09-20 14:30:09 +01:00