Commit graph

367 commits

Author SHA1 Message Date
Wong Hoi Sing Edison
1a1d154e14 Support multiple inventory files under individual inventory directory 2018-02-08 08:08:15 +08:00
Spencer Smith
1869aa3985 remove references to deprecated kubespray cli 2018-02-05 10:00:21 -05:00
Julian Hübenthal
7f79210ed1 reworked vsphere-cloud-config template 2018-01-31 16:51:23 +01:00
Julian Hübenthal
fc29764911 fixed broken variables table 2018-01-31 09:27:45 +01:00
Julian Hübenthal
989e9174c2 Added vSphere cloud provider config update for Kubernetes >= 1.9.2 2018-01-31 09:15:46 +01:00
Miouge1
240d4193ae Update information about network sizes 2018-01-26 15:23:21 +01:00
Matthew Mosesohn
bf1411060e Add optional manual dns_mode (#2178) 2018-01-23 14:28:42 +01:00
Bogdan Dobrelya
278ac08087 Fix HA docs API access endpoints explained (#2126)
* Fix HA docs API access endpoints explained

Follow-up commit 81347298a3
and fix the endpoint value provided in HA docs.

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>

* Clarify internal LB with external LB use case

* Clarify how to use both internal and external, non-cluster aware and
  not managed with Kubespray, LB solutions.
* Clarify the requirements, like TLS/SSL termination, for such an external LB.
  Unlike to the 'cluster-aware' external LB config, endpoints' security must be
  managed by that non-cluster aware external LB.
* Note that masters always contact their local apiservers via https://bip:sp.
  It's highly unlikely to go down and it reduces latency that might be
  introduced when going host->lb->host. Only computes go that path.

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>

* Add a note for supplementary_addresses_in_ssl_keys

Explain how to benefit from supplementary_addresses_in_ssl_keys

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
2018-01-09 16:01:50 +03:00
Aaron Crickenberger
10a5273f07 Follow CLA doc to kubernetes/community 2018-01-03 16:48:53 -08:00
Bogdan Dobrelya
bac3bf1a5f
Fix auto-evaluated API access endpoint for bind IP (#2086)
Auto configure API access endpoint with a custom bind IP, if provided.
Fix HA docs' http URLs are https in fact, clarify the insecure vs secure
API access modes as well.

Closes: #issues/2051

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
2018-01-03 17:40:21 +01:00
Brad Beam
c2347db934
Merge pull request #1953 from chadswen/dashboard-refactor
Kubernetes Dashboard v1.7.1 Refactor
2017-12-05 08:50:55 -06:00
Steven Hardy
d39a88d63f Allow setting --bind-address for apiserver hyperkube (#1985)
* Allow setting --bind-address for apiserver hyperkube

This is required if you wish to configure a loadbalancer (e.g haproxy)
running on the master nodes without choosing a different port for the
vip from that used by the API - in this case you need the API to bind to
a specific interface, then haproxy can bind the same port on the VIP:

root@overcloud-controller-0 ~]# netstat -taupen | grep 6443
tcp        0      0 192.168.24.6:6443       0.0.0.0:*               LISTEN      0          680613     134504/haproxy
tcp        0      0 192.168.24.16:6443      0.0.0.0:*               LISTEN      0          653329     131423/hyperkube
tcp        0      0 192.168.24.16:6443      192.168.24.16:58404     ESTABLISHED 0          652991     131423/hyperkube
tcp        0      0 192.168.24.16:58404     192.168.24.16:6443      ESTABLISHED 0          652986     131423/hyperkube

This can be achieved e.g via:

kube_apiserver_bind_address: 192.168.24.16

* Address code review feedback

* Update kube-apiserver.manifest.j2
2017-11-29 15:24:02 +00:00
unclejack
e5d353d0a7 contiv network support (#1914)
* Add Contiv support

Contiv is a network plugin for Kubernetes and Docker. It supports
vlan/vxlan/BGP/Cisco ACI technologies. It support firewall policies,
multiple networks and bridging pods onto physical networks.

* Update contiv version to 1.1.4

Update contiv version to 1.1.4 and added SVC_SUBNET in contiv-config.

* Load openvswitch module to workaround on CentOS7.4

* Set contiv cni version to 0.1.0

Correct contiv CNI version to 0.1.0.

* Use kube_apiserver_endpoint for K8S_API_SERVER

Use kube_apiserver_endpoint as K8S_API_SERVER to make contiv talks
to a available endpoint no matter if there's a loadbalancer or not.

* Make contiv use its own etcd

Before this commit, contiv is using a etcd proxy mode to k8s etcd,
this work fine when the etcd hosts are co-located with contiv etcd
proxy, however the k8s peering certs are only in etcd group, as a
result the etcd-proxy is not able to peering with the k8s etcd on
etcd group, plus the netplugin is always trying to find the etcd
endpoint on localhost, this will cause problem for all netplugins
not runnign on etcd group nodes.
This commit make contiv uses its own etcd, separate from k8s one.
on kube-master nodes (where net-master runs), it will run as leader
mode and on all rest nodes it will run as proxy mode.

* Use cp instead of rsync to copy cni binaries

Since rsync has been removed from hyperkube, this commit changes it
to use cp instead.

* Make contiv-etcd able to run on master nodes

* Add rbac_enabled flag for contiv pods

* Add contiv into CNI network plugin lists

* migrate contiv test to tests/files

Signed-off-by: Cristian Staretu <cristian.staretu@gmail.com>

* Add required rules for contiv netplugin

* Better handling json return of fwdMode

* Make contiv etcd port configurable

* Use default var instead of templating

* roles/download/defaults/main.yml: use contiv 1.1.7

Signed-off-by: Cristian Staretu <cristian.staretu@gmail.com>
2017-11-29 14:24:16 +00:00
Bogdan Dobrelya
8aafe64397
Defaults for apiserver_loadbalancer_domain_name (#1993)
* Defaults for apiserver_loadbalancer_domain_name

When loadbalancer_apiserver is defined, use the
apiserver_loadbalancer_domain_name with a given default value.

Fix unconsistencies for checking if apiserver_loadbalancer_domain_name
is defined AND using it with a default value provided at once.

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>

* Define defaults for LB modes in common defaults

Adjust the defaults for apiserver_loadbalancer_domain_name and
loadbalancer_apiserver_localhost to come from a single source, which is
kubespray-defaults. Removes some confusion and simplefies the code.

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
2017-11-23 16:15:48 +00:00
Chad Swenson
0c6f172e75 Kubernetes Dashboard v1.7.1 Refactor
This version required changing the previous access model for dashboard completely but it's a change for the better. Docs were updated.

* New login/auth options that use apiserver auth proxying by default
* Requires RBAC in `authorization_modes`
* Only serves over https
* No longer available at https://first_master:6443/ui until apiserver is updated with the https proxy URL:
* Can access from https://first_master:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login you will be prompted for credentials
* Or you can run 'kubectl proxy' from your local machine to access dashboard in your browser from: http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
* It is recommended to access dashboard from behind a gateway that enforces an authentication token, details and other access options here: https://github.com/kubernetes/dashboard/wiki/Accessing-Dashboard---1.7.X-and-above
2017-11-15 10:05:48 -06:00
abelgana
56047c1c83
fix a typo 2017-11-10 09:30:27 -05:00
Amit Kumar Jaiswal
125267544e Fix Typo (#1935) 2017-11-06 13:51:22 +00:00
Günther Grill
a2bda9e5f1 Eliminate jinja2 template expression warning and rename coreos-python var (#1911)
* Change deprecated vagrant ansible flag 'sudo' to 'become'

* Emphasize, that the name of the pip_pyton_modules is only considered in coreos

* Remove useless unused variable

* Fix warning when jinja2 template-delimiters used in when statement

There is no need for jinja2 template-delimiters like {{ }} or {% %}
any more. They can just be omitted as described in https://github.com/ansible/ansible/issues/22397

* Fix broken link in getting-started guide
2017-11-03 07:11:36 +00:00
Fernando Ripoll
257280a050
Typo in apt-get command
Typo in apt-get command
2017-11-02 11:40:08 +01:00
Matthew Mosesohn
c0e989b17c
New addon: local_volume_provisioner (#1909) 2017-11-01 14:25:35 +00:00
Fernando Ripoll
160e479f8d
Typo in the apt-get command
Typo in the apt-get command
2017-10-30 13:47:39 +01:00
Matthew Mosesohn
a9f4038fcd Update roadmap (#1814) 2017-10-16 17:02:53 +01:00
neith00
77f1d4b0f1 Revert "Update roadmap" (#1809)
* Revert "Debian jessie docs (#1806)"

This reverts commit d78577c810.

* Revert "[contrib/network-storage/glusterfs] adds service for glusterfs endpoint (#1800)"

This reverts commit 5fb6b2eaf7.

* Revert "[contrib/network-storage/glusterfs] bootstrap for glusterfs nodes (#1799)"

This reverts commit 404caa111a.

* Revert "Fixed kubelet standard log environment (#1780)"

This reverts commit b838468500.

* Revert "Add support for fedora atomic host (#1779)"

This reverts commit f2235be1d3.

* Revert "Update network-plugins to use portmap plugin (#1763)"

This reverts commit 6ec45b10f1.

* Revert "Update roadmap (#1795)"

This reverts commit d9879d8026.
2017-10-16 14:09:24 +01:00
Marc Zahn
d78577c810 Debian jessie docs (#1806)
* Add Debian Jessie notes

* Add installation notes for Debian Jessie
2017-10-16 09:02:12 +01:00
Matthew Mosesohn
d9879d8026 Update roadmap (#1795) 2017-10-16 07:06:06 +01:00
Matthew Mosesohn
d487b2f927 Security best practice fixes (#1783)
* Disable basic and token auth by default

* Add recommended security params

* allow basic auth to fail in tests

* Enable TLS authentication for kubelet
2017-10-15 20:41:17 +01:00
Matthew Mosesohn
92d038062e Fix node authorization for cloudprovider installs (#1794)
In 1.8, the Node authorization mode should be listed first to
allow kubelet to access secrets. This seems to only impact
environments with cloudprovider enabled.
2017-10-14 11:28:46 +01:00
Vijay Katam
27ed73e3e3 Rename dns_server, add var for selinux. (#1572)
* Rename dns_server to dnsmasq_dns_server so that it includes role prefix
as the var name is generic and conflicts when integrating with existing ansible automation.
*  Enable selinux state to be configurable with new var preinstall_selinux_state
2017-10-11 20:40:21 +01:00
Matthew Mosesohn
eb0dcf6063 Improve proxy (#1771)
* Set no_proxy to all local ips

* Use proxy settings on all necessary tasks
2017-10-11 19:47:27 +01:00
Matthew Mosesohn
f14f04c5ea Upgrade to kubernetes v1.8.0 (#1730)
* Upgrade to kubernetes v1.8.0

hyperkube no longer contains rsync, so now use cp

* Enable node authorization mode

* change kube-proxy cert group name
2017-10-05 10:51:21 +01:00
Maxim Krasilnikov
da61b8e7c9 Added workaround for vagrant 1.9 and centos vm box (#1738) 2017-10-03 11:32:19 +01:00
shiftky
a927ed2da4 Improve playbook example of integration document 2017-09-29 18:00:01 +09:00
Matthew Mosesohn
bd272e0b3c Upgrade to kubeadm (#1667)
* Enable upgrade to kubeadm

* fix kubedns upgrade

* try upgrade route

* use init/upgrade strategy for kubeadm and ignore kubedns svc

* Use bin_dir for kubeadm

* delete more secrets

* fix waiting for terminating pods

* Manually enforce kube-proxy for kubeadm deploy

* remove proxy. update to kubeadm 1.8.0rc1
2017-09-26 10:38:58 +01:00
Maxim Krasilnikov
bc15ceaba1 Update var doc about users accounts (#1685) 2017-09-25 12:20:00 +01:00
Junaid Ali
6f17d0817b Updating getting-started.md (#1683)
Signed-off-by: Junaid Ali <junaidali.yahya@gmail.com>
2017-09-25 12:19:38 +01:00
Jiri Stransky
70d0235770 Set correct kubelet cgroup-driver also for kubeadm deployments
This follows pull request #1677, adding the cgroup-driver
autodetection also for kubeadm way of deploying.

Info about this and the possibility to override is added to the docs.
2017-09-22 13:19:04 +02:00
Matthew Mosesohn
ef8e35e39b Create admin credential kubeconfig (#1647)
New files: /etc/kubernetes/admin.conf
           /root/.kube/config
           $GITDIR/artifacts/{kubectl,admin.conf}

Optional method to download kubectl and admin.conf if
kubeconfig_lcoalhost is set to true (default false)
2017-09-18 13:30:57 +01:00
Matthew Mosesohn
943aaf84e5 Update getting-started.md 2017-09-11 12:47:04 +03:00
Matthew Mosesohn
9fa1873a65 Add kube dashboard, enabled by default (#1643)
* Add kube dashboard, enabled by default

Also add rbac role for kube user

* Update main.yml
2017-09-09 23:38:03 +03:00
Matthew Mosesohn
7117614ee5 Use a generated password for kube user (#1624)
Removed unnecessary root user
2017-09-06 20:20:25 +03:00
Maxim Krasilnikov
6eb22c5db2 Change single Vault pki mount to multi pki mounts paths for etcd and kube CA`s (#1552)
* Added update CA trust step for etcd and kube/secrets roles

* Added load_balancer_domain_name to certificate alt names if defined. Reset CA's in RedHat os.

* Rename kube-cluster-ca.crt to vault-ca.crt, we need separated CA`s for vault, etcd and kube.

* Vault role refactoring, remove optional cert vault auth because not not used and worked. Create separate CA`s fro vault and etcd.

* Fixed different certificates set for vault cert_managment

* Update doc/vault.md

* Fixed condition create vault CA, wrong group

* Fixed missing etcd_cert_path mount for rkt deployment type. Distribute vault roles for all vault hosts

* Removed wrong when condition in create etcd role vault tasks.
2017-08-30 16:03:22 +03:00
Chad Swenson
a39e78d42d Initial version of Flannel using CNI (#1486)
* Updates Controller Manager/Kubelet with Flannel's required configuration for CNI
* Removes old Flannel installation
* Install CNI enabled Flannel DaemonSet/ConfigMap/CNI bins and config (with portmap plugin) on host
* Uses RBAC if enabled
* Fixed an issue that could occur if br_netfilter is not a module and net.bridge.bridge-nf-call-iptables sysctl was not set
2017-08-25 10:07:50 +03:00
Hassan Zamani
01ce09f343 Add feature_gates var for customizing Kubernetes feature gates (#1520) 2017-08-24 23:18:38 +03:00
Matthew Mosesohn
277fa6c12d Add node to docs about kubelet deployment type changes 2017-08-21 09:13:59 +01:00
Vijay Katam
c92506e2e7 Add calico variable that enables ignoring Kernel's RPF Setting (#1493) 2017-08-20 14:01:09 +03:00
Joseph Heck
bc5159a1f5 Update comparisons.md (#1519)
Minor grammar fixes
2017-08-14 18:48:35 +03:00
Brad Beam
ca6535f210 Merge pull request #1488 from timtoum/weave_docs
added Weave documentation
2017-08-10 08:26:19 -05:00
Brad Beam
383d582b47 Merge pull request #1382 from jwfang/rbac
basic rbac support
2017-08-07 08:01:51 -05:00
timtoum
b1a5bb593c update docs 2017-08-01 15:55:38 +02:00
timtoum
9369c6549a update docs 2017-08-01 14:30:12 +02:00
email
c7731a3b93 update docs 2017-08-01 14:24:19 +02:00
email
24706c163a update docs 2017-08-01 14:12:21 +02:00
email
a276dc47e0 update docs 2017-08-01 10:52:21 +02:00
email
5de7896ffb update docs 2017-07-31 13:28:47 +02:00
email
01af45d14a update docs 2017-07-31 13:23:01 +02:00
email
87cdb81fae update docs 2017-07-28 11:33:13 +02:00
email
74403f2003 update docs 2017-07-27 17:00:54 +02:00
email
2c21672de6 update docs 2017-07-27 15:10:08 +02:00
email
f7dc21773d new doc for weave 2017-07-27 14:40:52 +02:00
jwfang
805d9f22ce note upgrade from non-RBAC not supported 2017-07-24 19:11:41 +08:00
Brad Beam
45845d4a2a Merge pull request #1437 from rajiteh/fix_aws_docs
Add more instructions to setting up AWS provider
2017-07-18 16:43:01 -05:00
John Ko
06b219217b fix some typos in HA doc 2017-07-18 10:44:08 -04:00
jwfang
a5b84a47b0 docs: experimental, no calico/vault 2017-07-17 19:29:59 +08:00
jwfang
552b2f0635 change authorization_modes default value 2017-07-17 19:29:59 +08:00
jwfang
092bf07cbf basic rbac support 2017-07-17 19:29:59 +08:00
nico
f4a3b31415 add vsphere cloud provider doc
fix typo
2017-07-12 11:01:06 +02:00
Raj Perera
5c7e309d13 Add more instructions to setting up AWS provider 2017-07-11 10:53:19 -04:00
Spencer Smith
e98b0371e5 Merge pull request #1368 from vgkowski/patch-3
change documentation from "self hosted" to "static pod" for the contr…
2017-06-30 07:31:52 -04:00
Spencer Smith
cf8c74cb07 Merge pull request #1342 from Abdelsalam-Abbas/patch-1
Create ansible.md
2017-06-27 13:58:18 -04:00
Spencer Smith
23565ebe62 Merge pull request #1356 from rsmitty/rename
Rename project to kubespray
2017-06-27 11:40:03 -04:00
Spencer Smith
83265b7f75 renaming kargo-cli to kubespray-cli 2017-06-23 12:35:10 -04:00
Brad Beam
5364a10033 Merge pull request #1374 from Lendico/doc_ansible_integration
Flow for intergation with existing ansible repo
2017-06-23 11:31:22 -05:00
Spencer Smith
bae5ce0bfa Merge branch 'master' into rename 2017-06-23 12:23:51 -04:00
Anton Nerozya
0cd83eadc0 README: Integration with existing ansible repo 2017-06-22 18:58:10 +02:00
vgkowski
d85f98d2a9 change documentation from "self hosted" to "static pod" for the control plane 2017-06-21 11:00:11 +02:00
TAKAHASHI Yuto
9e123011c2 Modify documented neutron commands for Calico setup 2017-06-21 15:11:39 +09:00
vgkowski
bb6f727f25 Update openstack documentation with Calico
Linked to the issue https://github.com/kubernetes-incubator/kubespray/issues/1359
2017-06-19 15:48:34 +02:00
Spencer Smith
8203383c03 rename almost all mentions of kargo 2017-06-16 13:25:46 -04:00
Abdelsalam Abbas
67eeccb31f Create ansible.md
fixing a typo
2017-06-12 13:20:15 +02:00
Kevin Jing Qiu
6d8a415b4d Update doc on Vagrant local override file 2017-06-02 20:09:37 -04:00
Kevin Jing Qiu
e7acc2fddf Update doc for Vagrant install 2017-06-02 19:03:43 -04:00
Spencer Smith
18a42e4b38 add scale.yml to do minimum needed for a node bootstrap 2017-05-24 15:49:21 -04:00
Spencer Smith
83f44b1ac1 Added example json 2017-05-18 17:57:30 -04:00
Spencer Smith
1f470eadd1 Added dynamic inventory for AWS as contrib 2017-05-18 17:52:44 -04:00
Charles Farquhar
d310acc1eb Fix link from ansible.md to calico.md
This commit fixes a broken link from ansible.md to calico.md.
2017-04-28 12:10:23 -05:00
Spencer Smith
5a7c50027f add some known tweaks that need to be made for coreos 2017-04-20 11:14:41 -04:00
Justin
9503434d53 Fix IPS array variable expansion
$IPS only expands to the first ip address in the array:

justin@box:~$ declare -a IPS=(10.10.1.3 10.10.1.4 10.10.1.5)
justin@box:~$ echo $IPS
10.10.1.3
justin@box:~$ echo ${IPS[@]}
10.10.1.3 10.10.1.4 10.10.1.5
2017-04-17 20:56:52 -04:00
Spencer Smith
c3c9e955e5 Merge pull request #1232 from rsmitty/custom-flags
add ability for custom flags
2017-04-17 14:01:32 -04:00
Spencer Smith
94596388f7 add ability for custom flags 2017-04-14 17:33:04 -04:00
Justin Downing
fbded9cdac Update upgrades.md
Clarify that the `kube_version` environment variable is needed for the CLI "graceful upgrade". Also add and example to check that the upgrade was successful.
2017-03-29 22:00:52 -04:00
Antoine Legrand
ac96d5ccf0 Merge pull request #1176 from zoidbergwill/patch-2
Update roadmap.md
2017-03-23 12:05:35 +01:00
Vladimir Rutsky
c4e57477fb replace non-breakable space with regular space
Non-brekable space is 0xc2 0xa0 byte sequence in UTF-8.

To find one:

    $ git grep -I -P '\xc2\xa0'

To replace with regular space:

    $ git grep -l -I -P '\xc2\xa0' | xargs sed -i 's/\xc2\xa0/ /g'

This commit doesn't include changes that will overlap with commit f1c59a91a1.
2017-03-23 00:25:01 +03:00
William Martin Stewart
f1c59a91a1 Update roadmap.md 2017-03-22 22:03:06 +02:00
Vincent Schwarzer
c8c6105ee2 Fixed Formatting / Ansbile-Playbook Command
- added -b and fixed typo in ansible-playbook command 
- fixed formatting issue
2017-03-16 17:53:48 +01:00
Vincent Schwarzer
3e8386cbf3 Fixed CoreOS Docu
CoreOS docu was referencing outdated bootstrap playbook that
is now part of kargo itself.
2017-03-15 13:04:01 +01:00
Matthew Mosesohn
a244aca6a4 Merge pull request #1113 from VincentS/AWS_IAM_PROFILES
Added Missing AWS IAM Profiles and Policies
2017-03-03 17:35:55 +03:00
Vincent Schwarzer
5ae85b9de5 Added Missing AWS IAM Profiles and Policies
The AWS IAM profiles and policies required to run Kargo on AWS
are no longer hosted in the kubernetes main repo since kube-up got
deprecated. Hence we have to move the files into the kargo repository.
2017-03-03 15:30:07 +01:00
Bogdan Dobrelya
aeec0f9a71 Merge pull request #1071 from vijaykatam/atomic_host
Add support for atomic host
2017-03-03 13:03:59 +01:00
Vijay Katam
a0b1eda1d0 Add support for atomic host
Updates based on feedback

Simplify checks for file exists

remove invalid char

Review feedback. Use regular systemd file.

Add template for docker systemd atomic
2017-03-01 09:38:19 -08:00
Vladimir Rutsky
ad80e09ac5 fix inline verbatim blocks formatting in markdown 2017-03-01 17:50:28 +04:00
Bogdan Dobrelya
0c66418dad Merge pull request #1090 from artem-panchenko/calicoAcceptHostEndpointConnections
Allow connections from pods to local endpoints
2017-03-01 13:37:05 +01:00
Artem Panchenko
fa05d15093 Allow connections from pods to local endpoints
By default Calico blocks traffic from endpoints
to the host itself by using an iptables DROP
action. It could lead to a situation when service
has one alive endpoint, but pods which run on
the same node can not access it. Changed the action
to RETURN.
2017-03-01 09:21:02 +02:00
Sergii Golovatiuk
d31c040dc0 Change kube-api default port from 443 to 6443
Operator can specify any port for kube-api (6443 default) This helps in
case where some pods such as Ingress require 443 exclusively.

Closes: 820
Signed-off-by: Sergii Golovatiuk <sgolovatiuk@mirantis.com>
2017-02-28 15:45:35 +01:00
Bogdan Dobrelya
069606947c Merge pull request #1063 from bogdando/fix
Align LB defaults with the HA docs
2017-02-27 10:14:42 +01:00
Kopylov German
d197ce230f Fix cluster.yml file extension in docs 2017-02-26 13:42:52 +03:00
Antoine Legrand
5f7607412b Add default var role 2017-02-23 12:07:17 +01:00
Bogdan Dobrelya
f2a4619c57 Align LB defaults with the HA docs
Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
2017-02-23 10:32:44 +01:00
Bogdan Dobrelya
712872efba Rework inventory all by real groups' vars
* Leave all.yml to keep only optional vars
* Store groups' specific vars by existing group names
* Fix optional vars casted as mandatory (add default())
* Fix missing defaults for an optional IP var
* Relink group_vars for terraform to reflect changes

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
2017-02-23 09:43:42 +01:00
Matthew Mosesohn
d821448e2f Merge branch 'master' into synthscale 2017-02-21 22:17:43 +03:00
Matthew Mosesohn
634e6a381c Merge pull request #1043 from rutsky/patch-3
fix typos in azure docs
2017-02-20 20:24:05 +03:00
Matthew Mosesohn
ce4eefff6a Use first kube-master to check results 2017-02-18 14:11:51 +04:00
Matthew Mosesohn
9c1701f2aa Add synthetic scale deployment mode
New deploy modes: scale, ha-scale, separate-scale
Creates 200 fake hosts for deployment with fake hostvars.

Useful for testing certificate generation and propagation to other
master nodes.

Updated test cases descriptions.
2017-02-18 14:09:55 +04:00
Antoine Legrand
b84cc14694 Merge pull request #1029 from mattymo/graceful
Add graceful upgrade process
2017-02-17 21:24:32 +01:00
Vladimir Rutsky
a84175b3b9 fix typo: "infrastructore" 2017-02-17 23:27:38 +04:00
Vladimir Rutsky
438b4e9625 fix typos in azure docs 2017-02-17 21:39:22 +04:00
Matthew Mosesohn
97ebbb9672 Add graceful upgrade process
Based on #718 introduced by rsmitty.

Includes all roles and all options to support deployment of
new hosts in case they were added to inventory.

Main difference here is that master role is evaluated first
so that master components get upgraded first.

Fixes #694
2017-02-16 17:18:38 +03:00
Matthew Mosesohn
2d65554cb9 Change default network plugin to Calico 2017-02-15 16:15:22 +03:00
Hung Nguyen Viet
d8f46c4410 Highlight important action 2017-02-14 17:18:25 +07:00
Hung Nguyen Viet
d0757ccc5e Fix typo 2017-02-14 17:18:22 +07:00
Matthew Mosesohn
779f20d64e Merge pull request #1010 from bogdando/fixes
Fix misleading HA docs
2017-02-10 13:01:29 +03:00
Bogdan Dobrelya
ed1ab11001 Fix misleading HA docs
Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
2017-02-10 10:28:27 +01:00
Matthew Mosesohn
2f88c9eefe Merge pull request #989 from holser/kubelet_remedy
Kubernetes Reliability Improvements
2017-02-10 09:29:29 +03:00
Sergii Golovatiuk
c07d60bc90 Kubernetes Reliability Improvements
- Exclude kubelet CPU/RAM (kube-reserved) from cgroup. It decreases a
  chance of overcommitment
- Add a possibility to modify Kubelet node-status-update-frequency
- Add a posibility to configure node-monitor-grace-period,
  node-monitor-period, pod-eviction-timeout for Kubernetes controller
  manager
- Add Kubernetes Relaibility Documentation with recomendations for
  various scenarios.

Signed-off-by: Sergii Golovatiuk <sgolovatiuk@mirantis.com>
2017-02-09 23:54:08 +01:00
Greg Althaus
fcd78eb1f7 Due to the nsenter and other reworks, it appears that
kubelet lost the ability to load kernel modules.  This
puts that back by adding the lib/modules mount to kubelet.

The new variable kubelet_load_modules can be set to true
to enable this item.  It is OFF by default.
2017-02-09 10:02:26 -06:00
Josh Conant
245e05ce61 Vault security hardening and role isolation 2017-02-08 21:41:36 +00:00
Aleksandr Didenko
54af533b31 Update playbooks to support new netchecker
Netchecker is rewritten in Go lang with some new args instead of
env variables. Also netchecker-server no longer requires kubectl
container. Updating playbooks accordingly.
2017-02-07 15:20:34 +01:00
Matthew Mosesohn
fd30131dc2 Revert "Drop linux capabilities and rework users/groups" 2017-02-06 15:58:54 +03:00
Bogdan Dobrelya
b7bf502e02 Merge pull request #978 from rutsky/patch-1
remove extra `~`
2017-02-06 12:07:54 +01:00
Bogdan Dobrelya
cae2982d81 Merge pull request #911 from bogdando/DROP_CAPS
Drop linux capabilities and rework users/groups
2017-02-06 12:05:51 +01:00
Vladimir Rutsky
b638c89556 remove extra ~ 2017-02-06 15:05:24 +04:00
Antoine Legrand
d818ac1d59 Update roadmap.md 2017-02-04 23:23:24 +01:00
Bogdan Dobrelya
cd25bfca91 Merge pull request #884 from mattymo/inventory_builder_scale
Add scale thresholds to split etcd and k8s-masters
2017-01-20 09:34:45 +01:00
Bogdan Dobrelya
cb2e5ac776 Drop linux capabilities and rework users/groups
* Drop linux capabilities for unprivileged containerized
  worlkoads Kargo configures for deployments.
* Configure required securityContext/user/group/groups for kube
  components' static manifests, etcd, calico-rr and k8s apps,
  like dnsmasq daemonset.
* Rework cloud-init (etcd) users creation for CoreOS.
* Fix nologin paths, adjust defaults for addusers role and ensure
  supplementary groups membership added for users.
* Add netplug user for network plugins (yet unused by privileged
  networking containers though).
* Grant the kube and netplug users read access for etcd certs via
  the etcd certs group.
* Grant group read access to kube certs via the kube cert group.
* Remove priveleged mode for calico-rr and run it under its uid/gid
  and supplementary etcd_cert group.
* Adjust docs.
* Align cpu/memory limits and dropped caps with added rkt support
  for control plane.

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
2017-01-20 08:50:42 +01:00
Matthew Mosesohn
f742fc3dd1 Add scale thresholds to split etcd and k8s-masters
Also adds calico-rr group if there are standalone etcd nodes.
Now if there are 50 or more nodes, 3 etcd nodes will be standalone.
If there are 200 or more nodes, 2 kube-masters will be standalone.
If thresholds are exceeded, kube-node group cannot add nodes that
belong to etcd or kube-master groups (according to above statements).
2017-01-19 17:30:56 +03:00
Greg Althaus
0022a2b29e Add doc updates. 2017-01-17 13:15:48 -06:00
Bogdan Dobrelya
caab0cdf27 Docs updates
Fix mismatching inventory examples.
Add command examples.
Clarify groups use cases.

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
2017-01-11 15:39:35 +01:00
Bogdan Dobrelya
6fb6947feb Fix inventory generator link
Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
2017-01-10 17:02:28 +01:00
Bogdan Dobrelya
3c107ef4dc Fix docs formatting
Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
2017-01-09 17:53:05 +01:00
Bogdan Dobrelya
a5f93d6013 Merge pull request #862 from bogdando/docs
Update docs
2017-01-09 17:43:36 +01:00
Bogdan Dobrelya
e9518072a8 Update docs
Link docs to README, update README with recent info.
Update comparsions, add kubeadm vs kargo.
Better describe variables precedence UX impact.

Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
2017-01-09 16:32:55 +01:00
Alexander Block
1d2a18b355 Introduce dns_mode and resolvconf_mode and implement docker_dns mode
Also update reset.yml to do more dns/network related cleanup.
2017-01-05 23:38:51 +01:00
Bogdan Dobrelya
4a2abc1a46 Merge pull request #845 from bogdando/docs
Comment cloud providers private networks use cases
2017-01-03 10:50:39 +01:00
Bogdan Dobrelya
031cf565ec Comment cloud providers private networks use cases
Signed-off-by: Bogdan Dobrelya <bogdando@mail.ru>
2017-01-02 11:40:56 +01:00
Bogdan Dobrelya
5ec4efe88e Merge pull request #814 from swizzlr/patch-1
Add section describing Kargo vs Kops
2016-12-30 13:58:28 +01:00
Thomas Catterall
b902110d75 Create comparisons.md 2016-12-29 19:41:11 +00:00
Bogdan Dobrelya
a56d9de502 Systemd units, limits, and bin path fixes
* Add restart for weave service unit
* Reuse docker_bin_dir everythere
* Limit systemd managed docker containers by CPU/RAM. Do not configure native
  systemd limits due to the lack of consensus in the kernel community
  requires out-of-tree kernel patches.

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-12-28 15:49:42 +01:00
Bogdan Dobrelya
bb0c3537cb Do not forward bogus domains for upstream resolvers
Also fix kube log level 4 to log dnsmasq queries.

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-12-23 11:53:14 +01:00
Matthew Mosesohn
36a5143478 Merge pull request #804 from mattymo/inventory_builder
Add inventory builder python script
2016-12-23 13:22:41 +03:00
Matthew Mosesohn
7b86b87dca Add inventory builder python script
Includes tox support for running unit tests.
Small note added to getting-started guide for using
inventory_builder.py

Also adds manual-only unit test.
2016-12-23 13:00:56 +03:00
Antoine Legrand
ac295de64c Merge pull request #812 from mattymo/vars_doc
Document commonly used Kargo vars
2016-12-22 18:01:23 +01:00
Matthew Mosesohn
111571b67a Document commonly used Kargo vars 2016-12-22 19:57:39 +03:00
Bogdan Dobrelya
c456a311d6 Add coreos-alpha weave manual CI builds
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-12-22 11:00:31 +01:00
Bogdan Dobrelya
f10d1327d4 Revert "Do not forward private domains for upstream resolvers" 2016-12-21 15:24:17 +01:00
Bogdan Dobrelya
b8bc8eee41 Add download_always_pull check and sha256 for docker images
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-12-20 17:02:09 +01:00
Bogdan Dobrelya
11380769cd Merge pull request #722 from bogdando/dnsmasq_armors
Do not forward private domains for upstream resolvers
2016-12-20 14:25:17 +01:00
Bogdan Dobrelya
8d5da5cfca Add gitlab CI auto builds for triggers
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-12-20 11:23:13 +01:00
Bogdan Dobrelya
101864c050 Do not forward private domains for upstream resolvers
Also fix kube log level 4 to log dnsmasq queries.

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
Co-authored-by: Matthew Mosesohn <mmosesohn@mirantis.com>
2016-12-19 11:01:41 +01:00
Antoine Legrand
768fe05eea Merge pull request #704 from vwfs/bastion_hosts
Add support for bastion hosts
2016-12-17 12:08:49 +01:00
Bogdan Dobrelya
046f3eebcb Rework Gitlab pipeline stages for PRs/merges
Run 1 of each part1/2/special for Gitlab CI only for PRs
Remaining 2/3 of each stage are manual steps for master only (merges)

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-12-16 16:33:14 +01:00
Bogdan Dobrelya
ad68b23d8a Manual steps for Gitlab CI pipeline
* Reduce default testcase to 2 nodes, add HA case.
* Adjust gen_matrix script for Travis/Gitlab CIs.
* Enable netchecker deploy foro gitlab CI.
* Sync other things from travis matrix and reorder them as build steps
  for pull requests, master branch, auto/manual.
* Do auto-step1 from part1 and manual step2,3 for branches/PRs.
* Do manual steps from part2, special for master merges.

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-12-15 17:23:18 +01:00
Bogdan Dobrelya
114ab5e4e6 Merge pull request #721 from adidenko/calico-add-rr
Add calico/routereflector support
2016-12-14 17:22:00 +01:00
Bogdan Dobrelya
1551fe01f9 Rebalance CI GCE zones for better CPU per region usage
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-12-14 16:23:57 +01:00
Aleksandr Didenko
d57c27ffcf Add calico/routereflector support
Add BGP route reflectors support in order to optimize BGP topology
for deployments with Calico network plugin.

Also bump version of calico/ctl for some bug fixes.
2016-12-14 13:44:10 +01:00
Alexander Block
96640e68e2 Add tags for bastion-ssh-config 2016-12-13 17:29:47 +01:00
Alexander Block
3e007df97c Add documentation about bastion hosts 2016-12-13 17:29:47 +01:00
Antoine Legrand
b4ce221002 Merge pull request #730 from vwfs/azurerm
Add Azure Resource Group templates and scripts to contrib
2016-12-13 17:07:41 +01:00
Alexander Block
94ce99eb0a Add documentation link for contrib/azurerm 2016-12-13 16:30:52 +01:00
Bogdan Dobrelya
0515814e0c Fix resolvconf
Do not repeat options and nameservers in the dhclient hooks.
Do not prepend nameservers for dhclient but supersede and fail back
to the upstream_dns_resolvers then default_resolver. Fixes order of
nameservers placement, which is cluster DNS ip goes always first.

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-12-13 15:48:53 +01:00
Bogdan Dobrelya
4e721bfd9d Merge pull request #667 from bogdando/fix_dns
Rework DNS stack to meet hostnet pods needs
2016-12-12 21:38:13 +01:00
Bogdan Dobrelya
3117858dcd Rework DNS stack to meet hostnet pods needs
* For Debian/RedHat OS families (with NetworkManager/dhclient/resolvconf
  optionally enabled) prepend /etc/resolv.conf with required nameservers,
  options, and supersede domain and search domains via the dhclient/resolvconf
  hooks.

* Drop (z)nodnsupdate dhclient hook and re-implement it to complement the
  resolvconf -u command, which is distro/cloud provider specific.
  Update docs as well.

* Enable network restart to apply and persist changes and simplify handlers
  to rely on network restart only. This fixes DNS resolve for hostnet K8s
  pods for Red Hat OS family. Skip network restart for canal/calico plugins,
  unless https://github.com/projectcalico/felix/issues/1185 fixed.

* Replace linefiles line plus with_items to block mode as it's faster.

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
Co-authored-by: Matthew Mosesohn <mmosesohn@mirantis.com>
2016-12-12 17:43:47 +01:00
Bogdan Dobrelya
774f4dbbf7 Merge branch 'master' into tags_download 2016-12-12 11:44:00 +01:00
Bogdan Dobrelya
a15d626771 Preconfigure DNS stack and docker early
In order to enable offline/intranet installation cases:
* Move DNS/resolvconf configuration to preinstall role. Remove
  skip_dnsmasq_k8s var as not needed anymore.

* Preconfigure DNS stack early, which may be the case when downloading
  artifacts from intranet repositories. Do not configure
  K8s DNS resolvers for hosts /etc/resolv.conf yet early (as they may be
  not existing).

* Reconfigure K8s DNS resolvers for hosts only after kubedns/dnsmasq
  was set up and before K8s apps to be created.

* Move docker install task to early stage as well and unbind it from the
  etcd role's specific install path. Fix external flannel dependency on
  docker role handlers. Also fix the docker restart handlers' steps
  ordering to match the expected sequence (the socket then the service).

* Add default resolver fact, which is
  the cloud provider specific and remove hardcoded GCE resolver.

* Reduce default ndots for hosts /etc/resolv.conf to 2. Multiple search
  domains combined with high ndots values lead to poor performance of
  DNS stack and make ansible workers to fail very often with the
  "Timeout (12s) waiting for privilege escalation prompt:" error.

* Update docs.

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-12-09 17:30:55 +01:00
Bogdan Dobrelya
fd9b26675e More granular control for download/upload images/binaries
Add upload tag allow users to exclude distributing images across nodes
when running with the download tag set.
Add related tags and update docs as well.

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-12-09 17:04:55 +01:00
Bogdan Dobrelya
8cc84e132a Add tags
Add tags to allow more granular tasks filtering.
Add generator script for MD formatted tags found.
Add docs for tags how-to.

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-12-09 12:14:28 +01:00
Matthew Mosesohn
fc769eb870 Merge pull request #693 from kubernetes-incubator/upgrades-doc
Add document outlining upgrade process
2016-12-08 13:02:55 +03:00
Bogdan Dobrelya
8eb26c21be Merge pull request #692 from bogdando/gce_fixes
Change GCE sysctls placement and docs
2016-12-07 16:17:30 +01:00
Matthew Mosesohn
3c66e4cdba Add document outlining upgrade process 2016-12-07 16:33:08 +03:00
Bogdan Dobrelya
f0f2b81276 Change GCE sysctls placement and docs
Override GCE sysctl in /etc/sysctl.d/99-sysctl.conf instead of
the /etc/sysctl.d/11-gce-network-security.conf. It is recreated
by GCE, f.e. if gcloud CLI invokes some security related changes,
thus losing customizations we want to be persistent.

Update cloud providers firewall requirements in calico docs.

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-12-07 12:53:45 +01:00
Aleksandr Didenko
b0079ccd77 Calico: fix peering with routers for new version
In new `calicoctl` version nodes peering with routers is broken.
We need to use predictable node names for calico-node and the
same names in calico `bgpPeer` resources and CNI.
2016-12-06 17:17:39 +01:00
Bogdan Dobrelya
a2331fec55 Reduce CI test matrix
Reduce the test cases from 15 to 9, bearing in mind that:
* Disable weave/coreos gate unless its deployment fixed
* If debian/centos7 fails with net plugin X, ubuntu-xenial/rhel-7 will
  likely fail as well
* Canal also covers the flannel plugin deployment, but keep at least one
  of the flannel plugin deployment, unless it's superseded and removed.
* Keep at least one of each OS/plugin family to be tested in the separate
  nodes layout
* Keep at least one of each OS family to be tested against each of the
  plugin types in default nodes layout
* Rebalance GCE regions for instances, replace asia to eu/us as they
  are the longest running jobs.

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-11-29 18:53:43 +01:00
Antoine Legrand
b6872a0be3 Merge pull request #657 from smelchior/master
add  azure support for kargo
2016-11-29 12:20:49 +01:00
Bogdan Dobrelya
7eab889c07 Switch to standard debian/centos/rhel for CI
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-11-29 10:25:07 +01:00
Sebastian Melchior
bb55f68f95 add basic azure support for kargo 2016-11-29 10:20:28 +01:00
Antoine Legrand
5b382668f5 Merge pull request #529 from bogdando/netcheck
Add a k8s app for advanced e2e netcheck for DNS
2016-11-28 15:26:30 +01:00
Bogdan Dobrelya
b7692fad09 Add advanced net check for DNS K8s app
* Add an option to deploy K8s app to test e2e network connectivity
  and cluster DNS resolve via Kubedns for nethost/simple pods
  (defaults to false).
* Parametrize existing k8s apps templates with kube_namespace and
  kube_config_dir instead of hardcode.
* For CoreOS, ensure nameservers from inventory to be put in the
  first place to allow hostnet pods connectivity via short names
  or FQDN and hostnet agents to pass as well, if netchecker
  deployed.

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-11-28 13:23:25 +01:00
Bogdan Dobrelya
2d18e19263 Tune dnsmasq/kubedns limits, replicas, logging
* Add dns_replicas, dns_memory/cpu_limit/requests vars for
dns related apps.
* When kube_log_level=4, log dnsmasq queries as well.
* Add log level control for skydns (part of kubedns app).
* Add limits/requests vars for dnsmasq (part of kubedns app) and
  dnsmasq daemon set.
* Drop string defaults for kube_log_level as it is int and
  is defined in the global vars as well.
* Add docs

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-11-25 12:49:17 +01:00
Bogdan Dobrelya
aa447585c4 Fix download dnsmasq image dependency on docker
When download_run_once with download_localhost is used, docker is
expected to be running on the delegate localhost. That may be not
the case for a non localhost delegate, which is the kube-master
otherwise. Then the dnsmasq role, had it been invoked early before
deployment starts, would fail because of the missing docker dependency.

* Fix that dependency on docker and do not pre download dnsmasq image
  for the dnsmasq role, if download_localhost is disabled.
* Remove become: false for docker CLI invocation because that's not
  the common pattern to allow users access docker CLI w/o sudo.
* Fix opt bin path hack for localhost delegate to ignore errors when
  it fails with "sudo password required" otherwise.
* Describe download_run_once with download_localhost use case in docs
  as well.

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-11-24 18:31:26 +01:00
Bogdan Dobrelya
dc58159d16 Merge pull request #621 from xenolog/calico_network_backend
Add ability to define network backend for Calico.
2016-11-22 14:55:47 +01:00
Sergey Vasilenko
f6d69d0a00 Add ability to define network backend for Calico.
This patch introduce `calico_network_backend` global variable,
which allow to describe alternative network backend.
Default behavior is unchanged.
2016-11-18 16:38:18 +03:00
Alexandre Bourget
5b9b2c0973 Update roadmap.md 2016-11-17 12:44:30 -05:00
Bogdan Dobrelya
e4d240b1b7 Improve CI test matrix
For Travis CI and GCE, add a naive generator script into a markdown table.
Add GCE/Travis CI matrix docs.
Add CoreOS test cases.
Rework existing cases w/o loosing of coverage.
Rework postinstall tests to support CoreOS as well.

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-11-15 18:17:30 +01:00
Artem Panchenko
c58bd33af7 Support new version of 'calicoctl' (>=v1.0.0)
Since version 'v1.0.0-beta' calicoctl is written
in Go and its API differs from old Python based
utility. Added support of both old and new version
of the utility.
2016-11-10 17:11:29 +02:00
Matthew Mosesohn
95b460ae94 Remove etcd-proxy from all nodes and use etcd multiaccess 2016-11-09 13:31:12 +03:00
Smaine Kahlouch
7e81855e24 Merge pull request #554 from bogdando/kubeadm_adoption
Update roadmap for the kubeadm LCM track
2016-10-18 13:52:55 +02:00
Smana
6113a3f350 update roadmap, kubeadm adoption 2016-10-18 13:51:36 +02:00
Bogdan Dobrelya
7d6fc1d680 Update roadmap for the kubeadm LCM track
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-10-18 13:44:45 +02:00
Bogdan Dobrelya
40de468413 Update ha docs
Fix mismatch in code and docs, see
https://github.com/kubespray/kargo/pull/528

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
2016-10-17 15:42:30 +02:00
Smaine Kahlouch
c490e5c8a1 Merge pull request #528 from kubespray/proxy-nginx
Use nginx proxy on non-master nodes to proxy apiserver traffic
2016-10-05 19:19:32 +02:00
Matthew Mosesohn
84052ff0b6 use nginx proxy on non-master nodes to proxy apiserver traffic
Also adds all masters by hostname and localhost/127.0.0.1 to
apiserver SSL certificate.

Includes documentation update on how localhost loadbalancer works.
2016-10-05 20:09:10 +03:00
keglevich3
d548cb6ac2 changed to the correct link 2016-09-29 17:44:24 +03:00