Commit graph

2862 commits

Author SHA1 Message Date
Maxime Guyot
b45f3f0004 Add tf-ovh_coreos CI job (#4763) 2019-05-28 01:51:53 -07:00
Dani Comnea
2a5721b4d4 Change CentOS CRI-O repo from developer repo to public one (#4807) 2019-05-27 05:33:51 -07:00
Vitaliy Dmitriev
333f1a4a40 kubeadm join path fixed for RH linux (#4798) 2019-05-27 01:49:51 -07:00
Andreas Krüger
1e470b0473 Fix certificate-key param for kubeadm init (#4789)
* Fix certificate-key param for kubeadm init

* Fix yamllint error
2019-05-22 02:06:11 -07:00
André R. de Miranda
0ef3a7914c Added pod psp in Rancher Local Path Provisioner (#4385)
* Added pod psp in Rancher Local Path Provisioner

Added pod security policy (psp) in Rancher Local Path Provisioner.

Signed-off-by: André R. de Miranda <andre@miranda.work>

* Apply psp for Rancher Local Path Provisioner only when local_path_provisioner_namespace is not kube-system and also reorganized the templates
2019-05-22 00:16:08 -07:00
bobahspb
a3fff1e438 cordon all deleted nodes before drain (#4756)
Kubespray waits exit of every drain before run other one.
Running drain every after each other seems better than parallel, because we should check resources availability every time.
But, this way, we have one additional problem: possible restart pods on the nodes that are killed little bit later.
Fast cordon before heavy drain seems like an easy solution.
2019-05-21 23:36:05 -07:00
André R. de Miranda
4bc204925a Error in nginx when starting registry-proxy (#4785)
Error starting nginx because in requiredDropCapabilities is dropped all capabilities.

The nginx requires the following capabilities:
- CHOWN
- SETGID
- SETUID

Signed-off-by: André R. de Miranda <andre@miranda.work>
2019-05-20 11:27:15 -07:00
Jacopo Secchiero
5d9946184a Add ignore_assert_errors to "kube-master, ... (#4779)
... kube-node or etcd is empty" task
As a assert must be ignored if ignore_assert_errors is true
2019-05-20 11:25:14 -07:00
Mateus Caruccio
8485136f9a var node_labels as string (#4764) 2019-05-19 12:31:10 -07:00
Maxime Guyot
ff1bc739f1 Change default for kubelet_flexvolumes_plugins_dir (#4752) 2019-05-19 12:29:10 -07:00
Florent Monbillard
8e28ba38d2 Add Load Balancer IP to API servers SANs (#4775)
- Add loadbalancer_apiserver.address to apiserver_sans
2019-05-16 01:23:42 -07:00
MarkusTeufelberger
73c2ff17dd Fix Ansible-lint error [E502] (#4743) 2019-05-16 00:27:43 -07:00
Timoses
13f225e6ae Only pull images for destined host groups (#4735)
Without this, pulls are considered for all
hosts groups, even if not targetted by the downloads
`groups` list. Hence, a download/sync is triggered
even though the host does not require the image.
2019-05-16 00:25:48 -07:00
Aleksey Kasatkin
14749df6f3 Fix "netchecker-server" ClusterRole (#4730)
* Add sha256 hashes for calicoctl v3.6.1

Hashes are added to calicoctl_binary_checksums for both adm and arm platforms.

* Add rules for "network-checker.ext" resource to "netchecker-server" ClusterRole

So that it could access the resource after it is created.

Corresponding issues:
https://github.com/Mirantis/k8s-netchecker-server/issues/125
https://github.com/kubernetes-sigs/kubespray/issues/3281
2019-05-09 01:30:49 -07:00
Sandro Modarelli
2db2898112 Fixed runc path in runtime for RedHat os family (#4731) 2019-05-09 01:28:48 -07:00
Andreas Krüger
044dcbaed0 Add Kubelet config, remove deprecated flags and fix minor bugs (#4724)
* Add kubelet config

* Change kubelet_authorization_mode_webhook to true

* Fix lint

* Sync env file

* Refactor the kubernetes node folder

* Remove deprecated flag and fix lint
2019-05-08 13:38:36 -07:00
Andreas Krüger
8a5eae94ea Minor cleanups of CoreDNS issues and CI job (#4719)
* Minor cleanups

* Add comment in docs that nodelocaldns cache is enabled by default
2019-05-07 13:20:36 -07:00
Andreas Krüger
bf3c6aeed1 Add kube anon auth settings to kubeadm config templates (#4713)
* Disable kube_api_anonymous_auth by default to secure the setup

* Disable metrics-server in addons. Health endpoint is slow and unstable

* Fix anonymous-auth missing in configuration

* Cleanup a bit

* Fix kube anon auth
2019-05-07 12:52:34 -07:00
Dmitri Rubinstein
03bded2b6b Fix adding output of kubeadm to the admin.conf downloaded to the artifacts directory (#4696)
Fixes issue https://github.com/kubernetes-sigs/kubespray/issues/4695
2019-05-06 03:29:36 -07:00
Manuel Cintron
d5c0829d61 Removing unnecessary httplib2 install (#4708) 2019-05-03 17:55:38 -07:00
Alex Barcelo
00369303de Fixing msg parameter for debug module (#4702)
According to [`debug` module documentation](https://docs.ansible.com/ansible/latest/modules/debug_module.html?highlight=msg), the correct parameter name is `msg`.

With the previous `message` parameter name I was getting FAILED messages while ansible was trying to debug previous FAILED tasks.
2019-05-03 12:21:42 -07:00
okamototk
1f1479c0a7 Update ingress nginx 0.24.1. (#4691) 2019-05-03 12:19:39 -07:00
MarkusTeufelberger
e67f848abc ansible-lint: add spaces around variables [E206] (#4699) 2019-05-02 14:24:21 -07:00
MarkusTeufelberger
560f50d3cd Add support for http(s)_proxy to CoreOS, Fedora and OpenSUSE (#4669)
* Add support for http(s)_proxy to CoreOS and Fedora

* fix opensuse proxy support

* Fix CoreOS proxy support

* update documentation
2019-05-02 12:28:22 -07:00
Stas
50bdaa573c Apply etcd_extra_vars to etcd-events.env as well. (#4219)
This change ensures that etcd_extra_vars variable applies
to events etcd as well.
2019-05-02 12:24:27 -07:00
Maxime Guyot
f29387316f Fix ansible-lint 602 (#4688) 2019-05-01 23:42:17 -07:00
Timoses
d6fd0d2aca Enable delegating all downloads (binaries, images, kubeadm images) (#4420)
* Download to delegate and sync files when download_run_once

* Fail on error after saving container image

* Do not set changed status when downloaded container was up to date

* Only sync containers when they are actually required

Previously, non-required images (pull_required=false as
image existed on target host) were synced to the target
hosts. This failed as the image was not downloaded to
the download_delegate and hence was not available for
syncing.

* Sync containers when only missing on some hosts

* Consider images with multiple repo tags

* Enable kubeadm images pull/syncing with download_delegate

* Use kubeadm images list to pull/sync

'kubeadm config images pull' is replaced by collecting the images
list with 'kubeadm config images list' and using the commonly
used method of pull/syncing the images.

* Ensure containers are downloaded and synced for all hosts

* Fix download/syncing when download_delegate is a kubernetes host
2019-05-01 01:10:56 -07:00
Christoffer Anselm
dcd9c9509b Add etcd role dependency on kube user to avoid etcd role failure when running scale.yml with a fresh node. (#3240) (#4479) 2019-04-30 04:01:36 -07:00
Matthew Mosesohn
15eb7db36d Fix k8s api endpoint for secondary nodes in control plane mode (#4675)
Change-Id: I1588458b54c52443ad8d0afbd266f77ac0afea67
2019-04-29 07:50:24 -07:00
Matthew Mosesohn
a5b46bfc8c Run dns_late preinstall tasks on all k8s nodes (#4672)
* Run dns_late preinstall tasks on all k8s nodes

Related issue: #4656

Change-Id: I63f8559ef1a497b7580ab084561e6603fe647834

* Fix ansible-lint

Change-Id: Ia5b33fa63dbc36d8c3e9557ef3f2ea02af2325a5

* Fix recover_control_plane lint issues

Change-Id: I16643a3193c11b6ba704e9698812cac7e4fd19a8
2019-04-29 05:12:21 -07:00
Youngchul Bang
fbba259933 ingress-nginx: enable --report-node-internal-ip-address flag (#4114)
Close #4113
2019-04-29 01:44:22 -07:00
Florent Monbillard
7b77e2d232 Remove docker-storage-setup dependency if not needed (#4077)
When docker_container_storage_setup is false,
docker service should not depend on docker-storage-setup service,
because it's not installed.

For example, when using overlay2 on recent RHEL 7/Centos 7 kernels,
you most likely don't need it.
2019-04-29 01:42:22 -07:00
qvicksilver
48a182844c Documentation and playbook for recovering control plane from node failure (#4146) 2019-04-29 01:40:20 -07:00
Andreas Krüger
38af93b60c Remove rkt support (#4671) 2019-04-29 01:14:20 -07:00
Matthew Mosesohn
741de6051c Fix nodeselectors for contiv and nginx-ingress (#4662)
* Fix nodeselectors for contiv and nginx-ingress

Change-Id: Ib3eb6bd87193c69a90ee944c9164a0b6792c79ba

* Set kube proxy mode to iptables for addons task

Change-Id: Iff71a71f672405c74b4708c71db15ddc4391a53a
2019-04-28 23:36:19 -07:00
Dmitry
b8f0de3074 Fixed etcd-servers-overrides in kubeadm config (#4668)
* kube-apiserver will fail if used comma as separator
2019-04-28 23:02:20 -07:00
MarkusTeufelberger
88d919337e ansible-lint: don't compare to empty string [E602] (#4665) 2019-04-28 23:00:20 -07:00
Matthew Mosesohn
338eb4ce65 Fix kubeadm upload certs with when condition (#4659)
* Fix kubeadm upload certs with when condition

Change-Id: I916dd2375b71eea2386047c7f185a2f8361f7a61

* Update kubeadm-secondary-experimental.yml
2019-04-27 01:14:20 -07:00
Andreas Krüger
3722acee85 Fix broken metrics-server deployment not starting (#4651)
* Fix metrics-server deployment

* Make metrics server work

* Fix sample inventory
2019-04-26 00:44:26 -07:00
grialeyur
82119ca923 Add support calico kubernetes datastore and typha. (#4498)
* Add support calico kubernetes datastore and typha.

* Add typha_enabled to kubespray-defaults.
2019-04-25 05:00:48 -07:00
gitareest
6ca2019002 Fix issue with etcd arm host installation case (#4589)
Use host_architecture variable.
2019-04-25 04:58:47 -07:00
Matthew Mosesohn
c9ed5f69d7 Prepend docker.io for all docker hub images (#4648)
Change-Id: I71dc793641bc168e40419e38f33f68f5325e77a9
2019-04-25 01:34:46 -07:00
Andreas Krüger
3fe66a1298 Update downloads role to download to correct group (#4638) 2019-04-24 10:48:03 -07:00
Sergey Kolekonov
4a10dca7d4 Add an ability to provide oidc cert in base64 (#4618) 2019-04-24 09:40:01 -07:00
Matthew Mosesohn
4d57ed314d Clean up check for setting kubeadm certificate key (#4634)
Change-Id: I2c97c4753089eb3ec2e6b01b2681a8be98ecbb57
2019-04-24 07:14:12 -07:00
iwankgb
4e81bcc147 Fixing Vagrant cluster provisioning (#4218)
* Pass ansible_ssh_user as host_var

Co-authored-by: Damian Darczuk <damian.darczuk@intel.com>
Co-authored-by: Paweł Pałucki <pawel.palucki@intel.com>

* Create a directory before downloading container images to ansible host

Co-authored-by: Damian Darczuk <damian.darczuk@intel.com>
Co-authored-by: Paweł Pałucki <pawel.palucki@intel.com>

* Set private key usuing synchronize task options

Co-authored-by: Damian Darczuk <damian.darczuk@intel.com>
Co-authored-by: Paweł Pałucki <pawel.palucki@intel.com>
2019-04-24 05:42:05 -07:00
Attilio Greco
6243467856 remove duble check for run this task just one time (#4613) 2019-04-24 05:38:01 -07:00
Vincent Gramer
f47a666227 support azure loadbalancer standard sku (#4150) (#4476)
add the support of the folling property in azure-credential-check.yml
  - azure_loadbalancer_sku: Sku of Load Balancer and Public IP. Candidate values are: basic and standard.
  - azure_exclude_master_from_standard_lb: excludes master nodes from standard load balancer.
  - azure_disable_outbound_snat: disables the outbound SNAT for public load balancer rules
  - useInstanceMetadata: Use instance metadata service where possible
  - azure_primary_availability_set: (Optional) The name of the availability set that should be used as the load balancer backend
2019-04-24 02:14:01 -07:00
Wilmar den Ouden
b708db4cd5 Update to v1.14.1 (#4481) 2019-04-24 02:08:01 -07:00
Matthew Mosesohn
fc072300ea Purge legacy cleanup tasks from older than 1 year (#4450)
We don't need to support upgrades from 2 year old installs,
just from the last major version.

Also changed most retried tasks to 1s delay instead of longer.
2019-04-24 00:08:05 -07:00
Chad Swenson
d25ecfe1c1 Update Docker defaults to 18.09.5 and drop deprecated (#4624)
As of kubernetes v1.14, docker 18.09 is [validated for use](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.14.md#external-dependencies). Docker 1.11 and 1.12 were dropped.

This patch:
- Updates the default docker version to 18.09
- Updates Docker packages to the latest 18.09 patch (18.09.5)
- Removes options for Docker 1.11 and 1.12
2019-04-23 22:24:01 -07:00
MarkusTeufelberger
a65605b17a ansible-lint: Don't use bare variables (#4608)
Circumvented one false positive from ansible-lint
Moved a block of jinja magic into its own variable
2019-04-23 22:20:00 -07:00
MarkusTeufelberger
424e59805f ansible-lint: Fix commands that are also available as module (#4619) 2019-04-23 22:18:00 -07:00
MarkusTeufelberger
76db060afb Define and implement specs for bootstrap-os (#4455)
* Add README to bootstrap-os role

* Rework bootstrap-os once more

* Document workarounds for bugs/deficiencies in Ansible modules
* Unify and document role variables
* Remove installation of additional packages and repositories
* Merge Ubuntu and Debian tasks
* Remove pipelining setting from default playbooks
* Fix OpenSUSE not running its required tasks
2019-04-23 15:46:02 -07:00
Andreas Krüger
d588532c9b Update probe timeouts, delays etc. (#4612)
* Fix merge conflict

* Add check delay

* Add more liveness and readiness options to metrics-server
2019-04-23 14:46:02 -07:00
Matthew Mosesohn
d6d7458d68 Fix control plane setup without a hardcoded key (#4610) 2019-04-23 14:37:59 -07:00
Matthew Mosesohn
d89ecb8308 disable metrics server and fix terraform (#4617)
* disable metrics server in centos7-flannel-addons job

Change-Id: I1d87923547584896f64dda9ea8feb5581ad48cbe

* Fix tf facility->facilities syntax

Change-Id: I434bfe53f47e8e4a546890e0b62d24bde6e6d6a7

* Update Terraform CI for facilities

* Fix undefined variable error
2019-04-23 12:06:03 -07:00
Maxime Guyot
50751bb610 Revert "Optimize kube resources creation (#4572)" (#4621)
This reverts commit f8fdc0cd93.
2019-04-23 20:37:23 +03:00
andreyshestakov
f8fdc0cd93 Optimize kube resources creation (#4572) 2019-04-22 23:34:10 -07:00
Matthew Mosesohn
09fe95bc60 Avoid creating k8s cert dir on non-k8s nodes (#4602) 2019-04-21 15:27:43 -07:00
Victor Morales
ada5941a70 Unmask Docker service in ClearLinux (#4583)
The docker service provided by the containers-basic bundle is masked
in ClearLinux distribution. This is causing errors in the following
steps. This commit ensures that the unit is not masked.
2019-04-21 07:31:43 -07:00
Vedran Bartonicek
33ab615072 Wait longer for node to join the cluster (#4549) 2019-04-20 07:05:40 -07:00
Rabi Mishra
5a1cf19278 Install cri-tools on fedora (#4350) 2019-04-20 06:29:40 -07:00
Matthew Mosesohn
05dc2b3a09 Use K8s 1.14 and add kubeadm experimental control plane mode (#4514)
* Use K8s 1.14 and add kubeadm experimental control plane mode

This reverts commit d39c273d96.

* Cleanup kubeadm setup run on first master

* pin kubeadm_certificate_key in test

* Remove kubelet autolabel of kube-node, add symlink for pki dir

Change-Id: Id5e74dd667c60675dbfe4193b0bc9fb44380e1ca
2019-04-19 06:01:54 -07:00
Aleksey Kasatkin
d0e628911c Add sha256 hashes for calicoctl v3.6.1 (#4580)
Hashes are added to calicoctl_binary_checksums for both adm and arm platforms.
2019-04-19 05:45:55 -07:00
Andreas Krüger
656633f784 YAMLLint everything (#4576) 2019-04-18 23:59:54 -07:00
Victor Morales
f5aec8add4 Fix runc absolute path (#4542)
The BINDIR variable defined on the runc's Makefile[1] defines
installation path is on $(PREFIX)/sbin which used for most of the
Linux distributions. This change fixes the absolute path used for
non-ClearLinux distributions (CentOS, Ubuntu).

[1] https://github.com/opencontainers/runc/blob/master/Makefile#L10
2019-04-18 15:41:58 -07:00
Maxime Guyot
f92309bfd0 Fix ansible-lint for ceph package (#4568) 2019-04-18 13:45:25 -07:00
Victor Morales
c6586829de Ensure /etc/bash_completion.d/ folder exists (#4543)
The Stateless ClearLinux feature[1] requires the creation of folders
in /etc folder. This change ensure the existence of the
/etc/bash_completion.d/ folder for ClearLinux Distribution.

[1] https://clearlinux.org/features/stateless
2019-04-18 02:24:10 -07:00
Maxime Guyot
b218e17f44 ansible-lint: E403 Package installs should not use latest (#4500) 2019-04-18 01:34:08 -07:00
Maxime Guyot
a6dc50e7cb Add host information for canal readiness probe (#4548) 2019-04-17 10:22:02 -07:00
Maxime Guyot
37eac010c8 ansible-lint: Don’t compare to literal True/False (#4499) 2019-04-17 08:42:03 -07:00
Maxime Guyot
ec3daedf9e Revert "Fix for unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels (#4320)" (#4553)
This reverts commit 586ad89d50.
2019-04-17 07:58:06 -07:00
Jugwan Eom
d83181a2be add RBD Provisioner Addon (#3667) (#3668)
Based on the CephFS Provisioner Addon, the following changes have been made:
 - Upstream v2.1.1-k8s1.11
 - Configurable Provisioner replicas
2019-04-16 23:14:02 -07:00
andreyshestakov
78f6f6b889 Mark "Calico | Set global as_num" as "unchanged" (#4539)
This command executes with "--skip-exists" parameter, so it is idempotent
and should not be marked as "changed".
2019-04-16 09:31:11 -07:00
Matthew Mosesohn
c5fb734098 Switch calicoctl from a container to a binary (#4524) 2019-04-15 04:24:04 -07:00
Matthew Mosesohn
d39c273d96 Revert "Use K8s 1.14 and add kubeadm experimental control plane mode (#4317)" (#4510)
This reverts commit 316508626d.
2019-04-11 12:52:43 -07:00
Matthew Mosesohn
316508626d Use K8s 1.14 and add kubeadm experimental control plane mode (#4317)
* Use Kubernetes 1.14 and experimental control plane support

* bump to v1.14.0
2019-04-11 05:30:13 -07:00
Maxime Guyot
46ba6a4154 ansible-lint: when lines should not include Jinja2 variables (#4496) 2019-04-11 03:06:10 -07:00
Andreas Krüger
4ff851b302 Enable nodelocaldns by default (#4461)
* Enable nodelocaldns by default

* Enable nodelocaldns by default

* nodelocaldns is now default

* Disable enable_nodelocaldns for the addons CI jobs

Disable enable_nodelocaldns for the addons CI jobs to make sure things still work without nodelocaldns
2019-04-11 00:24:08 -07:00
Qasim Sarfraz
3af90f8772 disable cloud-routes for non-cloud plugin (#4443) 2019-04-10 23:50:09 -07:00
Andreas Krüger
9032e271f1 Upgrade CoreDNS to 1.5.0 (#4494) 2019-04-10 13:40:08 -07:00
Andreas Krüger
15597aa493 Do not force TCP connections to upstreams. (#4492) 2019-04-10 12:40:09 -07:00
Sergey
3b9d13fda9 Return back bind API server node loadbalancer to 127.0.0.1 for security purposes. (#4489) 2019-04-10 12:20:08 -07:00
Andreas Krüger
5e0249ae7c Add HAProxy as internal loadbalancer (#4480) 2019-04-10 05:56:18 -07:00
Maxime Guyot
353afa7cb0 Fix ipip: false in calico v3 (#4473) 2019-04-10 05:50:15 -07:00
Neven Miculinic
a30ad1e5a5 Added generic CNI network plugin (#4322)
* Added generic CNI network plugin

* Added CNI network plugin documentation

* added necessary fix
2019-04-10 04:16:15 -07:00
Robert Neumann
586ad89d50 Fix for unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels (#4320)
* Fix the file path for all.yml and k8s-cluster.yml

* Fix --node-labels namespace error "unknown labels specified"

* Update templates and configs kubelet node-labels
2019-04-10 04:14:12 -07:00
Sidharth Anupkrishnan
6caa639243 Update CoreDNS label as specified in the kubernetes coredns repository (#3920) 2019-04-10 04:12:13 -07:00
MarkusTeufelberger
d2a1ac3b0c Add Ansible-lint CI step (#4411)
* Add ansible-lint as gitlab-ci step

* Fix jinja2 syntax in include_tasks that breaks ansible-lint

* Use a block scalar to get around gitlab quoting/escaping rules

* Run ansible-lint in verbose mode in CI
2019-04-10 02:04:16 -07:00
André R. de Miranda
097806dfe8 Added tag kube-proxy (#4272)
Signed-off-by: André R. de Miranda <andre@miranda.work>
2019-04-09 05:25:06 -07:00
Abdulaziz AlMalki
7cdf1fd388 quote values for kube_oidc_groups_prefix and kube_oidc_username_prefix values to accept colon, e.g oidc: (#4305)
This will fix error: error converting YAML to JSON: yaml: line 36: mapping values are not allowed in this context

Signed-off-by: Abdulaziz AlMalki <almalki.a@gmail.com>
2019-04-09 05:23:06 -07:00
Andreas Krüger
aa162b0d5d Update kube-router to 0.2.5 (#4469) 2019-04-09 03:37:04 -07:00
Maxime Guyot
b15f3e182d add default routing to canal and disable bird checks (#4468)
Co-Author: Paweł Skrzyński
2019-04-09 02:45:07 -07:00
Andreas Krüger
4d39c1856e Fix jinja filters (#4470) 2019-04-09 02:19:06 -07:00
Maxime Guyot
913fed0089 kubeadmn init: add 'until' to make 'retries' effective (#4464)
an 'until' clause is required or 'retries' is ignored

(see note @ https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html#do-until-loops)
2019-04-09 00:21:04 -07:00
Markos Chandras
12c6b5c3eb openSUSE: Use Leap 15.0 instead of 42.3 (#4442)
* Vagrantfile: Bump openSUSE to Leap 15.0

* roles: container-engine: Add 'containerd' package for openSUSE

The 'containerd' package contains the docker-containerd and
docker-containerd-shim binaries. We also need to ensure that the latest
version is installed since an older version may already be present (eg GCE
images)

* Remove docker log-opts for opensuse

* roles: bootstrap-os: Use lowercase 'o' for openSUSE

OpenSUSE is not a valid family name. The correct one is openSUSE

* roles: bootstrap-os: Update zypper cache before first installation

The zypper cache may be outdated so ensure that it's fully updated
before we try and install the bootstrap packages.
2019-04-09 00:17:05 -07:00
rptaylor
f52584a715 robust handling of API server SANs (#4435)
* robust handling of API server SANs

* use apiserver_loadbalancer_domain_name if it is defined, according to PR 3977
2019-04-08 08:10:35 -07:00
Erwan Miran
09bbdadcee remove nodelocaldns iface on reset (#4460) 2019-04-08 02:26:25 -07:00
Xinghong Fang
d711a0c83f [nodelocaldns] expand tolerations on the daemonset (#4451) 2019-04-08 02:24:26 -07:00