Add option to force docker to use systemd as cgroup manager #7815

priyawadhwa · 2020-04-21T00:47:45Z

Since minikube is running systemd, kubeadm expects kubeadm to be the cgroup manager. If docker is using a different cgroup manager like cgroupfs, this can cause unstable resource allocation. We were seeing this in Cloud Shell, and forcing docker to use systemd resolved the issue.

This k8s doc provides good context: https://kubernetes.io/docs/setup/production-environment/container-runtimes

I modified the kic base image to include the docker daemon config which forces systemd. I upgraded kic to v0.0.10. This file didn't exist before, so it isn't being overwritten.

I elected to go with --force-systemd instead of --cgroup-manager[cgroupfs,systemd] since the latter would introduce more complexity than needed. Crio defaults to systemd while docker/containerd default to cgroupfs, and we'd need to keep track of acceptable combos with the latter option.

Fixes #6404

Since minikube is running systemd, kubeadm expects kubeadm to be the cgroup manager. If docker is using a different cgroup manager like cgroupfs, this can cause unstable resource allocation. We were seeing this in Cloud Shell, and forcing docker to use systemd resolved the issue.

priyawadhwa · 2020-04-21T00:47:52Z

/ok-to-test

k8s-ci-robot · 2020-04-21T00:48:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: priyawadhwa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [priyawadhwa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

codecov-io · 2020-04-21T00:54:19Z

Codecov Report

Merging #7815 into master will decrease coverage by 0.05%.
The diff coverage is 20.00%.

@@            Coverage Diff             @@
##           master    #7815      +/-   ##
==========================================
- Coverage   35.50%   35.45%   -0.06%     
==========================================
  Files         148      148              
  Lines        9330     9347      +17     
==========================================
+ Hits         3313     3314       +1     
- Misses       5620     5635      +15     
- Partials      397      398       +1

Impacted Files	Coverage Δ
pkg/minikube/cruntime/cruntime.go	`58.06% <ø> (ø)`
pkg/minikube/cruntime/docker.go	`27.27% <5.88%> (-2.40%)`	⬇️
cmd/minikube/cmd/start_flags.go	`51.14% <100.00%> (+0.16%)`	⬆️
pkg/minikube/cruntime/containerd.go	`28.65% <100.00%> (ø)`
pkg/minikube/cruntime/crio.go	`52.12% <100.00%> (ø)`

minikube-pr-bot · 2020-04-21T00:59:22Z

kvm2 Driver
docker Driver
Times for minikube: [24.872658683 25.123001697999996 25.756288509000004]
Average time for minikube: 25.25064963

Times for Minikube (PR 7815): [27.389023917 26.465480006 26.028746423999998]
Average time for Minikube (PR 7815): 26.62775011566666

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7815) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.9.2 on Debian            |  0.068726 |                    |
|                                   9.11 |           |                    |
| * Using the docker driver              |  0.003072 |           0.002558 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.056598 |           0.055325 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.373381 |           7.757155 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.0         |  0.000169 |           0.000160 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 16.932184 |          17.655101 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  0.745760 |           0.892541 |
| components...                          |           |                    |
| * Enabled addons:                      |  0.002294 |           0.127638 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.063237 |           0.062939 |
| configured to use "minikube"           |           |                    |
|                                        |  0.005228 |           0.069351 |
+----------------------------------------+-----------+--------------------+

pkg/minikube/cruntime/docker.go

priyawadhwa · 2020-04-21T16:46:52Z

/retest

pkg/minikube/cruntime/docker.go

afbjorklund · 2020-04-21T17:08:45Z

@priyawadhwa

Docker has some weird reason for not using this by default:

# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

But like you say, Kubernetes recommends using the same as host.

Control groups are used to constrain resources that are allocated to processes. A single cgroup manager will simplify the view of what resources are being allocated and will by default have a more consistent view of the available and in-use resources.

…systemd

…mage builds

priyawadhwa · 2020-04-21T21:22:20Z

@afbjorklund yah there's definitely a disparity. Since the goal here is to run k8s, I feel like we should stick to what they are recommending, especially if tests pass.

If we start to see issues around this, we could put it behind a flag so that only users who need it can use it.

minikube-pr-bot · 2020-04-21T21:33:05Z

kvm2 Driver
docker Driver
Times for minikube: [26.599362852000002 26.093345421 26.803032682]
Average time for minikube: 26.498580318333335

Times for Minikube (PR 7815): [26.787263245000002 27.170312600000003 26.924779744000002]
Average time for Minikube (PR 7815): 26.960785196333333

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7815) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.0 on           |  0.072636 |           0.070769 |
| Debian 9.11                            |           |                    |
| * Using the docker driver              |  0.002554 |           0.002559 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.055088 |           0.057376 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.362950 |           7.715262 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.0         |  0.000250 |           0.000176 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 17.752281 |          17.655175 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  1.057231 |           1.388046 |
| components...                          |           |                    |
| * Enabled addons:                      |  0.123217 |           0.002446 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.067499 |           0.065524 |
| configured to use "minikube"           |           |                    |
|                                        |  0.004874 |           0.003453 |
+----------------------------------------+-----------+--------------------+

priyawadhwa · 2020-04-22T00:23:09Z

It looks like the github actions environment doesn't like this --

functional_test_docker_ubuntu : completed with 20 / 33 failures in 27 min 44 seconds
----------------20 Failures----------------------------
[
  "TestFunctional/serial/StartWithProxy",
  "TestFunctional/serial/SoftStart",
  "TestFunctional/serial/KubeContext",
  "TestFunctional/serial/KubectlGetPods",
  "TestFunctional/serial/CacheCmd/cache/verify_cache_inside_node",
  "TestFunctional/serial/CacheCmd/cache/cache_reload",
  "TestFunctional/serial/MinikubeKubectlCmd",
  "TestFunctional/parallel/ComponentHealth",
  "TestFunctional/parallel/DashboardCmd",
  "TestFunctional/parallel/DNS",
  "TestFunctional/parallel/StatusCmd",
  "TestFunctional/parallel/LogsCmd",
  "TestFunctional/parallel/MountCmd",
  "TestFunctional/parallel/ServiceCmd",
  "TestFunctional/parallel/PersistentVolumeClaim",
  "TestFunctional/parallel/TunnelCmd",
  "TestFunctional/parallel/MySQL",
  "TestFunctional/parallel/UpdateContextCmd",
  "TestFunctional/parallel/DockerEnv",
  "TestFunctional/parallel/NodeLabels"
]
-------------------------------------------------------
*** 13 Passed ***
*** 20 Failed ***

I'm still not totally sure what's causing some environments to only work with systemd (Cloud Shell) and some to totally fail with systemd (Github Actions).

I'm thinking of putting this behind a flag for now, WDYT @afbjorklund @medyagh ?

afbjorklund · 2020-04-22T17:21:22Z

We already noticed that GitHub Actions is running a totally different docker

i.e. they have their own fork of moby, rather than using the regular binary

afbjorklund · 2020-04-22T17:22:21Z

Ouch, I just noticed there is a podman and crio upgrade in here as well...

hack/images/kicbase.Dockerfile

priyawadhwa · 2020-04-22T17:39:48Z

Ouch, I just noticed there is a podman and crio upgrade in here as well...

@afbjorklund yah the Dockerfile build is failing with "version cannot be found" errors for both crio and podman, so they need to be updated for the v0.0.10 image to be built :/

minikube-pr-bot · 2020-04-23T22:36:05Z

kvm2 Driver
Times for minikube: [68.07052166 65.328770165 66.878327623]
Average time for minikube: 66.75920648266666

Times for Minikube (PR 7815): [67.377331953 66.864556251 66.00641521200001]
Average time for Minikube (PR 7815): 66.749434472

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 7815) |
+--------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.1 on   |  0.069793 |           0.067569 |
| Debian 9.11                    |           |                    |
| * Using the kvm2 driver based  |  0.021234 |           0.023051 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.003703 |           0.004379 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 41.293816 |          41.396553 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.0 | 23.152054 |          23.049452 |
| on Docker 19.03.8 ...          |           |                    |
| * Verifying Kubernetes         |  1.412131 |           1.431647 |
| components...                  |           |                    |
| * Enabled addons:              |  0.734440 |           0.696026 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.068457 |           0.075144 |
| configured to use "minikube"   |           |                    |
|                                |  0.003577 |           0.005612 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [28.175077165999998 27.05879206099999 26.650161091999998]
Average time for minikube: 27.294676772999996

Times for Minikube (PR 7815): [28.445735458000005 27.601543418000002 27.508156965]
Average time for Minikube (PR 7815): 27.851811947

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7815) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.1 on           |  0.073896 |           0.073344 |
| Debian 9.11                            |           |                    |
| * Using the docker driver              |  0.002634 |           0.002500 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.062225 |           0.058510 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.826686 |           7.842191 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.0         |  0.000289 |           0.000314 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 18.052175 |          18.614617 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  1.204962 |           1.115073 |
| components...                          |           |                    |
| * Enabled addons:                      |  0.003024 |           0.078036 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.065991 |           0.061922 |
| configured to use "minikube"           |           |                    |
|                                        |  0.002795 |           0.005306 |
+----------------------------------------+-----------+--------------------+

…systemd

minikube-pr-bot · 2020-04-23T23:15:58Z

kvm2 Driver
Times for minikube: [64.948530635 64.620499841 65.331963629]
Average time for minikube: 64.96699803499999

Times for Minikube (PR 7815): [65.409322369 66.154082117 65.688193337]
Average time for Minikube (PR 7815): 65.75053260766667

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 7815) |
+--------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.1 on   |  0.061288 |           0.066073 |
| Debian 9.11                    |           |                    |
| * Using the kvm2 driver based  |  0.023250 |           0.023773 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.003036 |           0.003119 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 41.238825 |          40.888938 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.0 | 21.685300 |          22.444736 |
| on Docker 19.03.8 ...          |           |                    |
| * Verifying Kubernetes         |  1.296673 |           1.745855 |
| components...                  |           |                    |
| * Enabled addons:              |  0.573068 |           0.494399 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.079584 |           0.079331 |
| configured to use "minikube"   |           |                    |
|                                |  0.005976 |           0.004309 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [25.592244189000002 26.28152721 25.412536669]
Average time for minikube: 25.762102689333336

Times for Minikube (PR 7815): [28.171486896 26.395919186000004 28.095956455]
Average time for Minikube (PR 7815): 27.554454179000004

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7815) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.1 on           |  0.076717 |           0.080548 |
| Debian 9.11                            |           |                    |
| * Using the docker driver              |  0.002438 |           0.003205 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.061563 |           0.065172 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.862619 |           7.851804 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.0         |  0.000239 |           0.000424 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 16.775269 |          17.684436 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  0.905287 |           1.796421 |
| components...                          |           |                    |
| * Enabled addons:                      |  0.003175 |           0.002673 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.071163 |           0.065506 |
| configured to use "minikube"           |           |                    |
|                                        |  0.003634 |           0.004264 |
+----------------------------------------+-----------+--------------------+

medyagh

one last nit: ForceSystemd() should be private, as it is not being called anywhere outside

minikube-pr-bot · 2020-04-24T00:07:57Z

kvm2 Driver
Times for minikube: [67.976863478 67.4307277 64.51547140199999]
Average time for minikube: 66.64102085999998

Times for Minikube (PR 7815): [67.919457965 64.93236012999999 66.66688270899999]
Average time for Minikube (PR 7815): 66.50623360133334

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 7815) |
+--------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.1 on   |  0.063764 |           0.063495 |
| Debian 9.11                    |           |                    |
| * Using the kvm2 driver based  |  0.020560 |           0.021617 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.003737 |           0.003913 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 41.704168 |          42.130958 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.0 | 22.610633 |          22.032791 |
| on Docker 19.03.8 ...          |           |                    |
| * Verifying Kubernetes         |  1.523609 |           1.412627 |
| components...                  |           |                    |
| * Enabled addons:              |  0.618302 |           0.747761 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.092496 |           0.087715 |
| configured to use "minikube"   |           |                    |
|                                |  0.003751 |           0.005356 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [27.804117106999996 27.345895842 25.779142564]
Average time for minikube: 26.976385171

Times for Minikube (PR 7815): [28.451083855000004 28.723881671 27.129281009]
Average time for Minikube (PR 7815): 28.101415511666662

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7815) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.1 on           |  0.077092 |           0.082735 |
| Debian 9.11                            |           |                    |
| * Using the docker driver              |  0.002624 |           0.002759 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.060013 |           0.065810 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.844556 |           8.047117 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.0         |  0.000259 |           0.000309 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 17.346511 |          18.858319 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  1.462169 |           0.943874 |
| components...                          |           |                    |
| * Enabled addons:                      |  0.111702 |           0.023232 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.067464 |           0.073241 |
| configured to use "minikube"           |           |                    |
|                                        |  0.003995 |           0.004020 |
+----------------------------------------+-----------+--------------------+

medyagh · 2020-04-24T06:07:59Z

/retest-this-please

medyagh · 2020-04-24T06:09:24Z

KVM tests look very scary https://storage.googleapis.com/minikube-builds/logs/7815/52b5737/KVM_Linux.html
but it could be just a bad CI machine, I invoked a retest. lets double check the KVM results.

…systemd

minikube-pr-bot · 2020-04-24T17:35:08Z

kvm2 Driver
Times for minikube: [66.503681624 66.09319483 69.256900846]
Average time for minikube: 67.28459243333333

Times for Minikube (PR 7815): [66.619740621 68.860728491 66.752424976]
Average time for Minikube (PR 7815): 67.41096469600001

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 7815) |
+--------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.1 on   |  0.062961 |           0.069238 |
| Debian 9.11                    |           |                    |
| * Using the kvm2 driver based  |  0.022089 |           0.020990 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.003213 |           0.003807 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 41.207920 |          41.012688 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.0 | 23.598368 |          23.920247 |
| on Docker 19.03.8 ...          |           |                    |
| * Verifying Kubernetes         |  1.536062 |           1.445200 |
| components...                  |           |                    |
| * Enabled addons:              |  0.771594 |           0.859078 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.077738 |           0.075101 |
| configured to use "minikube"   |           |                    |
|                                |  0.004648 |           0.004616 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [26.546487178000003 25.964671826 26.932369685]
Average time for minikube: 26.48117622966667

Times for Minikube (PR 7815): [27.155278621 28.76059046 27.136706439999998]
Average time for Minikube (PR 7815): 27.68419184033333

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7815) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.1 on           |  0.076349 |           0.077302 |
| Debian 9.11                            |           |                    |
| * Using the docker driver              |  0.002508 |           0.002589 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.060140 |           0.058603 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.776117 |           7.622477 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.0         |  0.000242 |           0.000217 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 17.565663 |          18.300229 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  0.816295 |           1.438555 |
| components...                          |           |                    |
| * Enabled addons:                      |  0.113758 |           0.114719 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.065471 |           0.064305 |
| configured to use "minikube"           |           |                    |
|                                        |  0.004634 |           0.005194 |
+----------------------------------------+-----------+--------------------+

priyawadhwa · 2020-04-24T18:54:03Z

@medyagh the KVM failures are normal now

minikube-pr-bot · 2020-04-24T21:28:29Z

kvm2 Driver
Times for minikube: [68.575935416 66.895081814 67.161923937]
Average time for minikube: 67.54431372233333

Times for Minikube (PR 7815): [66.634958373 64.80708744 64.223911174]
Average time for Minikube (PR 7815): 65.22198566233334

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 7815) |
+--------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.1 on   |  0.060748 |           0.064048 |
| Debian 9.11                    |           |                    |
| * Using the kvm2 driver based  |  0.021217 |           0.021214 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.003155 |           0.003126 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 41.862535 |          40.284083 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.0 | 23.258207 |          22.804167 |
| on Docker 19.03.8 ...          |           |                    |
| * Verifying Kubernetes         |  1.841598 |           1.483520 |
| components...                  |           |                    |
| * Enabled addons:              |  0.414843 |           0.489994 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.078588 |           0.067020 |
| configured to use "minikube"   |           |                    |
|                                |  0.003422 |           0.004814 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [27.758045159 27.826507727 28.926681746]
Average time for minikube: 28.170411544

Times for Minikube (PR 7815): [28.562015612 27.303079212999997 26.302191173]
Average time for Minikube (PR 7815): 27.389095332666667

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7815) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.1 on           |  0.075384 |           0.079729 |
| Debian 9.11                            |           |                    |
| * Using the docker driver              |  0.002854 |           0.003025 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.058927 |           0.061586 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.806792 |           7.802093 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.0         |  0.000196 |           0.000284 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 19.229091 |          18.111480 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  0.917744 |           1.172210 |
| components...                          |           |                    |
| * Enabled addons:                      |  0.004187 |           0.089233 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.071278 |           0.064470 |
| configured to use "minikube"           |           |                    |
|                                        |  0.003958 |           0.004986 |
+----------------------------------------+-----------+--------------------+

…systemd

minikube-pr-bot · 2020-04-24T22:48:56Z

kvm2 Driver
Times for minikube: [66.761154956 66.985106589 65.77785349199999]
Average time for minikube: 66.50803834566666

Times for Minikube (PR 7815): [64.79517546 65.32403882700001 65.487059282]
Average time for Minikube (PR 7815): 65.20209118966666

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 7815) |
+--------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.1 on   |  0.064480 |           0.065669 |
| Debian 9.11                    |           |                    |
| * Using the kvm2 driver based  |  0.022751 |           0.021020 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.003518 |           0.009384 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 40.669287 |          40.735962 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.0 | 23.488506 |          22.510134 |
| on Docker 19.03.8 ...          |           |                    |
| * Verifying Kubernetes         |  1.511025 |           1.368766 |
| components...                  |           |                    |
| * Enabled addons:              |  0.640338 |           0.416431 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.100608 |           0.070711 |
| configured to use "minikube"   |           |                    |
|                                |  0.007524 |           0.004014 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [27.266954648000006 26.529594409999998 28.186146810000004]
Average time for minikube: 27.327565289333336

Times for Minikube (PR 7815): [26.967782229 26.816947358999997 28.15798285]
Average time for Minikube (PR 7815): 27.314237479333332

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7815) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.1 on           |  0.080031 |           0.076937 |
| Debian 9.11                            |           |                    |
| * Using the docker driver              |  0.002943 |           0.002558 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.059916 |           0.059341 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  8.024936 |           7.659283 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.0         |  0.000209 |           0.000213 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 17.578186 |          17.891871 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  1.509875 |           1.470057 |
| components...                          |           |                    |
| * Enabled addons:                      |  0.002700 |           0.084531 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.066094 |           0.065807 |
| configured to use "minikube"           |           |                    |
|                                        |  0.002676 |           0.003639 |
+----------------------------------------+-----------+--------------------+

medyagh · 2020-04-25T03:22:58Z

@priyawadhwa
the docker driver tests are abnormally high and seems to be affecting the multi node,https://storage.googleapis.com/minikube-builds/logs/7815/71ca99b/Docker_Linux.html

could it be because we need to make the docker on all the nodes use systemd ?

or is it a coinsidence?

medyagh · 2020-04-25T05:59:11Z

/retest-this-please

medyagh · 2020-04-25T23:05:41Z

/retest-this-please

priyawadhwa · 2020-04-27T17:33:01Z

@medyagh this seems to be a coincidence. It also doesn't seem like the tests reran, so I'll merge w/ master and we can rerun them

…systemd

minikube-pr-bot · 2020-04-27T17:44:57Z

kvm2 Driver
Times for minikube: [65.260365721 65.98561031300001 66.145139248]
Average time for minikube: 65.79703842733333

Times for Minikube (PR 7815): [65.52541886099999 62.54227959400001 65.857554524]
Average time for Minikube (PR 7815): 64.641750993

Averages Time Per Log

+--------------------------------+-----------+--------------------+
|              LOG               | MINIKUBE  | MINIKUBE (PR 7815) |
+--------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.1 on   |  0.070153 |           0.061089 |
| Debian 9.11                    |           |                    |
| * Using the kvm2 driver based  |  0.022264 |           0.023410 |
| on existing profile            |           |                    |
| * Starting control plane node  |  0.008442 |           0.003059 |
| minikube in cluster minikube   |           |                    |
| * Creating kvm2 VM (CPUs=2,    | 40.309809 |          40.190318 |
| Memory=3700MB, Disk=20000MB)   |           |                    |
| ...                            |           |                    |
| * Preparing Kubernetes v1.18.0 | 22.765529 |          21.978105 |
| on Docker 19.03.8 ...          |           |                    |
| * Verifying Kubernetes         |  1.537970 |           1.441750 |
| components...                  |           |                    |
| * Enabled addons:              |  0.991934 |           0.856786 |
| default-storageclass,          |           |                    |
| storage-provisioner            |           |                    |
| * Done! kubectl is now         |  0.087625 |           0.082401 |
| configured to use "minikube"   |           |                    |
|                                |  0.003314 |           0.004833 |
+--------------------------------+-----------+--------------------+

docker Driver
Times for minikube: [26.801710952 26.983131289 25.826524566]
Average time for minikube: 26.537122268999997

Times for Minikube (PR 7815): [27.149376945999997 27.420185357 27.484454285]
Average time for Minikube (PR 7815): 27.351338862666665

Averages Time Per Log

+----------------------------------------+-----------+--------------------+
|                  LOG                   | MINIKUBE  | MINIKUBE (PR 7815) |
+----------------------------------------+-----------+--------------------+
| * minikube v1.10.0-beta.1 on           |  0.075132 |           0.079928 |
| Debian 9.11                            |           |                    |
| * Using the docker driver              |  0.003708 |           0.003317 |
| based on existing profile              |           |                    |
| * Starting control plane node          |  0.061438 |           0.059247 |
| minikube in cluster minikube           |           |                    |
| * Creating docker container            |  7.802396 |           7.739465 |
| (CPUs=2, Memory=3700MB) ...            |           |                    |
| * Preparing Kubernetes v1.18.0         |  0.000244 |           0.000222 |
| on Docker 19.03.2 ...                  |           |                    |
|   -                                    | 17.670533 |          18.150806 |
| kubeadm.pod-network-cidr=10.244.0.0/16 |           |                    |
| * Verifying Kubernetes                 |  0.844150 |           1.152766 |
| components...                          |           |                    |
| * Enabled addons:                      |  0.002782 |           0.097424 |
| default-storageclass,                  |           |                    |
| storage-provisioner                    |           |                    |
| * Done! kubectl is now                 |  0.072834 |           0.063568 |
| configured to use "minikube"           |           |                    |
|                                        |  0.003906 |           0.004595 |
+----------------------------------------+-----------+--------------------+

medyagh · 2020-04-27T20:00:10Z

@medyagh this seems to be a coincidence. It also doesn't seem like the tests reran, so I'll merge w/ master and we can rerun them

sounds good. please make sure there is a comment explaining every failure associated with an issue.

medyagh

approved with required comments for integration tests failure associated with an issue

priyawadhwa · 2020-04-27T23:28:34Z

Docker_Linux Logs

TestErrorSpam - 198.53s
Same error as #7719

TestPreload - 214.58s

TestStartStop/group/containerd - 273.53s
Seen in #7704

TestStartStop/group/crio - 1000.61s
Same as #7808

TestStartStop/group/embed-certs - 844.77s

start_stop_delete_test.go:141: (dbg) TestStartStop/group/embed-certs: waiting 9m0s for pods matching "k8s-app=kubernetes-dashboard" in namespace "kubernetes-dashboard" ...
start_stop_delete_test.go:141: ***** TestStartStop/group/embed-certs: pod "k8s-app=kubernetes-dashboard" failed to start within 9m0s: timed out waiting for the condition ****

Opened #7921

TestFunctional/parallel/StatusCmd - 5.23s

functional_test.go:313: (dbg) Non-zero exit: out/minikube-linux-amd64 -p functional-20200427175241-5551 status: exit status 2 (2.43335847s)
-- stdout --
	functional-20200427175241-5551
	type: Control Plane
	host: Running
	kubelet: Running
	apiserver: Error
	kubeconfig: Configured
	
-- /stdout --
functional_test.go:315: failed to run minikube status. args "out/minikube-linux-amd64 -p functional-20200427175241-5551 status" : exit status 2

Opened #7922

TestMultiNode/serial/DeleteNode - 5.54s

multinode_test.go:174: (dbg) Non-zero exit: out/minikube-linux-amd64 -p multinode-20200427175733-5551 node delete m03: exit status 69 (4.220365448s)
-- stdout --
	* This control plane is not running! (state=Error)
	  - To fix this, run: "minikube start -p multinode-20200427175733-5551"

Seems related to #7922, as it's also a minikube status error

Comments no longer applicable as this is now behind a flag

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 21, 2020

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 21, 2020

k8s-ci-robot requested review from blueelvis and tstromberg April 21, 2020 00:48

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 21, 2020

medyagh requested changes Apr 21, 2020

View reviewed changes

pkg/minikube/cruntime/docker.go Outdated Show resolved Hide resolved

afbjorklund previously requested changes Apr 21, 2020

View reviewed changes

pkg/minikube/cruntime/docker.go Outdated Show resolved Hide resolved

Priya Wadhwa added 3 commits April 21, 2020 13:14

Merge branch 'master' of https://github.com/kubernetes/minikube into …

89ba35d

…systemd

Set systemd in kicbase Dockerfile

80b331c

Update kic to v0.0.10, upgrade crio and podman so that the kic base i…

3ae85a1

…mage builds

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 21, 2020

medyagh reviewed Apr 22, 2020

View reviewed changes

hack/images/kicbase.Dockerfile Outdated Show resolved Hide resolved

Priya Wadhwa added 3 commits April 22, 2020 14:10

Add --force-systemd flag to force container runtime to use systemd

ae388d0

Update docs

1f2b32c

Add integration test

56fa90f

remove unused code

746d225

Priya Wadhwa added 2 commits April 23, 2020 15:57

Merge branch 'master' of https://github.com/kubernetes/minikube into …

cb6cd27

…systemd

fix lint

cfb1d21

priyawadhwa changed the title ~~Docker driver: force docker to use systemd as cgroup manager~~ Add option to force docker to use systemd as cgroup manager Apr 23, 2020

medyagh reviewed Apr 23, 2020

View reviewed changes

make forceSystemd private

52b5737

Merge branch 'master' of https://github.com/kubernetes/minikube into …

29dc41e

…systemd

Restart docker if we've changed the daemon config

1825546

Merge branch 'master' of https://github.com/kubernetes/minikube into …

71ca99b

…systemd

Merge branch 'master' of https://github.com/kubernetes/minikube into …

e3397d5

…systemd

medyagh approved these changes Apr 27, 2020

View reviewed changes

priyawadhwa merged commit 278b92f into kubernetes:master Apr 28, 2020

priyawadhwa deleted the systemd branch April 28, 2020 01:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add option to force docker to use systemd as cgroup manager #7815

Add option to force docker to use systemd as cgroup manager #7815

priyawadhwa commented Apr 21, 2020 •

edited

Loading

priyawadhwa commented Apr 21, 2020

k8s-ci-robot commented Apr 21, 2020

codecov-io commented Apr 21, 2020 •

edited

Loading

minikube-pr-bot commented Apr 21, 2020

priyawadhwa commented Apr 21, 2020

afbjorklund commented Apr 21, 2020 •

edited

Loading

priyawadhwa commented Apr 21, 2020

minikube-pr-bot commented Apr 21, 2020

priyawadhwa commented Apr 22, 2020

afbjorklund commented Apr 22, 2020

afbjorklund commented Apr 22, 2020

priyawadhwa commented Apr 22, 2020 •

edited

Loading

minikube-pr-bot commented Apr 23, 2020

minikube-pr-bot commented Apr 23, 2020

medyagh left a comment

minikube-pr-bot commented Apr 24, 2020

medyagh commented Apr 24, 2020

medyagh commented Apr 24, 2020

minikube-pr-bot commented Apr 24, 2020

priyawadhwa commented Apr 24, 2020

minikube-pr-bot commented Apr 24, 2020

minikube-pr-bot commented Apr 24, 2020

medyagh commented Apr 25, 2020 •

edited

Loading

medyagh commented Apr 25, 2020

medyagh commented Apr 25, 2020

priyawadhwa commented Apr 27, 2020

minikube-pr-bot commented Apr 27, 2020

medyagh commented Apr 27, 2020

medyagh left a comment

priyawadhwa commented Apr 27, 2020 •

edited

Loading

Add option to force docker to use systemd as cgroup manager #7815

Add option to force docker to use systemd as cgroup manager #7815

Conversation

priyawadhwa commented Apr 21, 2020 • edited Loading

priyawadhwa commented Apr 21, 2020

k8s-ci-robot commented Apr 21, 2020

codecov-io commented Apr 21, 2020 • edited Loading

Codecov Report

minikube-pr-bot commented Apr 21, 2020

priyawadhwa commented Apr 21, 2020

afbjorklund commented Apr 21, 2020 • edited Loading

priyawadhwa commented Apr 21, 2020

minikube-pr-bot commented Apr 21, 2020

priyawadhwa commented Apr 22, 2020

afbjorklund commented Apr 22, 2020

afbjorklund commented Apr 22, 2020

priyawadhwa commented Apr 22, 2020 • edited Loading

minikube-pr-bot commented Apr 23, 2020

minikube-pr-bot commented Apr 23, 2020

medyagh left a comment

Choose a reason for hiding this comment

minikube-pr-bot commented Apr 24, 2020

medyagh commented Apr 24, 2020

medyagh commented Apr 24, 2020

minikube-pr-bot commented Apr 24, 2020

priyawadhwa commented Apr 24, 2020

minikube-pr-bot commented Apr 24, 2020

minikube-pr-bot commented Apr 24, 2020

medyagh commented Apr 25, 2020 • edited Loading

medyagh commented Apr 25, 2020

medyagh commented Apr 25, 2020

priyawadhwa commented Apr 27, 2020

minikube-pr-bot commented Apr 27, 2020

medyagh commented Apr 27, 2020

medyagh left a comment

Choose a reason for hiding this comment

priyawadhwa commented Apr 27, 2020 • edited Loading

priyawadhwa commented Apr 21, 2020 •

edited

Loading

codecov-io commented Apr 21, 2020 •

edited

Loading

afbjorklund commented Apr 21, 2020 •

edited

Loading

priyawadhwa commented Apr 22, 2020 •

edited

Loading

medyagh commented Apr 25, 2020 •

edited

Loading

priyawadhwa commented Apr 27, 2020 •

edited

Loading