Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the default container runtime dynamic #13251

Merged
merged 3 commits into from
Feb 3, 2022

Conversation

afbjorklund
Copy link
Collaborator

@afbjorklund afbjorklund commented Dec 28, 2021

Since the dockershim removal, there is no longer a constant
default container runtime provided by upstream. Only CRI.

Closes #13250

@afbjorklund afbjorklund requested a review from medyagh December 28, 2021 14:15
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 28, 2021
@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 28, 2021
@afbjorklund afbjorklund force-pushed the default-runtime branch 2 times, most recently from c21f216 to 96b8857 Compare December 28, 2021 18:04
Since the dockershim removal, there is no longer a constant
default container runtime provided by upstream. Only CRI.
Copy link
Member

@medyagh medyagh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this PR mean, if minikube default kubernete is 1.24+

minikube start without any args will start with containerd ?

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Jan 12, 2022

minikube start without any args will start with containerd ?

You don't know. It is dynamic and random, depending on k8s version.

// defaultRuntime returns the default container runtime
func defaultRuntime(k8sVersion string) string

It might be docker, it might be containerd. Something following the CRI.
Probably not rkt, but who knows. Maybe it will make a great comeback ?

If you want a special one, you have to use --container-runtime explicitly.


But no changes yet, the minikube default CRI is the same as it was before.

People should still start to write --container-runtime=docker if they need it.

For instance, it is required if you plan on using minikube docker-env later.

@medyagh
Copy link
Member

medyagh commented Jan 12, 2022

minikube start without any args will start with containerd ?

You don't know. It is dynamic and random, depending on k8s version.

so this might not be what some of our embedded users expect like "skaffold" who would expect docker runtime by default, this would make them have to explicitly add the flag....

I assume this means for 1.24+ k8s versions the dynamic will choose containerd ?

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Jan 12, 2022

who would expect docker runtime by default, this would make them have to explicitly add the flag....

they will need to start doing that, since only CRI is guaranteed in future k8s versions

all the special hacks and privileges of "assuming" a docker daemon, are no longer true

I assume this means for 1.24+ k8s versions the dynamic will choose containerd ?

I gave up on that concept, since skaffold is dragging their legs and since docker is still the most popular

anyway, which container runtime should be selected by default was moved into a different issue

once skaffold is on track again, and there are lots of requests, the default could be changed.

@sharifelgamal
Copy link
Collaborator

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Jan 27, 2022
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13251) |
+----------------+----------+---------------------+
| minikube start | 44.8s    | 44.1s               |
| enable ingress | 32.6s    | 33.3s               |
+----------------+----------+---------------------+

Times for minikube start: 48.0s 44.1s 44.2s 44.0s 43.9s
Times for minikube (PR 13251) start: 44.5s 44.0s 43.6s 44.0s 44.2s

Times for minikube ingress: 29.5s 30.0s 41.5s 31.1s 31.0s
Times for minikube (PR 13251) ingress: 25.5s 45.6s 34.0s 32.0s 29.1s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13251) |
+----------------+----------+---------------------+
| minikube start | 26.9s    | 25.9s               |
| enable ingress | 34.6s    | 27.0s               |
+----------------+----------+---------------------+

Times for minikube (PR 13251) start: 25.9s 25.0s 26.0s 25.9s 26.6s
Times for minikube start: 29.6s 27.1s 26.2s 24.9s 26.5s

Times for minikube ingress: 24.5s 26.0s 26.9s 23.9s 71.9s
Times for minikube (PR 13251) ingress: 24.5s 23.0s 27.4s 23.9s 36.4s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13251) |
+----------------+----------+---------------------+
| minikube start | 39.4s    | 40.6s               |
| enable ingress | 26.2s    | 31.6s               |
+----------------+----------+---------------------+

Times for minikube start: 33.3s 40.2s 40.5s 41.6s 41.3s
Times for minikube (PR 13251) start: 34.4s 41.5s 40.8s 44.8s 41.4s

Times for minikube ingress: 31.4s 25.5s 25.4s 25.4s 23.4s
Times for minikube (PR 13251) ingress: 24.9s 26.0s 24.5s 29.9s 52.9s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_macOS TestNetworkPlugins/group/false (gopogh) n/a
Hyperkit_macOS TestNetworkPlugins/group/false (gopogh) n/a
KVM_Linux TestNetworkPlugins/group/false (gopogh) n/a
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/VerifyKubernetesImages (gopogh) 0.00 (chart)
Docker_macOS TestDownloadOnly/v1.23.2/binaries (gopogh) 0.00 (chart)
Docker_macOS TestDownloadOnly/v1.23.3-rc.0/binaries (gopogh) 0.00 (chart)
Docker_macOS TestFunctional/serial/CacheCmd/cache/cache_reload (gopogh) 0.00 (chart)
Docker_macOS TestNetworkPlugins/group/auto/KubeletFlags (gopogh) 0.00 (chart)
Hyperkit_macOS TestDownloadOnly/v1.16.0/binaries (gopogh) 0.00 (chart)
Hyperkit_macOS TestDownloadOnly/v1.23.2/binaries (gopogh) 0.00 (chart)
Hyperkit_macOS TestDownloadOnly/v1.23.3-rc.0/binaries (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/serial/CacheCmd/cache/cache_reload (gopogh) 0.00 (chart)
Hyperkit_macOS TestNetworkPlugins/group/auto/KubeletFlags (gopogh) 0.00 (chart)
KVM_Linux TestDownloadOnly/v1.16.0/binaries (gopogh) 0.00 (chart)
KVM_Linux TestDownloadOnly/v1.16.0/cached-images (gopogh) 0.00 (chart)
KVM_Linux TestDownloadOnly/v1.23.2/binaries (gopogh) 0.00 (chart)
KVM_Linux TestDownloadOnly/v1.23.2/cached-images (gopogh) 0.00 (chart)
KVM_Linux TestDownloadOnly/v1.23.3-rc.0/binaries (gopogh) 0.00 (chart)
KVM_Linux TestDownloadOnly/v1.23.3-rc.0/cached-images (gopogh) 0.00 (chart)
KVM_Linux TestFunctional/serial/CacheCmd/cache/cache_reload (gopogh) 0.00 (chart)
KVM_Linux TestNetworkPlugins/group/auto/KubeletFlags (gopogh) 0.00 (chart)
Hyperkit_macOS TestDownloadOnly/v1.16.0/cached-images (gopogh) 9.09 (chart)
Hyperkit_macOS TestDownloadOnly/v1.23.2/cached-images (gopogh) 9.09 (chart)
Hyperkit_macOS TestDownloadOnly/v1.23.3-rc.0/cached-images (gopogh) 9.09 (chart)
Docker_Linux_containerd TestRunningBinaryUpgrade (gopogh) 11.11 (chart)
KVM_Linux_containerd TestNetworkPlugins/group/enable-default-cni/Start (gopogh) 12.50 (chart)
Docker_macOS TestFunctional/serial/LogsFileCmd (gopogh) 13.04 (chart)
KVM_Linux TestNetworkPlugins/group/enable-default-cni/Start (gopogh) 15.38 (chart)
KVM_Linux_containerd TestNetworkPlugins/group/auto/Start (gopogh) 20.83 (chart)
Docker_macOS TestDownloadOnly/v1.23.2/cached-images (gopogh) 22.22 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@sharifelgamal
Copy link
Collaborator

the DownloadOnly tests are failing because we're not always passing in container-runtime into our tests suites (and therefore into our minikube start calls). this is usually fine since we default back to docker in a normal codepath, but these tests call download.PreloadExists directly, and then passes and an empty string for containerRuntime (since ContainerRuntime() in main_test.go defaults to constants.DefaultContainerRuntime.

the solution i think is to change one line (main_test.go at L165) that to constants.Docker.

@sharifelgamal
Copy link
Collaborator

/ok-to-test

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13251) |
+----------------+----------+---------------------+
| minikube start | 53.2s    | 51.6s               |
| enable ingress | 29.9s    | 29.8s               |
+----------------+----------+---------------------+

Times for minikube start: 54.3s 53.4s 52.6s 52.9s 53.0s
Times for minikube (PR 13251) start: 52.3s 53.1s 52.9s 52.2s 47.6s

Times for minikube ingress: 26.7s 30.6s 30.1s 30.1s 32.0s
Times for minikube (PR 13251) ingress: 26.1s 31.0s 30.1s 30.6s 31.0s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13251) |
+----------------+----------+---------------------+
| minikube start | 26.8s    | 26.6s               |
| enable ingress | 23.6s    | 23.7s               |
+----------------+----------+---------------------+

Times for minikube ingress: 26.4s 23.9s 22.4s 22.5s 22.9s
Times for minikube (PR 13251) ingress: 23.9s 22.9s 24.5s 24.4s 23.0s

Times for minikube start: 26.8s 26.3s 26.7s 27.4s 27.1s
Times for minikube (PR 13251) start: 26.4s 27.4s 26.1s 25.6s 27.7s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13251) |
+----------------+----------+---------------------+
| minikube start | 43.0s    | 42.8s               |
| enable ingress | 20.5s    | 26.0s               |
+----------------+----------+---------------------+

Times for minikube start: 36.4s 45.9s 45.5s 45.3s 42.1s
Times for minikube (PR 13251) start: 45.3s 44.6s 40.9s 41.3s 42.1s

Times for minikube ingress: 20.0s 20.9s 19.4s 18.9s 23.4s
Times for minikube (PR 13251) ingress: 20.9s 37.9s 23.4s 23.4s 24.4s

@kubernetes kubernetes deleted a comment from minikube-pr-bot Feb 3, 2022
@kubernetes kubernetes deleted a comment from minikube-pr-bot Feb 3, 2022
@sharifelgamal sharifelgamal merged commit 423b816 into kubernetes:master Feb 3, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: afbjorklund, sharifelgamal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [afbjorklund,sharifelgamal]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_Linux TestFunctional/serial/ComponentHealth (gopogh) 0.00 (chart)
Docker_Linux TestNoKubernetes/serial/StartNoArgs (gopogh) 4.60 (chart)
KVM_Linux TestStartStop/group/embed-certs/serial/DeployApp (gopogh) 5.15 (chart)
KVM_Linux TestStartStop/group/embed-certs/serial/EnableAddonWhileActive (gopogh) 5.15 (chart)
KVM_Linux TestStartStop/group/embed-certs/serial/FirstStart (gopogh) 5.15 (chart)
Docker_Linux_containerd TestNoKubernetes/serial/Start (gopogh) 6.78 (chart)
KVM_Linux TestNetworkPlugins/group/flannel/Start (gopogh) 9.28 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/cilium/Start (gopogh) 15.38 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/auto/Start (gopogh) 17.31 (chart)
Docker_macOS TestAddons/parallel/MetricsServer (gopogh) 20.43 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/custom-weave/Start (gopogh) 28.85 (chart)
Docker_Linux_containerd TestOffline (gopogh) 30.88 (chart)
Docker_Linux_containerd TestPause/serial/Start (gopogh) 33.82 (chart)
Docker_Linux_containerd TestAddons/parallel/Registry (gopogh) 34.69 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/kindnet/Start (gopogh) 50.94 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh) 52.38 (chart)
Docker_Linux_containerd TestFunctional/parallel/DashboardCmd (gopogh) 52.94 (chart)
Docker_Linux_containerd TestFunctional/parallel/MountCmd/any-port (gopogh) 52.94 (chart)
Docker_Linux_containerd TestFunctional/parallel/MySQL (gopogh) 52.94 (chart)
Docker_Linux_containerd TestFunctional/parallel/PersistentVolumeClaim (gopogh) 52.94 (chart)
Docker_Linux_containerd TestFunctional/parallel/ServiceCmd (gopogh) 52.94 (chart)
Docker_Linux_containerd TestFunctional/parallel/TunnelCmd/serial/AccessDirect (gopogh) 52.94 (chart)
Docker_Linux_containerd TestFunctional/parallel/TunnelCmd/serial/WaitService/Setup (gopogh) 52.94 (chart)
Docker_Linux_containerd TestFunctional/serial/ExtraConfig (gopogh) 52.94 (chart)
Docker_Linux_containerd TestFunctional/serial/SoftStart (gopogh) 52.94 (chart)
Docker_Linux_containerd TestFunctional/serial/StartWithProxy (gopogh) 52.94 (chart)
Docker_Linux_containerd TestMultiNode/serial/DeployApp2Nodes (gopogh) 52.94 (chart)
Docker_Linux_containerd TestMultiNode/serial/PingHostFrom2Pods (gopogh) 52.94 (chart)
Docker_macOS TestNetworkPlugins/group/calico/Start (gopogh) 53.85 (chart)
Docker_macOS TestNetworkPlugins/group/kubenet/Start (gopogh) 61.54 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make default container runtime dynamic
5 participants