Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ISO: Update docker, containerd, and cri-dockerd #15541

Merged
merged 9 commits into from
Jan 24, 2023

Conversation

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Dec 21, 2022
@spowelljr
Copy link
Member Author

ok-to-build-iso

@k8s-ci-robot k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Dec 21, 2022
@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 21, 2022
@minikube-bot
Copy link
Collaborator

Hi @spowelljr, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

@mqasimsarfraz
Copy link
Contributor

@spowelljr do we need to stick to the containerd version supported by docker ? I see there is a new containerd release (v1.6.14) out there already. Does it make sense to use that?

@afbjorklund
Copy link
Collaborator

afbjorklund commented Dec 22, 2022

Historically we used the same versions as Docker, to not have to test/verify it... But it was separated, since then.

	# As of 2019-05, we use upstream containerd so that we may update it independently of docker.

	# As of 2019-01, we use upstream runc so that we may update it independently of docker.

https://download.docker.com/linux/static/stable/

@afbjorklund
Copy link
Collaborator

afbjorklund commented Dec 22, 2022

It would be easier to provision different versions, if the container runtimes were separated from eachother.

@spowelljr
Copy link
Member Author

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Dec 22, 2022
@mqasimsarfraz
Copy link
Contributor

It would be easier to provision different versions, if the container runtimes were separated from eachother.

+1. Cool, makes sense. Thanks for a quick reply. The reason I bought this up was mainly I was wondering if somehow minikube will be impacted by the issue in v1.16.13.

@afbjorklund
Copy link
Collaborator

afbjorklund commented Dec 22, 2022

Sounds like a good reason to upgrade (I forgot if the Buildroot ISO has swap memory controller or not)

But somehow I think that pesky setting is debian-specific ? Anyway, it is enabled by default in recent kernels

@mqasimsarfraz
Copy link
Contributor

mqasimsarfraz commented Dec 22, 2022

Sounds like a good reason to upgrade (I forgot if the Buildroot ISO has swap memory controller or not)

I believe the ISO do have swap memory controller.

But somehow I think that pesky setting is debian-specific ?

Yes (maybe Ubuntu 20.04 as well, need to confirm) so better upgrade I reckon.

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15541) |
+----------------+----------+---------------------+
| minikube start | 52.9s    | 55.1s               |
| enable ingress | 28.1s    | 28.1s               |
+----------------+----------+---------------------+

Times for minikube start: 52.3s 53.3s 53.4s 51.6s 53.9s
Times for minikube (PR 15541) start: 52.4s 56.8s 55.3s 57.8s 53.3s

Times for minikube ingress: 28.7s 28.2s 29.7s 28.7s 25.2s
Times for minikube (PR 15541) ingress: 25.7s 29.8s 28.2s 27.7s 29.2s

docker driver with docker runtime

+-------------------+----------+---------------------+
|      COMMAND      | MINIKUBE | MINIKUBE (PR 15541) |
+-------------------+----------+---------------------+
| minikube start    | 24.9s    | 26.3s               |
| ⚠️  enable ingress | 20.8s    | 26.9s ⚠️             |
+-------------------+----------+---------------------+

Times for minikube ingress: 19.5s 20.0s 20.9s 22.0s 21.5s
Times for minikube (PR 15541) ingress: 20.5s 20.5s 21.0s 50.0s 22.5s

Times for minikube start: 23.9s 24.9s 25.2s 25.9s 24.5s
Times for minikube (PR 15541) start: 25.5s 25.4s 25.5s 25.9s 29.2s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15541) |
+----------------+----------+---------------------+
| minikube start | 24.1s    | 24.2s               |
| enable ingress | 26.4s    | 26.1s               |
+----------------+----------+---------------------+

Times for minikube start: 21.9s 22.2s 32.7s 21.5s 22.1s
Times for minikube (PR 15541) start: 21.9s 22.0s 21.9s 32.9s 22.4s

Times for minikube ingress: 27.0s 26.4s 26.5s 26.0s 26.0s
Times for minikube (PR 15541) ingress: 26.0s 26.0s 26.4s 26.0s 26.0s

@mqasimsarfraz
Copy link
Contributor

mqasimsarfraz commented Dec 22, 2022

Anyway, it is enabled by default in recent kernels.

Ah okay, got it! Just for understanding, would upgrading the version here also updates it for docker driver? Like kickbase image?

[Answer] Found it, kickbase image will use latest available containerd version while building image so all good on that.

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
KVM_Linux TestRunningBinaryUpgrade (gopogh) 0.00 (chart)
KVM_Linux TestStoppedBinaryUpgrade/MinikubeLogs (gopogh) 0.00 (chart)
KVM_Linux TestStoppedBinaryUpgrade/Upgrade (gopogh) 0.00 (chart)
Docker_macOS TestPause/serial/SecondStartNoReconfiguration (gopogh) 18.46 (chart)
Docker_macOS TestStartStop/group/newest-cni/serial/Pause (gopogh) 25.17 (chart)
KVM_Linux TestMultiNode/serial/RestartMultiNode (gopogh) 41.43 (chart)
Hyper-V_Windows TestNetworkPlugins/group/calico/Start (gopogh) 50.59 (chart)
Hyper-V_Windows TestNetworkPlugins/group/bridge/Start (gopogh) 51.76 (chart)
Hyper-V_Windows TestNetworkPlugins/group/enable-default-cni/Start (gopogh) 51.76 (chart)
Hyper-V_Windows TestNetworkPlugins/group/flannel/Start (gopogh) 51.76 (chart)
Hyper-V_Windows TestStartStop/group/default-k8s-diff-port/serial/Pause (gopogh) 51.76 (chart)
Hyper-V_Windows TestNetworkPlugins/group/kubenet/Start (gopogh) 54.12 (chart)
Docker_macOS TestMultiNode/serial/RestartKeepsNodes (gopogh) 60.51 (chart)
Docker_Linux TestNetworkPlugins/group/calico/Start (gopogh) 70.68 (chart)
Docker_Linux TestNetworkPlugins/group/bridge/DNS (gopogh) 77.44 (chart)
Docker_Linux TestNetworkPlugins/group/enable-default-cni/DNS (gopogh) 77.44 (chart)
Docker_Linux TestNetworkPlugins/group/kubenet/DNS (gopogh) 79.70 (chart)
Docker_macOS TestIngressAddonLegacy/serial/ValidateIngressDNSAddonActivation (gopogh) 98.09 (chart)
Docker_macOS TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh) 100.00 (chart)
Docker_macOS TestIngressAddonLegacy/serial/ValidateIngressAddons (gopogh) 100.00 (chart)
Docker_macOS TestIngressAddonLegacy/StartLegacyK8sCluster (gopogh) 100.00 (chart)
Docker_macOS TestKubernetesUpgrade (gopogh) 100.00 (chart)
Docker_macOS TestMissingContainerUpgrade (gopogh) 100.00 (chart)
Docker_macOS TestNetworkPlugins/group/kubenet/HairPin (gopogh) 100.00 (chart)
Docker_macOS TestRunningBinaryUpgrade (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/AddonExistsAfterStop (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/EnableAddonWhileActive (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/FirstStart (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh) 100.00 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@spowelljr
Copy link
Member Author

ok-to-build-iso

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15541) |
+----------------+----------+---------------------+
| minikube start | 54.7s    | 55.0s               |
| enable ingress | 26.0s    | 27.2s               |
+----------------+----------+---------------------+

Times for minikube start: 54.1s 55.9s 54.8s 53.2s 55.7s
Times for minikube (PR 15541) start: 54.7s 54.2s 54.9s 56.6s 54.4s

Times for minikube ingress: 26.7s 23.2s 26.7s 25.7s 27.7s
Times for minikube (PR 15541) ingress: 25.6s 28.1s 26.2s 27.2s 28.7s

docker driver with docker runtime

+-------------------+----------+---------------------+
|      COMMAND      | MINIKUBE | MINIKUBE (PR 15541) |
+-------------------+----------+---------------------+
| minikube start    | 26.0s    | 24.7s               |
| ⚠️  enable ingress | 20.8s    | 32.8s ⚠️             |
+-------------------+----------+---------------------+

Times for minikube start: 26.3s 25.9s 24.9s 28.3s 24.8s
Times for minikube (PR 15541) start: 24.8s 24.4s 24.8s 24.9s 24.7s

Times for minikube ingress: 21.0s 21.0s 20.9s 21.4s 19.5s
Times for minikube (PR 15541) ingress: 19.4s 81.5s 20.4s 21.5s 21.0s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15541) |
+----------------+----------+---------------------+
| minikube start | 25.1s    | 22.9s               |
| enable ingress | 26.3s    | 26.4s               |
+----------------+----------+---------------------+

Times for minikube start: 24.7s 22.2s 33.4s 22.6s 22.7s
Times for minikube (PR 15541) start: 22.2s 21.1s 24.5s 21.9s 24.9s

Times for minikube (PR 15541) ingress: 26.5s 25.9s 26.5s 26.5s 26.5s
Times for minikube ingress: 25.9s 26.4s 26.4s 26.5s 26.4s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Hyper-V_Windows TestMultiNode/serial/PingHostFrom2Pods (gopogh) n/a
Hyper-V_Windows TestMultiNode/serial/RestartKeepsNodes (gopogh) n/a
Hyper-V_Windows TestNetworkPlugins/group/bridge/Start (gopogh) n/a
Hyper-V_Windows TestNetworkPlugins/group/calico/Start (gopogh) n/a
Hyper-V_Windows TestNetworkPlugins/group/enable-default-cni/Start (gopogh) n/a
Hyper-V_Windows TestNetworkPlugins/group/flannel/Start (gopogh) n/a
Hyper-V_Windows TestNetworkPlugins/group/kubenet/Start (gopogh) n/a
Hyper-V_Windows TestNoKubernetes/serial/StartWithStopK8s (gopogh) n/a
Hyper-V_Windows TestStartStop/group/default-k8s-diff-port/serial/Pause (gopogh) n/a
Docker_macOS TestFunctional/parallel/UpdateContextCmd/no_minikube_cluster (gopogh) 2.56 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/DeployApp (gopogh) 14.05 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/UserAppExistsAfterStop (gopogh) 14.05 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/FirstStart (gopogh) 14.75 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/SecondStart (gopogh) 15.45 (chart)
KVM_Linux TestPause/serial/SecondStartNoReconfiguration (gopogh) 37.61 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/FirstStart (gopogh) 40.50 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh) 40.50 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh) 40.98 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh) 40.98 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/UserAppExistsAfterStop (gopogh) 50.41 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/DeployApp (gopogh) 51.22 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/SecondStart (gopogh) 51.22 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/FirstStart (gopogh) 51.61 (chart)
Docker_Linux TestNetworkPlugins/group/calico/Start (gopogh) 68.60 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/enable-default-cni/DNS (gopogh) 73.23 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/bridge/DNS (gopogh) 74.81 (chart)
Docker_Linux TestNetworkPlugins/group/bridge/DNS (gopogh) 85.12 (chart)
Docker_Linux TestNetworkPlugins/group/enable-default-cni/DNS (gopogh) 85.12 (chart)
Docker_Linux TestNetworkPlugins/group/false/DNS (gopogh) 85.12 (chart)
Docker_Linux TestNetworkPlugins/group/kubenet/DNS (gopogh) 87.60 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@minikube-bot
Copy link
Collaborator

Hi @spowelljr, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15541) |
+----------------+----------+---------------------+
| minikube start | 55.0s    | 55.7s               |
| enable ingress | 27.1s    | 28.1s               |
+----------------+----------+---------------------+

Times for minikube (PR 15541) start: 58.1s 55.8s 54.6s 55.9s 54.3s
Times for minikube start: 54.0s 55.1s 54.0s 55.1s 57.0s

Times for minikube (PR 15541) ingress: 29.1s 28.2s 28.7s 28.2s 26.2s
Times for minikube ingress: 28.2s 25.2s 26.2s 28.2s 27.7s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15541) |
+----------------+----------+---------------------+
| minikube start | 25.9s    | 25.2s               |
| enable ingress | 21.4s    | 21.3s               |
+----------------+----------+---------------------+

Times for minikube start: 24.6s 26.0s 25.8s 25.8s 27.1s
Times for minikube (PR 15541) start: 25.4s 25.5s 24.6s 25.2s 25.1s

Times for minikube ingress: 21.0s 20.5s 21.0s 22.0s 22.5s
Times for minikube (PR 15541) ingress: 22.0s 20.5s 21.5s 22.0s 20.5s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15541) |
+----------------+----------+---------------------+
| minikube start | 22.3s    | 22.9s               |
| enable ingress | 26.3s    | 26.1s               |
+----------------+----------+---------------------+

Times for minikube start: 22.1s 21.8s 22.5s 23.0s 22.2s
Times for minikube (PR 15541) start: 22.5s 22.3s 22.5s 21.7s 25.6s

Times for minikube ingress: 26.5s 26.5s 26.0s 26.5s 26.0s
Times for minikube (PR 15541) ingress: 26.0s 26.5s 26.0s 26.0s 26.0s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Hyper-V_Windows TestMultiNode/serial/PingHostFrom2Pods (gopogh) n/a
Hyper-V_Windows TestMultiNode/serial/RestartKeepsNodes (gopogh) n/a
Hyper-V_Windows TestNetworkPlugins/group/bridge/Start (gopogh) n/a
Hyper-V_Windows TestNetworkPlugins/group/calico/Start (gopogh) n/a
Hyper-V_Windows TestNetworkPlugins/group/enable-default-cni/Start (gopogh) n/a
Hyper-V_Windows TestNetworkPlugins/group/flannel/Start (gopogh) n/a
Hyper-V_Windows TestNetworkPlugins/group/kubenet/Start (gopogh) n/a
Hyper-V_Windows TestNoKubernetes/serial/StartWithStopK8s (gopogh) n/a
Hyper-V_Windows TestStartStop/group/default-k8s-diff-port/serial/Pause (gopogh) n/a
none_Linux TestAddons/parallel/MetricsServer (gopogh) 0.00 (chart)
KVM_Linux_containerd TestRunningBinaryUpgrade (gopogh) 1.27 (chart)
Docker_macOS TestPause/serial/SecondStartNoReconfiguration (gopogh) 9.60 (chart)
Hyperkit_macOS TestPause/serial/SecondStartNoReconfiguration (gopogh) 30.41 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/FirstStart (gopogh) 40.71 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh) 40.71 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh) 41.23 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh) 41.23 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/FirstStart (gopogh) 43.36 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/UserAppExistsAfterStop (gopogh) 43.36 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/DeployApp (gopogh) 43.86 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/SecondStart (gopogh) 45.30 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/UserAppExistsAfterStop (gopogh) 49.56 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/DeployApp (gopogh) 50.43 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/SecondStart (gopogh) 50.43 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-diff-port/serial/FirstStart (gopogh) 51.28 (chart)
Docker_macOS TestMultiNode/serial/RestartKeepsNodes (gopogh) 52.08 (chart)
Docker_Linux TestNetworkPlugins/group/calico/Start (gopogh) 68.70 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/enable-default-cni/DNS (gopogh) 73.73 (chart)
Docker_Linux_containerd TestNetworkPlugins/group/bridge/DNS (gopogh) 74.59 (chart)
Docker_Linux TestNetworkPlugins/group/false/DNS (gopogh) 86.09 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 19, 2023
@spowelljr
Copy link
Member Author

ok-to-build-iso

@spowelljr
Copy link
Member Author

ok-to-build-image

@spowelljr spowelljr changed the title ISO: Update docker and containerd ISO: Update docker, containerd, and cri-dockerd Jan 19, 2023
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: afbjorklund, medyagh, mqasimsarfraz, spowelljr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [afbjorklund,medyagh,spowelljr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@minikube-bot
Copy link
Collaborator

Hi @spowelljr, we have updated your PR with the reference to newly built kicbase image. Pull the changes locally if you want to test with them or update your PR further.

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15541) |
+----------------+----------+---------------------+
| minikube start | 56.0s    | 55.4s               |
| enable ingress | 24.9s    | 27.1s               |
+----------------+----------+---------------------+

Times for minikube ingress: 24.6s 25.6s 24.6s 24.6s 25.1s
Times for minikube (PR 15541) ingress: 28.1s 24.7s 25.1s 28.2s 29.6s

Times for minikube start: 55.9s 56.7s 57.0s 55.2s 55.2s
Times for minikube (PR 15541) start: 55.9s 55.8s 55.1s 55.1s 55.1s

docker driver with docker runtime
error collecting results for docker driver: timing run 0 with minikube: timing cmd: [out/minikube addons enable ingress]: waiting for minikube: exit status 10
docker driver with containerd runtime
error collecting results for docker driver: timing run 0 with minikube: timing cmd: [out/minikube addons enable ingress]: waiting for minikube: exit status 10

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15541) |
+----------------+----------+---------------------+
| minikube start | 58.7s    | 58.5s               |
| enable ingress | 28.4s    | 28.5s               |
+----------------+----------+---------------------+

Times for minikube start: 58.0s 60.0s 57.5s 58.1s 59.7s
Times for minikube (PR 15541) start: 58.2s 57.9s 59.2s 57.6s 59.5s

Times for minikube ingress: 28.7s 28.7s 27.3s 29.2s 28.2s
Times for minikube (PR 15541) ingress: 29.2s 29.7s 29.7s 28.7s 25.2s

docker driver with docker runtime

+-------------------+----------+---------------------+
|      COMMAND      | MINIKUBE | MINIKUBE (PR 15541) |
+-------------------+----------+---------------------+
| minikube start    | 28.2s    | 30.7s               |
| ⚠️  enable ingress | 21.6s    | 32.3s ⚠️             |
+-------------------+----------+---------------------+

Times for minikube start: 27.7s 26.3s 27.5s 28.5s 30.8s
Times for minikube (PR 15541) start: 39.2s 27.5s 29.0s 27.0s 30.9s

Times for minikube ingress: 22.0s 21.0s 20.0s 21.5s 23.5s
Times for minikube (PR 15541) ingress: 19.0s 21.5s 81.0s 21.0s 19.0s

docker driver with containerd runtime

+-------------------+----------+---------------------+
|      COMMAND      | MINIKUBE | MINIKUBE (PR 15541) |
+-------------------+----------+---------------------+
| minikube start    | 24.6s    | 22.8s               |
| ⚠️  enable ingress | 36.8s    | 42.5s ⚠️             |
+-------------------+----------+---------------------+

Times for minikube start: 22.0s 22.7s 22.7s 33.5s 22.2s
Times for minikube (PR 15541) start: 22.2s 22.9s 23.0s 22.5s 23.5s

Times for minikube ingress: 80.0s 19.5s 19.5s 32.5s 32.5s
Times for minikube (PR 15541) ingress: 47.5s 32.5s 20.5s 32.5s 79.5s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_Windows TestKicCustomNetwork/create_custom_network (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/CertSync (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/DockerEnv/bash (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/ImageCommands/ImageBuild (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/ImageCommands/ImageListJson (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/ImageCommands/ImageListShort (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/ImageCommands/ImageListTable (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/ImageCommands/ImageListYaml (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/ImageCommands/ImageLoadDaemon (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/ImageCommands/ImageLoadFromFile (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/ImageCommands/ImageReloadDaemon (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/ImageCommands/ImageSaveToFile (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/ImageCommands/ImageTagAndLoadDaemon (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/MountCmd/any-port (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/MySQL (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/NodeLabels (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/PersistentVolumeClaim (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/ServiceCmd (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/ServiceCmdConnect (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/StatusCmd (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/TunnelCmd/serial/AccessDirect (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/TunnelCmd/serial/AccessThroughDNS (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/TunnelCmd/serial/DNSResolutionByDig (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/TunnelCmd/serial/WaitService/Setup (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/parallel/UpdateContextCmd/no_changes (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/serial/CacheCmd/cache/cache_reload (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/serial/CacheCmd/cache/verify_cache_inside_node (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/serial/ComponentHealth (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/serial/ExtraConfig (gopogh) 0.00 (chart)
Hyperkit_macOS TestFunctional/serial/KubeContext (gopogh) 0.00 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_Linux_containerd TestForceSystemdEnv (gopogh) 0.00 (chart)
Docker_Linux_containerd TestForceSystemdFlag (gopogh) 0.00 (chart)
Docker_Linux_containerd TestFunctional/parallel/MountCmd/specific-port (gopogh) 0.00 (chart)
Docker_Linux TestKVMDriverInstallOrUpdate (gopogh) 0.00 (chart)
Hyper-V_Windows TestStoppedBinaryUpgrade/Upgrade (gopogh) 1.52 (chart)
Docker_Windows TestMultiNode/serial/RestartKeepsNodes (gopogh) 2.78 (chart)
KVM_Linux_containerd TestFunctional/serial/LogsFileCmd (gopogh) 6.04 (chart)
Hyperkit_macOS TestPause/serial/SecondStartNoReconfiguration (gopogh) 31.21 (chart)
Docker_macOS TestIngressAddonLegacy/serial/ValidateIngressDNSAddonActivation (gopogh) 99.30 (chart)
Docker_macOS TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh) 100.00 (chart)
Docker_macOS TestIngressAddonLegacy/serial/ValidateIngressAddons (gopogh) 100.00 (chart)
Docker_macOS TestIngressAddonLegacy/StartLegacyK8sCluster (gopogh) 100.00 (chart)
Docker_macOS TestKubernetesUpgrade (gopogh) 100.00 (chart)
Docker_macOS TestMissingContainerUpgrade (gopogh) 100.00 (chart)
Docker_macOS TestRunningBinaryUpgrade (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/AddonExistsAfterStop (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/EnableAddonWhileActive (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/FirstStart (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh) 100.00 (chart)
Docker_macOS TestStoppedBinaryUpgrade/Upgrade (gopogh) 100.00 (chart)
Docker_Windows TestFunctional/parallel/ServiceCmd (gopogh) 100.00 (chart)
Hyper-V_Windows TestMultiNode/serial/PingHostFrom2Pods (gopogh) 100.00 (chart)
Hyper-V_Windows TestMultiNode/serial/RestartKeepsNodes (gopogh) 100.00 (chart)
Hyper-V_Windows TestNoKubernetes/serial/StartWithStopK8s (gopogh) 100.00 (chart)
KVM_Linux_containerd TestPreload (gopogh) 100.00 (chart)

To see the flake rates of all tests by environment, click here.

@minikube-bot
Copy link
Collaborator

Hi @spowelljr, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15541) |
+----------------+----------+---------------------+
| minikube start | 59.2s    | 57.2s               |
| enable ingress | 29.9s    | 30.1s               |
+----------------+----------+---------------------+

Times for minikube start: 57.2s 59.8s 59.4s 60.7s 59.1s
Times for minikube (PR 15541) start: 55.9s 57.6s 57.0s 57.8s 57.7s

Times for minikube ingress: 29.8s 29.8s 30.3s 29.4s 30.3s
Times for minikube (PR 15541) ingress: 30.8s 28.8s 30.9s 29.4s 30.8s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15541) |
+----------------+----------+---------------------+
| minikube start | 29.8s    | 31.5s               |
| enable ingress | 22.5s    | 22.0s               |
+----------------+----------+---------------------+

Times for minikube ingress: 25.0s 22.1s 22.0s 22.0s 21.1s
Times for minikube (PR 15541) ingress: 21.1s 22.1s 22.0s 22.1s 22.5s

Times for minikube start: 32.0s 29.3s 29.4s 31.2s 27.2s
Times for minikube (PR 15541) start: 30.7s 33.0s 29.4s 32.1s 32.2s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15541) |
+----------------+----------+---------------------+
| minikube start | 24.2s    | 24.5s               |
| enable ingress | 41.5s    | 36.0s               |
+----------------+----------+---------------------+

Times for minikube ingress: 79.5s 31.5s 30.5s 33.5s 32.5s
Times for minikube (PR 15541) ingress: 32.6s 47.6s 33.6s 32.5s 33.5s

Times for minikube start: 24.5s 26.6s 23.2s 23.1s 23.5s
Times for minikube (PR 15541) start: 25.7s 26.7s 24.4s 23.0s 22.9s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_Linux_containerd TestForceSystemdEnv (gopogh) 0.00 (chart)
Docker_Linux_containerd TestForceSystemdFlag (gopogh) 0.00 (chart)
KVM_Linux_containerd TestForceSystemdEnv (gopogh) 0.00 (chart)
KVM_Linux_containerd TestForceSystemdFlag (gopogh) 0.00 (chart)
KVM_Linux_containerd TestJSONOutput/stop/parallel/DistinctCurrentSteps (gopogh) 0.00 (chart)
KVM_Linux_containerd TestJSONOutput/stop/parallel/IncreasingCurrentSteps (gopogh) 0.00 (chart)
Docker_Linux_containerd TestFunctional/parallel/ConfigCmd (gopogh) 0.66 (chart)
KVM_Linux_containerd TestRunningBinaryUpgrade (gopogh) 1.33 (chart)
Docker_Linux TestMultiNode/serial/RestartKeepsNodes (gopogh) 1.94 (chart)
Hyper-V_Windows TestStoppedBinaryUpgrade/Upgrade (gopogh) 2.99 (chart)
Hyper-V_Windows TestNoKubernetes/serial/StartWithK8s (gopogh) 10.45 (chart)
KVM_Linux TestPause/serial/SecondStartNoReconfiguration (gopogh) 48.04 (chart)
Docker_Windows TestStartStop/group/newest-cni/serial/Pause (gopogh) 65.75 (chart)
Docker_macOS TestIngressAddonLegacy/serial/ValidateIngressDNSAddonActivation (gopogh) 99.30 (chart)
Docker_macOS TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh) 100.00 (chart)
Docker_macOS TestIngressAddonLegacy/serial/ValidateIngressAddons (gopogh) 100.00 (chart)
Docker_macOS TestIngressAddonLegacy/StartLegacyK8sCluster (gopogh) 100.00 (chart)
Docker_macOS TestKubernetesUpgrade (gopogh) 100.00 (chart)
Docker_macOS TestMissingContainerUpgrade (gopogh) 100.00 (chart)
Docker_macOS TestRunningBinaryUpgrade (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/AddonExistsAfterStop (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/EnableAddonWhileActive (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/FirstStart (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh) 100.00 (chart)
Docker_macOS TestStoppedBinaryUpgrade/Upgrade (gopogh) 100.00 (chart)
Docker_Windows TestFunctional/parallel/ServiceCmd (gopogh) 100.00 (chart)
Hyper-V_Windows TestMultiNode/serial/PingHostFrom2Pods (gopogh) 100.00 (chart)
Hyper-V_Windows TestMultiNode/serial/RestartKeepsNodes (gopogh) 100.00 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jan 20, 2023
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15541) |
+----------------+----------+---------------------+
| minikube start | 57.0s    | 54.4s               |
| enable ingress | 29.2s    | 28.7s               |
+----------------+----------+---------------------+

Times for minikube ingress: 28.8s 30.2s 28.8s 29.2s 29.3s
Times for minikube (PR 15541) ingress: 30.2s 28.7s 27.7s 28.7s 28.2s

Times for minikube (PR 15541) start: 54.1s 54.6s 54.4s 54.3s 54.5s
Times for minikube start: 56.7s 57.2s 59.0s 55.7s 56.4s

docker driver with docker runtime

+-------------------+----------+---------------------+
|      COMMAND      | MINIKUBE | MINIKUBE (PR 15541) |
+-------------------+----------+---------------------+
| minikube start    | 26.8s    | 27.4s               |
| ⚠️  enable ingress | 27.1s    | 33.6s ⚠️             |
+-------------------+----------+---------------------+

Times for minikube start: 26.5s 25.0s 27.4s 25.2s 29.9s
Times for minikube (PR 15541) start: 26.7s 27.9s 26.0s 26.6s 29.7s

Times for minikube ingress: 22.5s 21.0s 49.5s 20.5s 22.0s
Times for minikube (PR 15541) ingress: 82.5s 21.5s 20.0s 21.0s 23.0s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15541) |
+----------------+----------+---------------------+
| minikube start | 23.2s    | 22.1s               |
| enable ingress | 49.3s    | 32.1s               |
+----------------+----------+---------------------+

Times for minikube start: 22.3s 21.8s 24.7s 24.7s 22.4s
Times for minikube (PR 15541) start: 22.7s 22.4s 22.0s 21.4s 21.9s

Times for minikube ingress: 32.5s 20.0s 81.0s 32.5s 80.5s
Times for minikube (PR 15541) ingress: 32.0s 32.5s 33.0s 29.5s 33.5s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Hyper-V_Windows TestFunctional/parallel/TunnelCmd/serial (gopogh) n/a
Hyper-V_Windows TestFunctional/parallel/UpdateContextCmd (gopogh) n/a
Docker_Linux_containerd TestForceSystemdEnv (gopogh) 0.00 (chart)
Docker_Linux_containerd TestForceSystemdFlag (gopogh) 0.00 (chart)
Hyperkit_macOS TestNoKubernetes/serial/StartWithK8s (gopogh) 0.00 (chart)
Hyperkit_macOS TestRunningBinaryUpgrade (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/DockerEnv/powershell (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/FileSync (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/ImageCommands/ImageBuild (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/ImageCommands/ImageListJson (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/ImageCommands/ImageListShort (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/ImageCommands/ImageListTable (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/ImageCommands/ImageListYaml (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/ImageCommands/ImageLoadDaemon (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/ImageCommands/ImageLoadFromFile (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/ImageCommands/ImageReloadDaemon (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/ImageCommands/ImageSaveToFile (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/ImageCommands/ImageTagAndLoadDaemon (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/MySQL (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/NodeLabels (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/PersistentVolumeClaim (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/ServiceCmd (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/ServiceCmdConnect (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/SSHCmd (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/StatusCmd (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/Version/components (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/parallel/Version/short (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/serial/ComponentHealth (gopogh) 0.00 (chart)
Hyper-V_Windows TestFunctional/serial/LogsFileCmd (gopogh) 0.00 (chart)
Hyper-V_Windows TestStartStop/group/newest-cni/serial/EnableAddonWhileActive (gopogh) 0.00 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15541) |
+----------------+----------+---------------------+
| minikube start | 55.5s    | 53.5s               |
| enable ingress | 27.5s    | 26.7s               |
+----------------+----------+---------------------+

Times for minikube start: 55.0s 54.4s 55.2s 57.1s 55.6s
Times for minikube (PR 15541) start: 54.4s 53.5s 53.6s 52.6s 53.3s

Times for minikube ingress: 25.7s 28.7s 28.7s 24.1s 30.1s
Times for minikube (PR 15541) ingress: 25.1s 28.7s 28.6s 25.7s 25.7s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 15541) |
+----------------+----------+---------------------+
| minikube start | 25.9s    | 25.3s               |
| enable ingress | 39.3s    | 27.0s               |
+----------------+----------+---------------------+

Times for minikube start: 24.8s 26.9s 25.9s 26.1s 25.9s
Times for minikube (PR 15541) start: 24.4s 25.2s 24.9s 26.0s 25.9s

Times for minikube ingress: 22.9s 20.9s 21.9s 81.4s 49.4s
Times for minikube (PR 15541) ingress: 23.0s 20.0s 49.9s 20.0s 21.9s

docker driver with containerd runtime

+-------------------+----------+---------------------+
|      COMMAND      | MINIKUBE | MINIKUBE (PR 15541) |
+-------------------+----------+---------------------+
| minikube start    | 22.7s    | 21.8s               |
| ⚠️  enable ingress | 31.2s    | 60.7s ⚠️             |
+-------------------+----------+---------------------+

Times for minikube start: 21.8s 23.8s 21.9s 21.8s 24.4s
Times for minikube (PR 15541) start: 21.6s 21.9s 21.2s 22.0s 22.3s

Times for minikube ingress: 32.9s 47.9s 19.9s 22.0s 33.5s
Times for minikube (PR 15541) ingress: 29.5s 32.0s 80.9s 79.9s 80.9s

@medyagh medyagh merged commit d27a234 into kubernetes:master Jan 24, 2023
@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Hyperkit_macOS TestStartStop/group/default-k8s-diff-port/serial/AddonExistsAfterStop (gopogh) 0.69 (chart)
Hyperkit_macOS TestStartStop/group/default-k8s-diff-port/serial/Pause (gopogh) 0.69 (chart)
Hyperkit_macOS TestStartStop/group/default-k8s-diff-port/serial/SecondStart (gopogh) 0.69 (chart)
Hyperkit_macOS TestStartStop/group/default-k8s-diff-port/serial/UserAppExistsAfterStop (gopogh) 0.69 (chart)
Hyperkit_macOS TestStartStop/group/default-k8s-diff-port/serial/VerifyKubernetesImages (gopogh) 0.69 (chart)
Hyper-V_Windows TestRunningBinaryUpgrade (gopogh) 34.62 (chart)
Hyper-V_Windows TestStoppedBinaryUpgrade/Upgrade (gopogh) 37.50 (chart)
Docker_macOS TestIngressAddonLegacy/serial/ValidateIngressDNSAddonActivation (gopogh) 99.30 (chart)
Docker_macOS TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh) 100.00 (chart)
Docker_macOS TestIngressAddonLegacy/serial/ValidateIngressAddons (gopogh) 100.00 (chart)
Docker_macOS TestIngressAddonLegacy/StartLegacyK8sCluster (gopogh) 100.00 (chart)
Docker_macOS TestKubernetesUpgrade (gopogh) 100.00 (chart)
Docker_macOS TestMissingContainerUpgrade (gopogh) 100.00 (chart)
Docker_macOS TestRunningBinaryUpgrade (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/AddonExistsAfterStop (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/EnableAddonWhileActive (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/FirstStart (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh) 100.00 (chart)
Docker_macOS TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh) 100.00 (chart)
Docker_macOS TestStoppedBinaryUpgrade/Upgrade (gopogh) 100.00 (chart)
Hyper-V_Windows TestMultiNode/serial/PingHostFrom2Pods (gopogh) 100.00 (chart)
Hyper-V_Windows TestMultiNode/serial/RestartKeepsNodes (gopogh) 100.00 (chart)
Hyper-V_Windows TestNoKubernetes/serial/StartWithStopK8s (gopogh) 100.00 (chart)
KVM_Linux_containerd TestPreload (gopogh) 100.00 (chart)

To see the flake rates of all tests by environment, click here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants