new containerd config causes KVM Containerd test to time out and Docker Containerd test addons to fail

[medyagh]

After New Containerd Config in this PR https://github.com/kubernetes/minikube/pull/11325/files

the KVM test times out like this in 90 minutes (used to be Green in 63 minutes)

here is full KVM logs that timed out

kvm_log.txt

Also the Docker Driver Test Addons Fail

https://storage.googleapis.com/minikube-builds/logs/11504/4567aaa/Docker_Linux_containerd.html#fail_TestAddons%2fparallel%2fRegistry

The integraiton tests run on Debian 9

jenkins@debian-jenkins-agent-9:~$ uname -r
4.9.0-15-amd64

the New Containerd Config improves the perofmrance on cloud shell

interestingly after that PR the performance chart on cloud shell shows, containerd has same performance as Docker after that PR

the cloud shell linux uses kernel 5

medya@cloudshell:~$ uname -r
5.4.104+

[AkihiroSuda]

If the CI env has DOCKER_RAMDISK, it might not work as expected, though I assume it is no longer set

[AkihiroSuda]

Removing NoPivotRoot = true may work?

[medyagh]

Removing NoPivotRoot = true may work?

do u mean for every case or only kernel 4 (debian 9)?

[medyagh]

If the CI env has DOCKER_RAMDISK, it might not work as expected, though I assume it is no longer set
doesn't seem like we have that ENV set

jenkins@debian-jenkins-agent-9:~$ echo $DOCKER_RAMDISK
jenkins@debian-jenkins-agent-9:~$ env | grep DOCKER

[AkihiroSuda]

Basically, NoPivotRoot is insecure and should never be used on any version of kernel, unless it is really needed (for some initrd stuff).

I added NoPivotRoot in the runtime v2 config PR just because the runtime v1 config had equivalent of that.

I'm not sure this is really related to the CI failure, but I think it is worth trying.

[AkihiroSuda]

echo $DOCKER_RAMDISK

Could you check the env of the dockerd process, not the env of the bash? (/proc/PID/env)

[medyagh]

echo $DOCKER_RAMDISK

Could you check the env of the dockerd process, not the env of the bash? (/proc/PID/env)

sure @AkihiroSuda this is the ENV on the Debian 9 CI (not inside minikube VM itself)

 $ sudo cat /proc/$(pgrep dockerd)/environ 
LANG=en_US.UTF-8PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binNOTIFY_SOCKET=/run/systemd/notifyLISTEN_PID=1006LISTEN_FDS=1
LISTEN_FDNAMES=docker.socketINVOCATION_ID=fd07a5d7394f4bc39f1983aa7a35a550JOURNAL_STREAM=8:11252

[medyagh]

Basically, NoPivotRoot is insecure and should never be used on any version of kernel, unless it is really needed (for some initrd stuff).

I added NoPivotRoot in the runtime v2 config PR just because the runtime v1 config had equivalent of that.

I'm not sure this is really related to the CI failure, but I think it is worth trying.

lets try that

[ilya-zuyev]

May 26 15:19:28 addons-20210526150513-7945 containerd[457]: time="2021-05-26T15:19:28.889667461Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:busybox,Uid:68a93b4a-33aa-4530-bb9a-fd2b06e110d0,Namespace:default,Attempt:0,} failed, error" error="failed to create containerd task: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: rootfs_linux.go:118: jailing process inside rootfs caused: invalid argument: unknown" looks like this causes the tests to fail

[medyagh]

May 26 15:19:28 addons-20210526150513-7945 containerd[457]: time="2021-05-26T15:19:28.889667461Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:busybox,Uid:68a93b4a-33aa-4530-bb9a-fd2b06e110d0,Namespace:default,Attempt:0,} failed, error" error="failed to create containerd task: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: rootfs_linux.go:118: jailing process inside rootfs caused: invalid argument: unknown" looks like this causes the tests to fail

this seems similar to this error containerd/containerd#5510

[medyagh]

After reverting the @AkihiroSuda #11509 the Containerd Performance on Cloud shell is back to "Worse" than docker again. interestingly it also affects the CRIO to be worse too

but it make the KVM Containerd on Debian 9 Pass Green and finish in 66 minutes (vs 90 minutes with few failures)

[sharifelgamal]

Fixed by #11632