CI: add nerdctl

[AkihiroSuda]

No description provided.

[AkihiroSuda]

https://github.com/rootless-containers/usernetes/actions/runs/8627684028/job/23648119883?pr=327

Run make kubeadm-init
nerdctl compose exec -e U7S_HOST_IP=10.1.0.133 -e U7S_NODE_NAME=u7s-fv-az569-243 -e U7S_NODE_SUBNET=10.100.227.0/24 -e U7S_NODE_IP=10.100.227.100 node sh -euc "envsubst </usernetes/kubeadm-config.yaml >/tmp/kubeadm-config.yaml"
panic: provided file is not a console

• panic: provided file is not a console containerd/nerdctl#3297

[AkihiroSuda]

https://github.com/rootless-containers/usernetes/actions/runs/8627684028/job/23648119731?pr=327

nerdctl compose up --build -d
time="2024-04-10T07:41:28Z" level=fatal msg="getwd: no such file or directory"
make: *** [Makefile:69: up] Error 1
make: Leaving directory '/home/runner/usernetes'

[apostasie]

Mmmm... did this get autoclosed because of the fix in nerdctl?

@AkihiroSuda did you mean this to be closed, or shall we revive it as soon as we have a new nerdctl RC with the fix?

[AkihiroSuda]

msg="getwd: no such file or directory" still remains

[apostasie]

getwd looked into above ^

I am more baffled by the other failure, which seems to be that (docker) runc is not allowed to pivot root (because... apparmor?). Are we missing a step in the installation flow?

Do you have a clue what's going on with this?

https://github.com/rootless-containers/usernetes/actions/runs/10441177644/job/28911999825?pr=327#step:6:1203

[AkihiroSuda]

The CI is now almost green, except the last "Test data persistency" step
https://github.com/rootless-containers/usernetes/actions/runs/12125498597/job/33806091150?pr=327

[apostasie]

So, Kube API is not responding after lima restart?

Maybe we can duplicate https://github.com/rootless-containers/usernetes/blob/master/.github/workflows/main.yaml#L109C1-L122C80 and add it after - name: "Test data persistency after restarting the node" so that we get better debugging info?

[apostasie]

Grumble grumble that was not super helpful.

Looks like the kube API indeed is not being restarted on reboot (or it fails starting for some reason).

Spitballing hypothesis:

• DNS

We need more debug in there: for eg some compose logs to see what happened with make up?

If you do not get to this before me ^ I'll debug on a separate PR later on.

[AkihiroSuda]

https://github.com/rootless-containers/usernetes/actions/runs/12129252319/job/33817308667?pr=327

Dec 02 22:48:31 lima-host0 containerd-rootless.sh[979]: time="2024-12-02T22:48:31.259129111Z" level=error msg="apply change" error="failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running createRuntime hook #0: exit status 1, stdout: , stderr: time=\"2024-12-02T22:48:31Z\" level=fatal msg=\"bypass4netnsd not running? (Hint: run `containerd-rootless-setuptool.sh install-bypass4netnsd`): bypass4netns failed\""

bypass4netnsd seems not working

[AkihiroSuda]

$ cat ~/.local/share/nerdctl/1935db59/containers/default/320813e981deea8eb35e422fc12ae2ce31897edf9528e20e92dd55f33f35906d/bypass4netns.log 
time="2024-12-03T05:22:16Z" level=info msg="LogFilePath: /home/suda.linux/.local/share/nerdctl/1935db59/containers/default/320813e981deea8eb35e422fc12ae2ce31897edf9528e20e92dd55f33f35906d/bypass4netns.log"
time="2024-12-03T05:22:16Z" level=fatal msg="Cannot write pid file: open /run/user/1001/bypass4netns/320813e981deea8.pid: no such file or directory"

bypass4netns refuses to work because the PID file in /run is missing after rebooting

[apostasie]

So...
It is failing here: https://github.com/rootless-containers/bypass4netns/blob/c6757e6185dba0448b0b63fe9222bd8a5590275a/cmd/bypass4netnsd/main.go#L95-L97

because the containing directory does not exist, right?

mkdirall(parent(pidfile))?

[apostasie]

We must have that problem outside of usernetes.

I think we can fix in nerdctl.

https://github.com/containerd/nerdctl/blob/main/pkg/bypass4netnsutil/bypass4netnsutil.go#L128-L136

[apostasie]

Tentatively: containerd/nerdctl#3724

If you get a chance to try it with this patch, curious if that would fix it.

Presumably, it works in certain cases because CreateSocketDir would have been called before.

[AkihiroSuda]

Thanks @apostasie , will try your patch later 👍