Segmentation fault with --runtime=kata-runtime

[mviereck]

First, thank you for tini!

I currently try out kata with --runtime=kata-runtime instead of docker default runtime runc. (Ticket)

I found that tini fails with a segmentation fault in container, even with tini --version.
I've checked with a shared docker-init.
Version:

$ docker-init --version
tini version 0.18.0 - git.fec3683

Do you have a thought on this?

[krallin]

I'm not very familiar with Kata, but my recollection is that Kata containers are actually VMs, correct? If so, then it might not be entirely surprising that just copying over a binary onto a separate VM doesn't work.

That being said, do you have any debug info here?

Can you check what happens if you run the container as privileged (that might not even be a thing for Kata containers ... but below I'll ask you to run strace and --privileged would be required in a regular container), and assuming that still fails, do the following:

• Create a new Docker image where you download Tini from the releases here, and install strace.
• Run strace -- tini -- true
• Collect the strace output so we can at least get an idea of what Tini was doing when it crashed.

[mviereck]

Thank you for giving some instruction.

A downloaded tini with same --version output works well, no error.
Checking its version in container:

# /tini --version
tini version 0.18.0 - git.fec3683

I've installed docker from the offical Docker repository. It ships with docker-init.
Checking its version on host:

$ docker-init --version
tini version 0.18.0 - git.fec3683

I've shared docker-init with the container.
Surprisingly, docker-init has a quite different binary size:

# ls -l tini docker-init
-rwxr-xr-x 1 root root 764144 Mar 27 18:36 docker-init
-rwx--x--x 1 root root  24064 Apr 21  2018 tini

# /docker-init --version
Segmentation fault

Sharing docker-init works with runc but not with kata:

# strace -- /docker-init -- true
execve("/docker-init", ["/docker-init", "--", "true"], 0x7fff0a3edc18 /* 8 vars */) = -1 ETXTBSY (Text file busy)
+++ killed by SIGSEGV +++
Segmentation fault

Both host and container run debian buster.

I'm not very familiar with Kata, but my recollection is that Kata containers are actually VMs, correct? If so, then it might not be entirely surprising that just copying over a binary onto a separate VM doesn't work.

kata somehow runs in a qemu VM. I am not experienced with it, too. I am running my first test runs. However, sharing files with --volume seems to work in general.

---

For the record:
Dockerfile

FROM debian:buster
RUN apt-get update

ENV TINI_VERSION v0.18.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini

RUN apt-get install -y strace psmisc
CMD bash

Command:

docker run -ti --runtime=kata-runtime --volume /usr/bin/docker-init:/docker-init test bash

[krallin]

Looks like you're hitting ETXTBSY when executing Tini before we even get to actually running Tini (i.e. Tini itself never started: the call to start it failed):

execve("/docker-init", ["/docker-init", "--", "true"], 0x7fff0a3edc18 /* 8 vars */) = -1 ETXTBSY (Text file busy)

My guess is that this issue probably has to do with how mount sharing works with Kata containers. Can you try mout-sharing another binary from your host into your container to compare (e.g. /bin/true).

• If that doesn't work, then that would confirm the issue isn't really Tini specific.
• If that works, can you run lsof /usr/bin/docker-init on your host to see if anything has htis open for writing?

Separately, can you confirm running the tini binary you downloaded in your container works properly?

---

Surprisingly, docker-init has a quite different binary size:

I think you downloaded the dynamically-linked version and you're comparing the size to the statically-built version docker-init uses (they have to do that, because this binary has to work in Docker images that don't have the libraries the dynamically-linked tini binary links to).

[mviereck]

Can you try mout-sharing another binary from your host into your container to compare (e.g. /bin/true).

I have tried --volume /bin/true:/true, it works to run /true with exit code 0.
Furthermore I have downloaded tini-static from your github repo on host and shared it with the container using --volume. It works:

# ls -l tini tini-static docker-init
-rwxr-xr-x 1 root root 764144 Mar 27 18:36 docker-init
-rwx--x--x 1 root root  24064 Apr 21  2018 tini
-rwxr-xr-x 1 1000 1000 870488 Apr 21  2018 tini-static

# /tini-static --version
tini version 0.18.0 - git.fec3683

# /tini-static -s -- pstree
bash---tini-static---pstree

# /tini -s -- pstree
bash---tini---pstree

# /docker-init -s -- pstree
Segmentation fault

It shows the same --version output, but has a slightly different binary size than docker-init.

If that works, can you run lsof /usr/bin/docker-init on your host to see if anything has htis open for writing?

At least it is shared with other containers to use it as init systems. The current output as user:

$ lsof /usr/bin/docker-init
lsof: WARNING: can't stat() overlay file system /sda7docker/overlay2/d8d6e8d14477c64dfa50019a26755a3c5393eac9775082f3abe086288e5d39c5/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /sda7docker/containers/9dbfd932744ea7d541255c86f3b7102881b0a1e8bf9e72927125df426996392e/mounts/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() nsfs file system /run/docker/netns/973203f75f7c
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /sda7docker/overlay2/fc4c8f1df7d6cacfac22e579d9ae4d7498499888a7788d9de107f3a5a35fec23/merged
      Output information may be incomplete.
lsof: WARNING: can't stat() tmpfs file system /sda7docker/containers/c957e75b239447fc83032184b8d8082c343eacc3f90fcf38ac15d52fb329408c/mounts/shm
      Output information may be incomplete.
lsof: WARNING: can't stat() nsfs file system /run/docker/netns/31cc0df940ef
      Output information may be incomplete.
lsof: WARNING: can't stat() overlay file system /run/kata-containers/shared/sandboxes/c957e75b239447fc83032184b8d8082c343eacc3f90fcf38ac15d52fb329408c/c957e75b239447fc83032184b8d8082c343eacc3f90fcf38ac15d52fb329408c/rootfs
      Output information may be incomplete.
lsof: WARNING: can't stat() ext4 file system /run/kata-containers/shared/sandboxes/c957e75b239447fc83032184b8d8082c343eacc3f90fcf38ac15d52fb329408c/c957e75b239447fc83032184b8d8082c343eacc3f90fcf38ac15d52fb329408c-32e13b48be077235-docker-init
      Output information may be incomplete.
lsof: WARNING: can't stat() ext4 file system /run/kata-containers/shared/sandboxes/c957e75b239447fc83032184b8d8082c343eacc3f90fcf38ac15d52fb329408c/c957e75b239447fc83032184b8d8082c343eacc3f90fcf38ac15d52fb329408c-293718519a4c7884-tini-static
      Output information may be incomplete.
lsof: WARNING: can't stat() ext4 file system /run/kata-containers/shared/sandboxes/c957e75b239447fc83032184b8d8082c343eacc3f90fcf38ac15d52fb329408c/c957e75b239447fc83032184b8d8082c343eacc3f90fcf38ac15d52fb329408c-63a148e6900675b8-true
      Output information may be incomplete.
lsof: WARNING: can't stat() ext4 file system /run/kata-containers/shared/sandboxes/c957e75b239447fc83032184b8d8082c343eacc3f90fcf38ac15d52fb329408c/c957e75b239447fc83032184b8d8082c343eacc3f90fcf38ac15d52fb329408c-d3fe9822f5b558ff-resolv.conf
      Output information may be incomplete.
lsof: WARNING: can't stat() ext4 file system /run/kata-containers/shared/sandboxes/c957e75b239447fc83032184b8d8082c343eacc3f90fcf38ac15d52fb329408c/c957e75b239447fc83032184b8d8082c343eacc3f90fcf38ac15d52fb329408c-af8028c7fd311c9a-hostname
      Output information may be incomplete.
lsof: WARNING: can't stat() ext4 file system /run/kata-containers/shared/sandboxes/c957e75b239447fc83032184b8d8082c343eacc3f90fcf38ac15d52fb329408c/c957e75b239447fc83032184b8d8082c343eacc3f90fcf38ac15d52fb329408c-962f0d5047144218-hosts
      Output information may be incomplete.
COMMAND   PID     USER  FD   TYPE DEVICE SIZE/OFF   NODE NAME
tini    23575 lauscher txt    REG    8,2   764144 919045 /usr/bin/docker-init

With sudo:

$ sudo lsof /usr/bin/docker-init
[sudo] Passwort für lauscher: 
COMMAND   PID     USER  FD   TYPE DEVICE SIZE/OFF   NODE NAME
tini    23575 lauscher txt    REG    8,2   764144 919045 /usr/bin/docker-init

I would have expected more/better output with sudo, not less. I am not sure what that means.

I'll stop all containers that use docker-init (including my current browser) and try again.

[mviereck]

Ok, it seems to be not a fault of tini. Sorry for coming up with this.

I am still a bit confused, but it seems to be an issue with docker option --init and how it provides docker-init to containers.

If I share /usr/bin/docker-init with multiple containers and use it as init system, some with runc, some with kata, it works.

But: If I run a container with option --init in runc, I get a segfault in following kata containers that want to access /usr/bin/docker-init.
Other way around (--init with kata, than runc) causes the same issue.

Edit: All that is wrong, too. It seems to be an issue with files that are shared with runtimes kata-runtime in one and runc in another container at the same time. (Shared read-only, though).

Thank you for looking at this!