Snap package shim not working with containerd

[joedborg]

Description of problem

Trying to connect the shim from the snap package to containerd. Config for containerd taken from https://github.com/kata-containers/documentation/blob/master/how-to/containerd-kata.md

Expected result

Be able to start containerd containers.

Actual result

$ ln -s /snap/kata-containers/288/usr/libexec/kata-containers/kata-shim /usr/bin/containerd-shim-kata-v2  # doing this so containerd picks it up; may be wrong.

$ sudo ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh

ctr: flag provided but not defined: -namespace
Usage of /usr/bin/containerd-shim-kata-v2:
  -agent string
    	agent gRPC socket endpoint
  -agent-logs-socket string
    	socket to listen on to retrieve agent logs
  -container string
    	container id for the shim
  -debug
    	enable debug mode
  -exec-id string
    	process id for the shim
  -log string
    	set shim log level: debug, info, warn, error, fatal or panic (default "warn")
  -proxy-exit-code
    	proxy exit code of the process (default true)
  -terminal
    	specify if a terminal is setup
  -trace
    	enable opentracing support
  -version
    	display program version and exit
: exit status 2: unknown

---

This should be fixed too:

$ sudo /snap/kata-containers/288/usr/bin/kata-collect-data.sh 
ERROR: kata-collect-data.sh: cannot find runtime 'kata-runtime'

[devimc]

@joedborg thanks for raising this, but seems like you are creating a symlink to the wrong shim

you are running

$ ln -s /snap/kata-containers/288/usr/libexec/kata-containers/kata-shim /usr/bin/containerd-shim-kata-v2

when it should be

$ ln -s /snap/kata-containers/288/usr/bin/containerd-shim-kata-v2 /usr/bin/containerd-shim-kata-v2

[devimc]

This should be fixed too:
$ sudo /snap/kata-containers/288/usr/bin/kata-collect-data.sh
ERROR: kata-collect-data.sh: cannot find runtime 'kata-runtime'

you're right

[joedborg]

$ ln -s /snap/kata-containers/288/usr/bin/containerd-shim-kata-v2 /usr/bin/containerd-shim-kata-v2

Ah thanks @devimc, that's my bad. I got this binary by running kata-containers.runtime kata-env and looking at the shim binary in there.

[joedborg]

@devimc have you any idea how to debug or fix this?

sudo ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh
ctr: no such file or directory: not found

[devimc]

@joedborg

try with sudo -E PATH=$PATH ctr run....

[joedborg]

@devimc Thanks, sadly it gives the same error. I think it's coming from something containerd is calling (maybe the shim) rather than from ctr itself. Could be another issue coming from being within the Snapd environment. I did try to install via the bash script, but I get apt dependency issues.

[devimc]

@joedborg that's weird, a symlink should be enough, see https://github.com/kata-containers/packaging/blob/master/.ci/setup.sh#L59-L63

[joedborg]

@devimc I've added all those sym links, with the same error resulting. Is anything obviously wrong?

root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = ""
  uid = 0
  gid = 0
  level = ""

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins.cgroups]
    no_prometheus = false
  [plugins.cri]
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    enable_selinux = false
    sandbox_image = "k8s.gcr.io/pause:3.1"
    stats_collect_period = 10
    systemd_cgroup = false
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    [plugins.cri.containerd]
      no_pivot = false
    [plugins.cri.containerd.runtimes]
      [plugins.cri.containerd.runtimes.runc]
         runtime_type = "io.containerd.runc.v1"
         [plugins.cri.containerd.runtimes.runc.options]
           NoPivotRoot = false
           NoNewKeyring = false
           ShimCgroup = ""
           IoUid = 0
           IoGid = 0
           BinaryName = "runc"
           Root = ""
           CriuPath = ""
           SystemdCgroup = false
      [plugins.cri.containerd.runtimes.kata]
         runtime_type = "io.containerd.kata.v2"
      [plugins.cri.containerd.runtimes.katacli]
         runtime_type = "io.containerd.runc.v1"
         [plugins.cri.containerd.runtimes.katacli.options]
           NoPivotRoot = false
           NoNewKeyring = false
           ShimCgroup = ""
           IoUid = 0
           IoGid = 0
           BinaryName = "/usr/bin/kata-runtime"
           Root = ""
           CriuPath = ""
           SystemdCgroup = false
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
      
    
    [plugins.cri.x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""
  [plugins.diff-service]
    default = ["walking"]
  [plugins.linux]
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false
  [plugins.opt]
    path = "/opt/containerd"
  [plugins.restart]
    interval = "10s"
  [plugins.scheduler]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"

[devimc]

@joedborg in my case the cri configuration is a little bit different

[plugins]
  [plugins.cri]
    [plugins.cri.containerd]
      [plugins.cri.containerd.runtimes.runc]
        runtime_type = "io.containerd.runtime.v1.linux"
        [plugins.cri.containerd.runtimes.runc.options]
          Runtime = "runc"
          RuntimeRoot = "/usr/sbin/runc"
      [plugins.cri.containerd.runtimes.kata]
        runtime_type = "io.containerd.runtime.v1.linux"
        [plugins.cri.containerd.runtimes.kata.options]
          Runtime = "kata-runtime"
          RuntimeRoot = "/usr/local/bin/kata-runtime"

[joedborg]

@devimc I get the same issue with your config :(

[devimc]

cc @chavafg @jodh-intel @egernst @bergwolf @lifupan

[devimc]

@joedborg I filed an issue for the second issue that you reported (#1720) , thanks

[lifupan]

Hi @joedborg , Can you paste the containerd logs?

[joedborg]

Hey @lifupan, here's the segment of log when I'm running ctr run ...

ay 22 13:13:15 ip-172-31-35-209 containerd[26457]: time="2019-05-22T13:13:15.913646756Z" level=debug msg="registering ttrpc server"
May 22 13:13:15 ip-172-31-35-209 containerd[26457]: time="2019-05-22T13:13:15.913793985Z" level=debug msg="serving api on abstract socket" socket="[inherited from parent]"
May 22 13:13:15 ip-172-31-35-209 containerd[26457]: time="2019-05-22T13:13:15.913816322Z" level=info msg="starting signal loop" namespace=default path=/run/containerd/io.containerd.runtime.v2.task/default/hello pid=27227
May 22 13:13:15 ip-172-31-35-209 containerd[26457]: time="2019-05-22T13:13:15.914450628Z" level=debug msg="converting /run/containerd/io.containerd.runtime.v2.task/default/hello/config.json" source=virtcontainers subsystem=oci
May 22 13:13:15 ip-172-31-35-209 kata[27227]: time="2019-05-22T13:13:15.915159907Z" level=info msg="loaded configuration" ID=hello file=/etc/kata-containers/configuration.toml format=TOML source=katautils
May 22 13:13:15 ip-172-31-35-209 containerd[26457]: time="2019-05-22T13:13:15.915159907Z" level=info msg="loaded configuration" ID=hello file=/etc/kata-containers/configuration.toml format=TOML source=katautils

[joedborg]

I've just been through all the manual build steps defined at https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md (will try the debs when kata-containers/kata-containers#44 is fixed and this works fine with containerd, so my belief is that is an issue with the snap package rather than kata.

[egernst]

@devimc any next steps for this? Are you able to reproduce?

[devimc]

@egernst nop, it works for me, @joedborg have you tried with the latest version of kata (1.7.1) ?

[devimc]

any update on this? - closing issue, feel free to reopen it