failed to init mount rootfs when running pouch container

[yeya24]

Ⅰ. Issue Description

I try to build a pouch environment by kind. I use the current kind node image (which is based on ubuntu:19.04) to install pouch.
But I failed to run containers. I started pouchd by pouchd -D.

root@kind-control-plane:/lib/systemd/system# pouch run -it docker.io/library/busybox:latest sh
Error: failed to run container: {"message":"failed to init container storage, id: (0ddad604a39b14f16f49740a2f96af2e818f99b4872d72f42e9d661a034cd7b2): failed to mount rootfs(/var/lib/pouch/containers/0ddad604a39b14f16f49740a2f96af2e818f99b4872d72f42e9d661a034cd7b2/rootfs): invalid argument"}

Because I use -D flag so here are some logs via journalctl -xefu pouch for pouchd and containerd.

May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33.541461961Z" level=debug msg="Calling GET /v1.24/images/docker.io/library/busybox:latest/json, client @"
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33.573062194Z" level=info msg="Calling POST /v1.24/containers/create, client @"
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33.573804048Z" level=info msg="create container with args: {"Cmd":["sh"],"Entrypoint":[],"Env":null,"Image":"docker.io/library/busybox:latest","MaskedPaths":null,"OnBuild":null,"OpenStdin":true,"ReadonlyPaths":null,"Shell":null,"Tty":true,"HostConfig":{"Binds":null,"CapAdd":null,"CapDrop":null,"ConsoleSize":null,"Dns":null,"DnsOptions":null,"DnsSearch":null,"ExtraHosts":null,"GroupAdd":null,"Links":null,"LogConfig":{"Type":"json-file"},"NetworkMode":"bridge","OomScoreAdj":-500,"Privileged":false,"RestartPolicy":{"Name":"no"},"SecurityOpt":null,"ShmSize":0,"VolumesFrom":null,"BlkioDeviceReadBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceWriteIOps":null,"BlkioWeight":0,"BlkioWeightDevice":null,"CgroupParent":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","DeviceCgroupRules":null,"Devices":[],"IOMaximumBandwidth":0,"IOMaximumIOps":0,"IntelRdtL3Cbm":"","KernelMemory":0,"Memory":0,"MemoryExtra":0,"MemoryForceEmptyCtl":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":0,"MemoryWmarkRatio":0,"NanoCpus":0,"OomKillDisable":false,"PidsLimit":0,"ScheLatSwitch":0,"Ulimits":null},"NetworkingConfig":{}}"
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33.573848302Z" level=info msg="invoke container pre-create hook in plugin"
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33.573908567Z" level=warning msg="container 0ddad604a39b14f16f49740a2f96af2e818f99b4872d72f42e9d661a034cd7b2 meta.json file not exist"
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33.573925921Z" level=debug msg="Get a grpc client" elapsed=2.05µs
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33Z" level=debug msg="prepare snapshot" key=0ddad604a39b14f16f49740a2f96af2e818f99b4872d72f42e9d661a034cd7b2 module="containerd/snapshot" parent="sha256:d1156b98822dccbb924b4e5fe16465a7ecac8bfc81d726177bed403a8e70c972"
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33Z" level=debug msg="event published" module="containerd/snapshot" ns=default topic="/snapshot/prepare" type=containerd.events.SnapshotPrepare
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33.732911231Z" level=debug msg="Get a grpc client" elapsed=1.806µs
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33.736040301Z" level=debug msg="Get a grpc client" elapsed=1.558µs
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33Z" level=debug msg="stat snapshot" key=0ddad604a39b14f16f49740a2f96af2e818f99b4872d72f42e9d661a034cd7b2 module="containerd/snapshot"
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33.736381841Z" level=debug msg="Get a grpc client" elapsed=769ns
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33Z" level=debug msg="get snapshot mounts" key=0ddad604a39b14f16f49740a2f96af2e818f99b4872d72f42e9d661a034cd7b2 module="containerd/snapshot"
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33.736827048Z" level=info msg="start to rollback allocated resources of container 0ddad6"
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33.736838878Z" level=info msg="start to cleanup snapshot, id is 0ddad604a39b14f16f49740a2f96af2e818f99b4872d72f42e9d661a034cd7b2"
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33.736848546Z" level=debug msg="Get a grpc client" elapsed=940ns
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33Z" level=debug msg="remove snapshot" key=0ddad604a39b14f16f49740a2f96af2e818f99b4872d72f42e9d661a034cd7b2 module="containerd/snapshot"
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33Z" level=debug msg="event published" module="containerd/snapshot" ns=default topic="/snapshot/remove" type=containerd.events.SnapshotRemove
May 23 14:52:33 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:33.774427052Z" level=error msg="Handler for POST /v1.24/containers/create, client @ returns error: failed to init container storage, id: (0ddad604a39b14f16f49740a2f96af2e818f99b4872d72f42e9d661a034cd7b2): failed to mount rootfs(/var/lib/pouch/containers/0ddad604a39b14f16f49740a2f96af2e818f99b4872d72f42e9d661a034cd7b2/rootfs): invalid argument"
May 23 14:52:36 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:36Z" level=debug msg="schedule snapshotter cleanup" module="containerd/io.containerd.gc.v1.scheduler" snapshotter=overlayfs
May 23 14:52:36 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:36Z" level=debug msg="removed snapshot" key="default/19/0ddad604a39b14f16f49740a2f96af2e818f99b4872d72f42e9d661a034cd7b2" snapshotter=overlayfs
May 23 14:52:36 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:36Z" level=debug msg="snapshot garbage collected" d=36.996633ms snapshotter=overlayfs
May 23 14:52:36 kind-control-plane pouchd[30081]: time="2019-05-23T14:52:36Z" level=debug msg="garbage collected" d=31.328403ms module="containerd/io.containerd.gc.v1.scheduler"

From the logs above, I think the problem is the meta.json not exist. And there is nothing in /var/lib/pouch/containers/xxx/rootfs. Maybe it is not a problem from pouch, how to fix it?

Ⅱ. Describe what happened

Failed to run container via pouch run xxx

Ⅲ. Describe what you expected to happen

Start this container and enter its shell

Ⅳ. How to reproduce it (as minimally and precisely as possible)

Ⅴ. Anything else we need to know?

Ⅵ. Environment:

• pouch version (use pouch version):

APIVersion:      1.24
Arch:            amd64
BuildTime:       2019-02-14T02:21:34+00:00
GitCommit:       1.2.0
GoVersion:       go1.10.4
KernelVersion:   4.15.0-29deepin-generic
Os:              linux
Version:         1.2.0

• OS (e.g. from /etc/os-release):

4.15.0-29deepin-generic

• Kernel (e.g. uname -a):

Linux kind-control-plane 4.15.0-29deepin-generic #31 SMP Fri Jul 27 07:12:08 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:
  Via apt-get install pouch. Just follow the installation guide.
• Others:

containerd version

root@kind-control-plane:/lib/systemd/system# containerd -v
containerd github.com/containerd/containerd v1.0.3 773c489c9c1b21a6d78b5c538cd395416ec50f88

runc version

root@kind-control-plane:/lib/systemd/system# runc -v
runc version 1.0.0-rc4
spec: 1.0.0

[CodeJuan]

Maybe add VOLUME /var/lib/pouch/ to dockerfile would works.
Hopefully that can help.