Restoring of docker container failed after restarting docker daemon

[kimh]

I experimented if I could restore a container after rebooting my docker daemon but I got Error response from daemon: Container f6b3120b3fce is not checkpointed error.

$ export CID=$(docker run -d busybox tail -f /dev/null)
$ docker checkpoint $CID
eb0ce45f8c7820978e05323cc5646affec73a3b962ed67db363e3ca744345b7d
$ sudo service docker restart
docker stop/waiting
docker start/running, process 5206
$ docker restore $CID
Error response from daemon: Container eb0ce45f8c7820978e05323cc5646affec73a3b962ed67db363e3ca744345b7d is not checkpointed
Error: failed to restore one or more containers

Is this supported by CRIU?

[xemul]

By CRIU -- yes :) It looks like some in-Docker problem with states management. @boucher, @SaiedKazemi , @avagin .

[kimh]

Ok. Here is the fork of docker that I'm using.

$ docker -v
Docker version 1.8.0-dev, build fe8c65c, experimental

[boucher]

I think this may be fixed in my branch if you try the latest version.

[kimh]

@boucher Tried your branch but now restore failed (without restarting docker daemon).

Here is the restore log.

(00.000045) Added /sys/fs/cgroup/cpuset:/sys/fs/cgroup/cpuset ext mount mapping
(00.000083) Added /sys/fs/cgroup/cpu:/sys/fs/cgroup/cpu ext mount mapping
(00.000089) Added /sys/fs/cgroup/cpuacct:/sys/fs/cgroup/cpuacct ext mount mapping
(00.000095) Added /sys/fs/cgroup/memory:/sys/fs/cgroup/memory ext mount mapping
(00.000101) Added /sys/fs/cgroup/devices:/sys/fs/cgroup/devices ext mount mapping
(00.000107) Added /sys/fs/cgroup/freezer:/sys/fs/cgroup/freezer ext mount mapping
(00.000112) Added /sys/fs/cgroup/blkio:/sys/fs/cgroup/blkio ext mount mapping
(00.000118) Added /sys/fs/cgroup/perf_event:/sys/fs/cgroup/perf_event ext mount mapping
(00.000124) Added /sys/fs/cgroup/hugetlb:/sys/fs/cgroup/hugetlb ext mount mapping
(00.000129) Added /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd ext mount mapping
(00.000135) Added /etc/resolv.conf:/var/lib/docker/containers/f8df6fa0c4b4ef6232042231b2d0131b2818d5b96cbb2b15f9e7a21873106a6a/resolv.conf ext mount mapping
(00.000141) Added /etc/hostname:/var/lib/docker/containers/f8df6fa0c4b4ef6232042231b2d0131b2818d5b96cbb2b15f9e7a21873106a6a/hostname ext mount mapping
(00.000146) Added /etc/hosts:/var/lib/docker/containers/f8df6fa0c4b4ef6232042231b2d0131b2818d5b96cbb2b15f9e7a21873106a6a/hosts ext mount mapping
(00.000319) TCP recv queue memory limit is 3145728
(00.000394) cpu: fpu:1 fxsr:1 xsave:0
(00.000546) vdso: Parsing at 7ffe464d5000 7ffe464d7000
(00.000553) vdso: PT_LOAD p_vaddr: ffffffffff700000
(00.000563) vdso: DT_HASH: 0xffffffffff700120
(00.000569) vdso: DT_STRTAB: 0xffffffffff700268
(00.000575) vdso: DT_SYMTAB: 0xffffffffff700160
(00.000580) vdso: DT_STRSZ: 94
(00.000586) vdso: DT_SYMENT: 24
(00.000591) vdso: nbucket 3 nchain 11 bucket 0x7ffe464d5128 chain 0x7ffe464d5134
(00.000613) vdso: rt [vdso] 7ffe464d5000-7ffe464d7000 [vvar] ffffffffffffffff-ffffffffffffffff
(00.000680) Reading image tree
(00.000742) Add mnt ns 11 pid 1
(00.000789) Will restore in 6c020000 namespaces
(00.000798) NS mask to use 6c020000
(00.000818) Collecting 37/21 (flags 0)
(00.000829) No inetsk.img image
(00.000835)  `- ... done
(00.000848) Error (tty.c:1595): tty: Standard stream is not a terminal, aborting

[boucher]

Can you tell me the exact commands you ran?

[avagin]

I think it's due to the shell-job option. We should not set this option to dump a container.

[boucher]

Yes, I think so too, just wanted to make sure. I misunderstood how the shell job flag works. I'll revert that particular change now.

[boucher]

@kimh If you pull again I think it will work.

[kimh]

@boucher Pull which project? CRIU or docker?

I pulled master branch of CRIU and https://github.com/boucher/docker/tree/cr-combined but I don't see your revert. (FYI I recompiled criu with latest master but the issue was not fixed).

[kimh]

Oh, btw, I run these commands:

$ export CID=$(docker run -d busybox tail -f /dev/null)
$ docker checkpoint $CID
$ docker restore $CID

[klesgidisold]

I think that what both guys said is, that you should run with the --allow-shell option

docker checkpoint --allow-shell $CID
docker restore --allow-shell $CID

[boucher]

There was no revert commit, I force pushed the change. I reverted the behavioral change though, there's no longer a --no-shell option, and its back to being --allow-shell, which you don't want here. Are you sure you're running commit 792fc89ab? Can you check the log to see that it isn't a new restore error?

[kimh]

There was no revert commit, I force pushed the change.

Ah, I didn't think about this, so haven't pulled docker.

Yes, using current 792fc89ab worked for me. Now I can restore a container even after restarting docker daemon.

--allow-shell, which you don't want here

Is this because I run the container without shell?

[boucher]

Yeah, it's a flag saying that the container is a shell.

[kimh]

I see. Thanks, @boucher I'll close the issue.