docker checkpoint failed

[kimh]

I'm trying to make docker checkpoint works, but I got the following error.

Here is the docker client log.

vagrant@vagrant-ubuntu-trusty:~/docker$ export CID=$(docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(e
xpr $i + 1); sleep 3; done')

vagrant@vagrant-ubuntu-trusty:~/docker$ docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS               NAMES
806b0fa36744        busybox:latest      "/bin/sh -c 'i=0; wh   3 seconds ago       Up 1 seconds                            determined_visvesvaraya

vagrant@vagrant-ubuntu-trusty:~/docker$ docker checkpoint $CID
Error response from daemon: Cannot checkpoint container 806b0fa36744339b60e41727c1c7f4aa65e30cd3ab86a63206cae9c4854b6fbb: criu failed: type NOTIFY errno 0

Here is the docker daemon log.

INFO[0840] POST /v1.20/containers/806b0fa36744339b60e41727c1c7f4aa65e30cd3ab86a63206cae9c4854b6fbb/checkpoint
ERRO[0840] Handler for POST /containers/{name:.*}/checkpoint returned error: Cannot checkpoint container 806b0fa36744339b60e41727c1c7f4aa65e30cd3ab86a63206cae9c4854b6fbb: criu failed: type NOTIFY errno 0
ERRO[0840] HTTP Error                                    err=Cannot checkpoint container 806b0fa36744339b60e41727c1c7f4aa65e30cd3ab86a63206cae9c4854b6fbb: criu failed: type NOTIFY errno 0 statusCode=500

Here is my environment.

Ubuntu: 14.04 (vagrant)

criu

vagrant@vagrant-ubuntu-trusty:~/docker$ criu --version
Version: 1.6
GitID: v1.6-95-gde70936

docker boucher/docker@dd06ea0

vagrant@vagrant-ubuntu-trusty:~/docker$ docker -v
Docker version 1.8.0-dev, build dd06ea0

Please let me know if you need more information.

[xemul]

Can you find dump logs?
Also adding @boucher to discussion :)

[boucher]

Hey @kimh, there's currently a regression in Docker thats causing the problem. Reverting this commit should work as a temporary fix: f18fb5b3efd59d54c00d4e1b1d4b88c4b21e96be

[kimh]

@boucher ok, reverted and now compiling....It's totally different discussion, but are there any good ways to compile docker fast? Using make create docker image everytime from scratch :(

[boucher]

When you build with make docker should be using its cache for everything
but the final step. My builds take less than a minute usually.

On Wednesday, July 15, 2015, Kim, Hirokuni notifications@github.com wrote:

@boucher https://github.com/boucher ok, reverted and now
compiling....It's totally different discussion, but are there any good ways
to compile docker fast? Using make create docker image everytime from
scratch :(

—
Reply to this email directly or view it on GitHub
https://github.com/xemul/criu/issues/6#issuecomment-121666031.

Sent from Gmail Mobile

[kimh]

@boucher Looks now docker checkpoint is not available.

vagrant@vagrant-ubuntu-trusty:~/docker$ docker -v
Docker version 1.8.0-dev, build f18fb5b
vagrant@vagrant-ubuntu-trusty:~/docker$ docker checkpoint
docker: 'checkpoint' is not a docker command.
See 'docker --help'.

Did I have to use different version of docker binary ? (daemon with reverted version and client with latest)

[boucher]

Are you sure you applied the revert to the right branch?

On Wed, Jul 15, 2015 at 9:30 AM, Kim, Hirokuni notifications@github.com
wrote:

@boucher https://github.com/boucher Looks now docker checkpoint is not
available.

vagrant@vagrant-ubuntu-trusty:/docker$ docker -v
Docker version 1.8.0-dev, build f18fb5b
vagrant@vagrant-ubuntu-trusty:/docker$ docker checkpoint
docker: 'checkpoint' is not a docker command.
See 'docker --help'.

Did I have to use different version of docker binary ? (daemon with
reverted version and client with latest)

—
Reply to this email directly or view it on GitHub
https://github.com/xemul/criu/issues/6#issuecomment-121669300.

[kimh]

Doesn't this look so?

vagrant@vagrant-ubuntu-trusty:~/docker$ docker -v
Docker version 1.8.0-dev, build f18fb5b

[kimh]

Oh, I think I misunderstood what you said. So reiterate just in case, I have to revert f18fb5b3efd59d54c00d4e1b1d4b88c4b21e96be commit?

[boucher]

Yes, from the cr-combined branch.

[kimh]

@boucher Now checkpoint works. However, restore doesn't.

docker restore db490e9a820014dca0c6e960b28381db7bc0dcf5778fcb6e68bc855fb96465c0

ERRO[0045] Error restoring container: criu failed: type NOTIFY errno 0, exitCode={-1 %!d(bool=false)}
ERRO[0045] Handler for POST /containers/{name:.*}/restore returned error: Cannot restore container db490e9a820014dca0c6e960b28381db7bc0dcf5778fcb6e68bc855fb96465c0: criu failed: type NOTIFY errno 0
ERRO[0045] HTTP Error                                    err=Cannot restore container db490e9a820014dca0c6e960b28381db7bc0dcf5778fcb6e68bc855fb96465c0: criu failed: type NOTIFY errno 0 statusCode=500
INFO[0050] GET /v1.20/containers/json

Any idea?

[boucher]

Can you pass --work-dir=/tmp/something to the restore and then paste the
criu restore log somewhere

On Wed, Jul 15, 2015 at 9:47 AM, Kim, Hirokuni notifications@github.com
wrote:

@boucher https://github.com/boucher Now checkpoint works. However,
restore doesn't.

docker restore db490e9a820014dca0c6e960b28381db7bc0dcf5778fcb6e68bc855fb96465c0

ERRO[0045] Error restoring container: criu failed: type NOTIFY errno 0, exitCode={-1 %!d(bool=false)}
ERRO[0045] Handler for POST /containers/{name:.*}/restore returned error: Cannot restore container db490e9a820014dca0c6e960b28381db7bc0dcf5778fcb6e68bc855fb96465c0: criu failed: type NOTIFY errno 0
ERRO[0045] HTTP Error err=Cannot restore container db490e9a820014dca0c6e960b28381db7bc0dcf5778fcb6e68bc855fb96465c0: criu failed: type NOTIFY errno 0 statusCode=500
INFO[0050] GET /v1.20/containers/json

Any idea?

—
Reply to this email directly or view it on GitHub
https://github.com/xemul/criu/issues/6#issuecomment-121676184.

[kimh]

Is restore.log supposed to be created under /tmp/something ?

docker restore $CID  --work-dir=/tmp/something
Error response from daemon: Cannot restore container e50472b26a0ce9e1369d0f76e5b25d11e68979709691b7dc66cf305cd18670df: a container has already joined the endpoint
Error response from daemon: no such id: --work-dir=/tmp/something
Error: failed to restore one or more containers

Looks like nothing is created.

vagrant@vagrant-ubuntu-trusty:/var$ ls /tmp/something
ls: cannot access /tmp/something: No such file or directory

Even if I first created something directory, nothing is created.

[boucher]

You must pass --work-dir before the $CID

[kimh]

Here is restore.log

https://gist.github.com/kimh/1077ad2a8e7aa58f3eed

[boucher]

Did you build criu from source? I believe there's a bug in the master branch that @xemul is working on (and may already have a patch for?)

[kimh]

Yes, I built from source. My HEAD is xemul@de70936

I live in JST, so I'll go to bed now. @xemul can you let me know if there is anything that I can try?

Thank you for your quick help @boucher

[boucher]

opencontainers/runc#130

I believe this is tracking the fix in runc, but it will then need to be merged into docker master as well.

[YaoZengzeng]

Hi,@boucher what do you mean "Reverting this commit should work as a temporary fix: f18fb5b3efd59d54c00d4e1b1d4b88c4b21e96be".
I'm under cr-combined branch and "git rest --hard f18fb5b3efd59d54c00d4e1b1d4b88c4b21e96be" ,then compile ,but the binary seems not support checkpoint and restore.
root@iZu1pbgsilbZ:~# docker version
Client:
Version: 1.8.0-dev
API version: 1.20
Go version: go1.4.2
Git commit: f18fb5b
Built: Sun Jul 19 03:16:51 UTC 2015
OS/Arch: linux/amd64

Server:
Version: 1.8.0-dev
API version: 1.20
Go version: go1.4.2
Git commit: f18fb5b
Built: Sun Jul 19 03:16:51 UTC 2015
OS/Arch: linux/amd64
My problem is same with @kimh.

[boucher]

Sorry, as of yesterday it's now behind the experimental flag. You need to
build with DOCKER_EXPERIMENTAL=1

Also the fix has been incorporated so reverting that commit is no longer
necessary. However, there is a new blocker that has not yet been figured
out.

On Saturday, July 18, 2015, YaoZengzeng notifications@github.com wrote:

Hi,@boucher https://github.com/boucher what do you mean "Reverting this
commit should work as a temporary fix:
f18fb5b3efd59d54c00d4e1b1d4b88c4b21e96be".
I'm under cr-combined branch and "git rest --hard
f18fb5b3efd59d54c00d4e1b1d4b88c4b21e96be" ,then compile ,but the binary
seems not support checkpoint and restore.
root@iZu1pbgsilbZ:~# docker version
Client:
Version: 1.8.0-dev
API version: 1.20
Go version: go1.4.2
Git commit: f18fb5b
Built: Sun Jul 19 03:16:51 UTC 2015
OS/Arch: linux/amd64

Server:
Version: 1.8.0-dev
API version: 1.20
Go version: go1.4.2
Git commit: f18fb5b
Built: Sun Jul 19 03:16:51 UTC 2015
OS/Arch: linux/amd64
My problem is same with @kimh https://github.com/kimh.

—
Reply to this email directly or view it on GitHub
https://github.com/xemul/criu/issues/6#issuecomment-122625309.

Sent from Gmail Mobile

[YaoZengzeng]

@boucher I compile like this
"root@iZu1pbgsilbZ:/home/monster/docker# make DOCKER_EXPERIMENTAL=1 binary",
and the result support checkpoint and restore command .But checkpoint still doesn't work

root@iZu1pbgsilbZ:/home/monster/docker# docker checkpoint $CID
Error response from daemon: Cannot checkpoint container be9da3d14c883491fee1d73a727b88d9d52e1db82996f9110080a544ade94b24: criu failed: type NOTIFY errno 0
Error: failed to checkpoint one or more containers

[boucher]

Yes, things are currently broken. You can follow this thread on the mailing
list:
http://lists.openvz.org/pipermail/criu/2015-July/021277.html

On Sat, Jul 18, 2015 at 10:03 PM, YaoZengzeng notifications@github.com
wrote:

@boucher https://github.com/boucher I compile like this
"root@iZu1pbgsilbZ:/home/monster/docker# make DOCKER_EXPERIMENTAL=1
binary",
and the result support checkpoint and restore command .But checkpoint
still doesn't work

root@iZu1pbgsilbZ:/home/monster/docker# docker checkpoint $CID
Error response from daemon: Cannot checkpoint container
be9da3d14c883491fee1d73a727b88d9d52e1db82996f9110080a544ade94b24: criu
failed: type NOTIFY errno 0
Error: failed to checkpoint one or more containers

—
Reply to this email directly or view it on GitHub
https://github.com/xemul/criu/issues/6#issuecomment-122629443.

[YaoZengzeng]

Hi @boucher I find that in docker 1.8.0 ,it will not create dockerinit-1.8.0-dev under /var/lib/docker/init/ or anywhere else .I don't if it is a bug or something else ....Becasuse of this ,I even can't do external C/R test in docker 1.8.0,the docker_cr.sh will report the missing of dockerinit-1.8.0-dev.

[xemul]

@kimh the error you see in logs should be fixed in recent head. The fixing commit is 7b20f42

[kimh]

@xemul thank you for the ping. I'll try later and let you know how it goes.

[kimh]

@xemul I tried 7b20f42 and restore once has worked. Then, I updated docker to use boucher/docker@24e477b and now checkpoint doesn't work... (DOCKER_EXPERIMENTAL is set).

The head for CRIU is xemul@ace699d

Here is the log for failed checkpoint which is useless.

vagrant@vagrant-ubuntu-trusty:~/criu$ docker checkpoint $CID
Error response from daemon: Cannot checkpoint container 94dd8db31db80dc3cc61c44cab05b6018595c0b48144350b82e191897177bae4: criu failed: type NOTIFY errno 0
Error: failed to checkpoint one or more containers

How can I get criu dump log?

[xemul]

OK :) Now we need the @boucher 's help with Docker part.

[boucher]

The dump log should be at /var/run/docker/execdriver/native/<container_id>/criu.work/dump.log

[kimh]

@boucher here is the dump.log

[boucher]

This is the same issue as before, in Docker. The fix has not yet made its way from libcontainer to docker. You can update libcontainer yourself for now, or you can try reverting the same commit from the top of this thread.

[Vallari-Mehta]

I am trying to build docker from following source: https://github.com/boucher/docker.git and using the branch: /cr-combined. But I do not see docker checkpoint option.

I am using the command make DOCKER_EXPERIMENTAL=1 build to build the docker
And then make DOCKER_EXPERIMENTAL=1 binary to compile.

After compiling when I try running the compiled binary: ./docker --help.
I do not see any checkpoint option.
I also tried ./docker checkpoint. But it says "docker: 'checkpoint' is not a docker command."

Please let me know if you need any information.

[xemul]

Please, open another issue if you have problems. This one is closed already.