Singularity restore "Can't mount at /tmp/.criu.mntns.ocwykx" error

[ShunyuYao515]

I come from #841 and I'm able to dump the Singularity container, but still have problems restoring it.

I'm running Singularity 3.5 with the command:
singularity run --contain /home/CRIU/ubuntu.img

My dump command is:

   sudo criu dump -v4 -o ~/dump.log --tree $1 --images-dir /home/CRIU/exit_dir \
    --external mnt[]:m \
    --external mnt[/etc]:etc \
    --external mnt[/etc/group]:etc_group \
    --external mnt[/etc/passwd]:etc_passwd \
    --external mnt[/etc/resolv.conf]:etc_resolv.conf \
    --external mnt[/var]:var \
    --external mnt[/var/tmp]:var_tmp \
    --external mnt[/tmp]:tmp \
    --external mnt[/proc]:proc \
    --external mnt[/proc/sys/fs/binfmt_misc]:proc_sys_fs_binfmt__misc \
    --external mnt[/.singularity.d/actions]:.singularity.d_actions \
    --external mnt[/dev]:dev \
    --external mnt[/dev/urandom]:dev_urandom \
    --external mnt[/dev/random]:dev_random \
    --external mnt[/dev/zero]:dev_zero \
    --external mnt[/dev/null]:dev_null \
    --external mnt[/dev/tty]:dev_tty \
    --external mnt[/dev/console]:dev_console \
    --leave-stopped --shell-job

And my restore command is:

    sudo strace -o ~/strace.log -s 256 -f /home/CRIU/criu/scripts/criu-ns restore -v4 -o ~/restore.log --images-dir /home/CRIU/exit_dir \
    --external mnt[]:m \
    --external mnt[etc]:/etc \
    --external mnt[etc_group]:/etc/group \
    --external mnt[etc_passwd]:/etc/passwd \
    --external mnt[etc_resolv.conf]:/etc/resolv.conf \
    --external mnt[var]:/var \
    --external mnt[var_tmp]:/var/tmp \
    --external mnt[tmp]:/tmp \
    --external mnt[proc]:/proc \
    --external mnt[proc_sys_fs_binfmt__misc]:/proc/sys/fs/binfmt_misc \
    --external mnt[.singularity.d_actions]:/.singularity.d/actions \
    --external mnt[dev]:/dev \
    --external mnt[dev_urandom]:/dev/urandom \
    --external mnt[dev_zero]:/dev/zero \
    --external mnt[dev_random]:/dev/random \
    --external mnt[dev_null]:/dev/null \
    --external mnt[dev_tty]:/dev/tty \
    --external mnt[dev_console]:/dev/console \
    --shell-job --root /usr/local/var/singularity/mnt/session/final

The current issues are:

1. The error now proceeds to:

(00.018105) 5826: mnt: Start with 0:/tmp/.criu.mntns.ocwykx
(00.018173) 5826: mnt: Start with 0:/tmp/.criu.mntns.ocwykx
(00.018634) 5826: mnt: Start with 0:/tmp/.criu.mntns.ocwykx
(00.018689) 5826: mnt: Mounting overlay @/tmp/.criu.mntns.ocwykx/8-0000000000/ (0)
(00.018797) 5826: mnt: 1207:/tmp/.criu.mntns.ocwykx/8-0000000000/ private 1 shared 0 slave 0
(00.018987) 5826: mnt: Mounting unsupported @/tmp/.criu.mntns.ocwykx/8-0000000000/.singularity.d/actions (0)
(00.019038) 5826: mnt: Bind /.singularity.d/actions to /tmp/.criu.mntns.ocwykx/8-0000000000/.singularity.d/actions
(00.019160) 5826: Error (criu/mount.c:2298): mnt: Can't mount at /tmp/.criu.mntns.ocwykx/8-0000000000/.singularity.d/actions: No such file or directory
(00.019256) 5826: Error (criu/mount.c:2525): mnt: Unable to statfs /tmp/.criu.mntns.ocwykx/8-0000000000/.singularity.d/actions: No such file or directory
(00.019307) 5826: mnt: Start with 0:/tmp/.criu.mntns.ocwykx
(00.020035) Error (criu/cr-restore.c:2334): Restoring FAILED.
(00.020301) Error (criu/cr-restore.c:1458): 5826 killed by signal 9: Killed

2. I ran the restore with strace, and I compared the result with Trying to checkpoint/restore from a Singularity container #600, and I found that there are more errors about illegal lseek and non-existing python files:
   _strace.log

3. This might most likely be irrelevant but the console that's running the container will be frozen despite that the "--leave-stopped" flag is set.

The dump/restore log:
_dump.log
_restore.log

Thank you so much for your help!!!

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[adrianreber]

Closing this as there has not been any real activity, as far as I know, to integrate CRIU into singularity.