Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mount-v2: workaround for multiple external bindmounts with no common root #1898

Conversation

Snorch
Copy link
Member

@Snorch Snorch commented May 18, 2022

It's a problem when while restoring sharing group we need to copy
sharing between to mounts with non-intersecting roots, because kernel
does not allow it.

We have a case opencontainers/runc#3442, where
runc adds different devtmpfs file-bindmounts to container and there is
no fsroot mount in container for this devtmpfs, thus mount-v2 faces the
above problem.

Luckily for the case of external mounts which are in one sharing group
and which have non-intersecting roots, these mounts likely only have
external master with no sharing, so we can just copy sharing from
external source and make it slave as a workaround.

#1886

Signed-off-by: Pavel Tikhomirov ptikhomirov@virtuozzo.com

@Snorch Snorch mentioned this pull request May 18, 2022
@adrianreber
Copy link
Member

Nice! I will try it out tomorrow.

@codecov-commenter
Copy link

codecov-commenter commented May 18, 2022

Codecov Report

Merging #1898 (8764992) into criu-dev (2b3763f) will increase coverage by 0.47%.
The diff coverage is 0.00%.

❗ Current head 8764992 differs from pull request most recent head 24bf471. Consider uploading reports for the commit 24bf471 to get more accurate results

@@             Coverage Diff              @@
##           criu-dev    #1898      +/-   ##
============================================
+ Coverage     69.20%   69.68%   +0.47%     
============================================
  Files           126      128       +2     
  Lines         33355    32201    -1154     
============================================
- Hits          23083    22438     -645     
+ Misses        10272     9763     -509     
Impacted Files Coverage Δ
criu/mount-v2.c 0.71% <0.00%> (+0.02%) ⬆️
criu/uffd.c 79.36% <0.00%> (-0.48%) ⬇️
criu/include/vma.h 100.00% <0.00%> (ø)
criu/include/parasite.h 100.00% <0.00%> (ø)
criu/crtools.c
criu/kerndat.c
compel/src/lib/infect.c
include/common/bitops.h 100.00% <0.00%> (ø)
criu/include/linux/mount.h 100.00% <0.00%> (ø)
criu/arch/x86/kerndat.c 65.71% <0.00%> (ø)
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2b3763f...24bf471. Read the comment docs.

@rst0git
Copy link
Member

rst0git commented May 19, 2022

@Snorch Would it be possible to add a test for this case?

@adrianreber
Copy link
Member

I just tested it and it does not seem to help:

    checkpoint_test.go:163: === /tmp/TestUsernsCheckpoint1975440553/003/criu/restore.log ===
    checkpoint_test.go:163: (00.000000) Parsing config file /etc/criu/runc.conf
    checkpoint_test.go:163: (00.000069) Version: 3.16.1 (gitid v3.16.1-319-g799ec1482)
    checkpoint_test.go:163: (00.000077) Running on fedora02 Linux 5.16.18-200.fc35.x86_64 #1 SMP PREEMPT Mon Mar 28 14:10:07 UTC 2022 x86_64
    checkpoint_test.go:163: (00.000081) Would overwrite RPC settings with values from /etc/criu/runc.conf
    checkpoint_test.go:163: (00.000108) Loaded kdat cache from /run/criu.kdat
    checkpoint_test.go:163: (00.000141) Hugetlb size 2 Mb is supported but cannot get dev's number
    checkpoint_test.go:163: (00.000156) Hugetlb size 1024 Mb is supported but cannot get dev's number
    checkpoint_test.go:163: (00.000221) Parsing config file /etc/criu/runc.conf
    checkpoint_test.go:163: (00.000266) rlimit: RLIMIT_NOFILE unlimited for self
    checkpoint_test.go:163: (00.000366) cpu: x86_family 25 x86_vendor_id AuthenticAMD x86_model_id AMD Ryzen 5 5600G with Radeon Graphics         
    checkpoint_test.go:163: (00.000376) cpu: fpu: xfeatures_mask 0x205 xsave_size 2440 xsave_size_max 2440 xsaves_size 840
    checkpoint_test.go:163: (00.000396) cpu: fpu: x87 floating point registers     xstate_offsets      0 / 0      xstate_sizes    160 / 160   
    checkpoint_test.go:163: (00.000401) cpu: fpu: AVX registers                    xstate_offsets    576 / 576    xstate_sizes    256 / 256   
    checkpoint_test.go:163: (00.000405) cpu: fpu: Protection Keys User registers   xstate_offsets   2432 / 832    xstate_sizes      8 / 8     
    checkpoint_test.go:163: (00.000409) cpu: fpu:1 fxsr:1 xsave:1 xsaveopt:1 xsavec:1 xgetbv1:1 xsaves:1
    checkpoint_test.go:163: (00.000492) kernel pid_max=4194304
    checkpoint_test.go:163: (00.000500) Reading image tree
    checkpoint_test.go:163: (00.000554) Add mnt ns 13 pid 1
    checkpoint_test.go:163: (00.000559) Add net ns 10 pid 1
    checkpoint_test.go:163: (00.000563) Add pid ns 9 pid 1
    checkpoint_test.go:163: (00.000568) pstree pid_max=1
    checkpoint_test.go:163: (00.000579) Will restore in 7c020000 namespaces
    checkpoint_test.go:163: (00.000583) NS mask to use 7c020000
    checkpoint_test.go:163: (00.000654) Collecting 51/56 (flags 3)
    checkpoint_test.go:163: (00.000664) No memfd.img image
    checkpoint_test.go:163: (00.000668)  `- ... done
    checkpoint_test.go:163: (00.000674) Collecting 40/54 (flags 2)
    checkpoint_test.go:163: (00.000698) Collected [bin/cat] ID 0x1
    checkpoint_test.go:163: (00.000706) Collected [lib/libc.so.6] ID 0x2
    checkpoint_test.go:163: (00.000712) Collected [lib/libresolv.so.2] ID 0x3
    checkpoint_test.go:163: (00.000716) Collected [lib/libm.so.6] ID 0x4
    checkpoint_test.go:163: (00.000721) Collected [lib/ld-linux-x86-64.so.2] ID 0x5
    checkpoint_test.go:163: (00.000727) Collected pipe entry ID 0x6 PIPE ID 0x71c1af1
    checkpoint_test.go:163: (00.000740) Found id pipe:[119282417] (fd 1) in inherit fd list
    checkpoint_test.go:163: (00.000746) Collected pipe entry ID 0x7 PIPE ID 0x71c1af5
    checkpoint_test.go:163: (00.000753) Found id pipe:[119282421] (fd 10) in inherit fd list
    checkpoint_test.go:163: (00.000758) Collected [dev/null] ID 0x8
    checkpoint_test.go:163: (00.000763) Collected [.] ID 0x9
    checkpoint_test.go:163: (00.000767) Collected [.] ID 0xa
    checkpoint_test.go:163: (00.000773)  `- ... done
    checkpoint_test.go:163: (00.000776) Collecting 46/68 (flags 0)
    checkpoint_test.go:163: (00.000781) No remap-fpath.img image
    checkpoint_test.go:163: (00.000785)  `- ... done
    checkpoint_test.go:163: (00.000791) No apparmor.img image
    checkpoint_test.go:163: (00.000888) cg: rewriting test/integration to /test/integration
    checkpoint_test.go:163: (00.000899) cg: rewriting test/integration to /test/integration
    checkpoint_test.go:163: (00.000905) cg: rewriting test/integration to /test/integration
    checkpoint_test.go:163: (00.000909) cg: rewriting test/integration to /test/integration
    checkpoint_test.go:163: (00.000914) cg: rewriting test/integration to /test/integration
    checkpoint_test.go:163: (00.000918) cg: rewriting test/integration to /test/integration
    checkpoint_test.go:163: (00.000923) cg: rewriting test/integration to /test/integration
    checkpoint_test.go:163: (00.000928) cg: rewriting  to /
    checkpoint_test.go:163: (00.000934) cg: rewriting test/integration to /test/integration
    checkpoint_test.go:163: (00.000940) cg: rewriting test/integration to /test/integration
    checkpoint_test.go:163: (00.000946) cg: rewriting test/integration to /test/integration
    checkpoint_test.go:163: (00.000951) cg: rewriting test/integration to /test/integration
    checkpoint_test.go:163: (00.000955) cg: rewriting test/integration to /test/integration
    checkpoint_test.go:163: (00.000959) cg: Preparing cgroups yard (cgroups restore mode 0x4)
    checkpoint_test.go:163: (00.001258) cg: Opening .criu.cgyard.SeVJMD as cg yard
    checkpoint_test.go:163: (00.001278) cg: 	Making controller dir .criu.cgyard.SeVJMD/pids (pids)
    checkpoint_test.go:163: (00.001353) cg: Determined cgroup dir pids//test/integration already exist
    checkpoint_test.go:163: (00.001358) cg: Skip restoring properties on cgroup dir pids//test/integration
    checkpoint_test.go:163: (00.001376) cg: 	Making controller dir .criu.cgyard.SeVJMD/hugetlb (hugetlb)
    checkpoint_test.go:163: (00.001553) cg: Determined cgroup dir hugetlb//test/integration already exist
    checkpoint_test.go:163: (00.001563) cg: Skip restoring properties on cgroup dir hugetlb//test/integration
    checkpoint_test.go:163: (00.001581) cg: 	Making controller dir .criu.cgyard.SeVJMD/devices (devices)
    checkpoint_test.go:163: (00.001635) cg: Determined cgroup dir devices//test/integration already exist
    checkpoint_test.go:163: (00.001639) cg: Skip restoring properties on cgroup dir devices//test/integration
    checkpoint_test.go:163: (00.001653) cg: 	Making controller dir .criu.cgyard.SeVJMD/cpuset (cpuset)
    checkpoint_test.go:163: (00.001726) cg: Determined cgroup dir cpuset//test/integration already exist
    checkpoint_test.go:163: (00.001749) cg: Skip restoring properties on cgroup dir cpuset//test/integration
    checkpoint_test.go:163: (00.001765) cg: 	Making controller dir .criu.cgyard.SeVJMD/blkio (blkio)
    checkpoint_test.go:163: (00.001822) cg: Determined cgroup dir blkio//test/integration already exist
    checkpoint_test.go:163: (00.001827) cg: Skip restoring properties on cgroup dir blkio//test/integration
    checkpoint_test.go:163: (00.001841) cg: 	Making controller dir .criu.cgyard.SeVJMD/perf_event (perf_event)
    checkpoint_test.go:163: (00.001873) cg: Determined cgroup dir perf_event//test/integration already exist
    checkpoint_test.go:163: (00.001877) cg: Skip restoring properties on cgroup dir perf_event//test/integration
    checkpoint_test.go:163: (00.001890) cg: 	Making controller dir .criu.cgyard.SeVJMD/memory (memory)
    checkpoint_test.go:163: (00.001919) cg: Determined cgroup dir memory//test/integration already exist
    checkpoint_test.go:163: (00.001923) cg: Skip restoring properties on cgroup dir memory//test/integration
    checkpoint_test.go:163: (00.001936) cg: 	Making controller dir .criu.cgyard.SeVJMD/misc (misc)
    checkpoint_test.go:163: (00.001962) cg: Determined cgroup dir misc// already exist
    checkpoint_test.go:163: (00.001965) cg: Skip restoring properties on cgroup dir misc//
    checkpoint_test.go:163: (00.001978) cg: Determined cgroup dir misc///user.slice already exist
    checkpoint_test.go:163: (00.001981) cg: Skip restoring properties on cgroup dir misc///user.slice
    checkpoint_test.go:163: (00.001993) cg: Determined cgroup dir misc///user.slice/user-0.slice already exist
    checkpoint_test.go:163: (00.001996) cg: Skip restoring properties on cgroup dir misc///user.slice/user-0.slice
    checkpoint_test.go:163: (00.002008) cg: Determined cgroup dir misc///user.slice/user-0.slice/user@0.service already exist
    checkpoint_test.go:163: (00.002011) cg: Skip restoring properties on cgroup dir misc///user.slice/user-0.slice/user@0.service
    checkpoint_test.go:163: (00.002024) cg: Determined cgroup dir misc///user.slice/user-0.slice/user@0.service/app.slice already exist
    checkpoint_test.go:163: (00.002027) cg: Skip restoring properties on cgroup dir misc///user.slice/user-0.slice/user@0.service/app.slice
    checkpoint_test.go:163: (00.002040) cg: Determined cgroup dir misc///machine.slice already exist
    checkpoint_test.go:163: (00.002043) cg: Skip restoring properties on cgroup dir misc///machine.slice
    checkpoint_test.go:163: (00.002056) cg: 	Making controller dir .criu.cgyard.SeVJMD/cpu,cpuacct (cpu,cpuacct)
    checkpoint_test.go:163: (00.002086) cg: Determined cgroup dir cpu,cpuacct//test/integration already exist
    checkpoint_test.go:163: (00.002090) cg: Skip restoring properties on cgroup dir cpu,cpuacct//test/integration
    checkpoint_test.go:163: (00.002103) cg: 	Making controller dir .criu.cgyard.SeVJMD/freezer (freezer)
    checkpoint_test.go:163: (00.002131) cg: Determined cgroup dir freezer//test/integration already exist
    checkpoint_test.go:163: (00.002134) cg: Skip restoring properties on cgroup dir freezer//test/integration
    checkpoint_test.go:163: (00.002147) cg: 	Making controller dir .criu.cgyard.SeVJMD/net_cls,net_prio (net_cls,net_prio)
    checkpoint_test.go:163: (00.002179) cg: Determined cgroup dir net_cls,net_prio//test/integration already exist
    checkpoint_test.go:163: (00.002183) cg: Skip restoring properties on cgroup dir net_cls,net_prio//test/integration
    checkpoint_test.go:163: (00.002197) cg: 	Making controller dir .criu.cgyard.SeVJMD/unified ()
    checkpoint_test.go:163: (00.002222) cg: Determined cgroup dir unified//test/integration already exist
    checkpoint_test.go:163: (00.002226) cg: Skip restoring properties on cgroup dir unified//test/integration
    checkpoint_test.go:163: (00.002240) cg: 	Making controller dir .criu.cgyard.SeVJMD/systemd (none,name=systemd)
    checkpoint_test.go:163: (00.004718) cg: Determined cgroup dir systemd//test/integration already exist
    checkpoint_test.go:163: (00.004728) cg: Skip restoring properties on cgroup dir systemd//test/integration
    checkpoint_test.go:163: (00.004754) Running pre-restore scripts
    checkpoint_test.go:163: (00.004759) 	RPC
    checkpoint_test.go:163: (00.006035) Saved netns fd for links restore
    checkpoint_test.go:163: (00.006092) uns: Daemon started
    checkpoint_test.go:163: (00.006284) mnt: Reading mountpoint images (id 13 pid 1)
    checkpoint_test.go:163: (00.006311) mnt: 		Will mount 479 from /
    checkpoint_test.go:163: (00.006319) mnt: 		Will mount 479 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000479 /sys/firmware
    checkpoint_test.go:163: (00.006322) mnt: 	Read 479 mp @ /sys/firmware
    checkpoint_test.go:163: (00.006332) mnt: 		Will mount 477 from /dev/null (E)
    checkpoint_test.go:163: (00.006339) mnt: 		Will mount 477 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000477 /proc/kcore
    checkpoint_test.go:163: (00.006343) mnt: 	Read 477 mp @ /proc/kcore
    checkpoint_test.go:163: (00.006348) mnt: 		Will mount 475 from /bus
    checkpoint_test.go:163: (00.006351) mnt: 		Will mount 475 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000475 /proc/bus
    checkpoint_test.go:163: (00.006354) mnt: 	Read 475 mp @ /proc/bus
    checkpoint_test.go:163: (00.006359) mnt: 		Will mount 473 from /irq
    checkpoint_test.go:163: (00.006362) mnt: 		Will mount 473 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000473 /proc/irq
    checkpoint_test.go:163: (00.006365) mnt: 	Read 473 mp @ /proc/irq
    checkpoint_test.go:163: (00.006373) mnt: 		Will mount 471 from /sysrq-trigger
    checkpoint_test.go:163: (00.006385) mnt: 		Will mount 471 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000471 /proc/sysrq-trigger
    checkpoint_test.go:163: (00.006389) mnt: 	Read 471 mp @ /proc/sysrq-trigger
    checkpoint_test.go:163: (00.006397) mnt: 		Will mount 469 from /sys
    checkpoint_test.go:163: (00.006404) mnt: 		Will mount 469 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000469 /proc/sys
    checkpoint_test.go:163: (00.006407) mnt: 	Read 469 mp @ /proc/sys
    checkpoint_test.go:163: (00.006412) mnt: 		Will mount 597 from /dev/urandom (E)
    checkpoint_test.go:163: (00.006458) mnt: 		Will mount 597 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000597 /dev/urandom
    checkpoint_test.go:163: (00.006470) mnt: 	Read 597 mp @ /dev/urandom
    checkpoint_test.go:163: (00.006481) mnt: 		Will mount 596 from /dev/zero (E)
    checkpoint_test.go:163: (00.006488) mnt: 		Will mount 596 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000596 /dev/zero
    checkpoint_test.go:163: (00.006491) mnt: 	Read 596 mp @ /dev/zero
    checkpoint_test.go:163: (00.006496) mnt: 		Will mount 595 from /dev/tty (E)
    checkpoint_test.go:163: (00.006500) mnt: 		Will mount 595 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000595 /dev/tty
    checkpoint_test.go:163: (00.006503) mnt: 	Read 595 mp @ /dev/tty
    checkpoint_test.go:163: (00.006513) mnt: 		Will mount 594 from /dev/full (E)
    checkpoint_test.go:163: (00.006517) mnt: 		Will mount 594 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000594 /dev/full
    checkpoint_test.go:163: (00.006519) mnt: 	Read 594 mp @ /dev/full
    checkpoint_test.go:163: (00.006541) mnt: 		Will mount 587 from /dev/random (E)
    checkpoint_test.go:163: (00.006548) mnt: 		Will mount 587 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000587 /dev/random
    checkpoint_test.go:163: (00.006551) mnt: 	Read 587 mp @ /dev/random
    checkpoint_test.go:163: (00.006556) mnt: 		Will mount 586 from /dev/null (E)
    checkpoint_test.go:163: (00.006560) mnt: 		Will mount 586 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000586 /dev/null
    checkpoint_test.go:163: (00.006562) mnt: 	Read 586 mp @ /dev/null
    checkpoint_test.go:163: (00.006567) mnt: 		Will mount 585 from /
    checkpoint_test.go:163: (00.006570) mnt: 		Will mount 585 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000585 /sys
    checkpoint_test.go:163: (00.006573) mnt: 	Read 585 mp @ /sys
    checkpoint_test.go:163: (00.006581) mnt: 		Will mount 584 from /
    checkpoint_test.go:163: (00.006585) mnt: 		Will mount 584 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000584 /dev/shm
    checkpoint_test.go:163: (00.006588) mnt: 	Read 584 mp @ /dev/shm
    checkpoint_test.go:163: (00.006592) mnt: 		Will mount 582 from /
    checkpoint_test.go:163: (00.006596) mnt: 		Will mount 582 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000582 /dev/pts
    checkpoint_test.go:163: (00.006598) mnt: 	Read 582 mp @ /dev/pts
    checkpoint_test.go:163: (00.006603) mnt: 		Will mount 579 from /
    checkpoint_test.go:163: (00.006606) mnt: 		Will mount 579 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000579 /dev
    checkpoint_test.go:163: (00.006609) mnt: 	Read 579 mp @ /dev
    checkpoint_test.go:163: (00.006613) mnt: 		Will mount 578 from /
    checkpoint_test.go:163: (00.006617) mnt: 		Will mount 578 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000578 /proc
    checkpoint_test.go:163: (00.006620) mnt: 	Read 578 mp @ /proc
    checkpoint_test.go:163: (00.006627) mnt: 		Will mount 577 from /tmp/TestUsernsCheckpoint1975440553/001
    checkpoint_test.go:163: (00.006631) mnt: 		Will mount 577 @ /tmp/.criu.mntns.j5eNLw/mnt-0000000577 /
    checkpoint_test.go:163: (00.006634) mnt: 	Read 577 mp @ /
    checkpoint_test.go:163: (00.006642) mnt: Building mountpoints tree
    checkpoint_test.go:163: (00.006645) mnt: 	Building plain mount tree
    checkpoint_test.go:163: (00.006648) mnt: 		Working on 577->467
    checkpoint_test.go:163: (00.006652) mnt: 		Working on 578->577
    checkpoint_test.go:163: (00.006654) mnt: 		Working on 579->577
    checkpoint_test.go:163: (00.006657) mnt: 		Working on 582->579
    checkpoint_test.go:163: (00.006660) mnt: 		Working on 584->579
    checkpoint_test.go:163: (00.006663) mnt: 		Working on 585->577
    checkpoint_test.go:163: (00.006665) mnt: 		Working on 586->579
    checkpoint_test.go:163: (00.006668) mnt: 		Working on 587->579
    checkpoint_test.go:163: (00.006671) mnt: 		Working on 594->579
    checkpoint_test.go:163: (00.006674) mnt: 		Working on 595->579
    checkpoint_test.go:163: (00.006676) mnt: 		Working on 596->579
    checkpoint_test.go:163: (00.006679) mnt: 		Working on 597->579
    checkpoint_test.go:163: (00.006682) mnt: 		Working on 469->578
    checkpoint_test.go:163: (00.006684) mnt: 		Working on 471->578
    checkpoint_test.go:163: (00.006687) mnt: 		Working on 473->578
    checkpoint_test.go:163: (00.006690) mnt: 		Working on 475->578
    checkpoint_test.go:163: (00.006693) mnt: 		Working on 477->578
    checkpoint_test.go:163: (00.006695) mnt: 		Working on 479->585
    checkpoint_test.go:163: (00.006698) mnt: 	Resorting children of 577 in mount order
    checkpoint_test.go:163: (00.006702) mnt: 	Resorting children of 578 in mount order
    checkpoint_test.go:163: (00.006705) mnt: 	Resorting children of 469 in mount order
    checkpoint_test.go:163: (00.006708) mnt: 	Resorting children of 471 in mount order
    checkpoint_test.go:163: (00.006711) mnt: 	Resorting children of 473 in mount order
    checkpoint_test.go:163: (00.006714) mnt: 	Resorting children of 475 in mount order
    checkpoint_test.go:163: (00.006716) mnt: 	Resorting children of 477 in mount order
    checkpoint_test.go:163: (00.006719) mnt: 	Resorting children of 579 in mount order
    checkpoint_test.go:163: (00.006723) mnt: 	Resorting children of 582 in mount order
    checkpoint_test.go:163: (00.006726) mnt: 	Resorting children of 584 in mount order
    checkpoint_test.go:163: (00.006728) mnt: 	Resorting children of 586 in mount order
    checkpoint_test.go:163: (00.006731) mnt: 	Resorting children of 587 in mount order
    checkpoint_test.go:163: (00.006734) mnt: 	Resorting children of 594 in mount order
    checkpoint_test.go:163: (00.006736) mnt: 	Resorting children of 595 in mount order
    checkpoint_test.go:163: (00.006739) mnt: 	Resorting children of 596 in mount order
    checkpoint_test.go:163: (00.006749) mnt: 	Resorting children of 597 in mount order
    checkpoint_test.go:163: (00.006752) mnt: 	Resorting children of 585 in mount order
    checkpoint_test.go:163: (00.006755) mnt: 	Resorting children of 479 in mount order
    checkpoint_test.go:163: (00.006758) mnt: Done:
    checkpoint_test.go:163: (00.006761) mnt: [/](577->467)
    checkpoint_test.go:163: (00.006764) mnt:  [/proc](578->577)
    checkpoint_test.go:163: (00.006768) mnt:   [/proc/sys](469->578)
    checkpoint_test.go:163: (00.006771) mnt:   <--
    checkpoint_test.go:163: (00.006774) mnt:   [/proc/sysrq-trigger](471->578)
    checkpoint_test.go:163: (00.006777) mnt:   <--
    checkpoint_test.go:163: (00.006780) mnt:   [/proc/irq](473->578)
    checkpoint_test.go:163: (00.006783) mnt:   <--
    checkpoint_test.go:163: (00.006785) mnt:   [/proc/bus](475->578)
    checkpoint_test.go:163: (00.006788) mnt:   <--
    checkpoint_test.go:163: (00.006791) mnt:   [/proc/kcore](477->578)
    checkpoint_test.go:163: (00.006794) mnt:   <--
    checkpoint_test.go:163: (00.006797) mnt:  <--
    checkpoint_test.go:163: (00.006800) mnt:  [/dev](579->577)
    checkpoint_test.go:163: (00.006802) mnt:   [/dev/pts](582->579)
    checkpoint_test.go:163: (00.006805) mnt:   <--
    checkpoint_test.go:163: (00.006808) mnt:   [/dev/shm](584->579)
    checkpoint_test.go:163: (00.006811) mnt:   <--
    checkpoint_test.go:163: (00.006814) mnt:   [/dev/null](586->579)
    checkpoint_test.go:163: (00.006817) mnt:   <--
    checkpoint_test.go:163: (00.006819) mnt:   [/dev/random](587->579)
    checkpoint_test.go:163: (00.006822) mnt:   <--
    checkpoint_test.go:163: (00.006825) mnt:   [/dev/full](594->579)
    checkpoint_test.go:163: (00.006828) mnt:   <--
    checkpoint_test.go:163: (00.006831) mnt:   [/dev/tty](595->579)
    checkpoint_test.go:163: (00.006834) mnt:   <--
    checkpoint_test.go:163: (00.006836) mnt:   [/dev/zero](596->579)
    checkpoint_test.go:163: (00.006839) mnt:   <--
    checkpoint_test.go:163: (00.006842) mnt:   [/dev/urandom](597->579)
    checkpoint_test.go:163: (00.006845) mnt:   <--
    checkpoint_test.go:163: (00.006847) mnt:  <--
    checkpoint_test.go:163: (00.006850) mnt:  [/sys](585->577)
    checkpoint_test.go:163: (00.006853) mnt:   [/sys/firmware](479->585)
    checkpoint_test.go:163: (00.006856) mnt:   <--
    checkpoint_test.go:163: (00.006859) mnt:  <--
    checkpoint_test.go:163: (00.006862) mnt: <--
    checkpoint_test.go:163: (00.006865) mnt: 	The mount 469 is bind for 578 (@/proc/sys -> @/proc)
    checkpoint_test.go:163: (00.006869) mnt: 	The mount 471 is bind for 578 (@/proc/sysrq-trigger -> @/proc)
    checkpoint_test.go:163: (00.006872) mnt: 	The mount 473 is bind for 578 (@/proc/irq -> @/proc)
    checkpoint_test.go:163: (00.006875) mnt: 	The mount 475 is bind for 578 (@/proc/bus -> @/proc)
    checkpoint_test.go:163: (00.006878) mnt: 	The mount 587 is bind for 586 (@/dev/random -> @/dev/null)
    checkpoint_test.go:163: (00.006881) mnt: 	The mount 594 is bind for 586 (@/dev/full -> @/dev/null)
    checkpoint_test.go:163: (00.006884) mnt: 	The mount 595 is bind for 586 (@/dev/tty -> @/dev/null)
    checkpoint_test.go:163: (00.006887) mnt: 	The mount 596 is bind for 586 (@/dev/zero -> @/dev/null)
    checkpoint_test.go:163: (00.006890) mnt: 	The mount 597 is bind for 586 (@/dev/urandom -> @/dev/null)
    checkpoint_test.go:163: (00.006893) mnt: 	The mount 477 is bind for 586 (@/proc/kcore -> @/dev/null)
    checkpoint_test.go:163: (00.006897) mnt: Start with 577:/
    checkpoint_test.go:163: (00.006904) mnt-v2: Inspecting sharing on 577 shared_id 0 master_id 1 (@/)
    checkpoint_test.go:163: (00.006917) mnt-v2: Inspecting sharing on 586 shared_id 0 master_id 21 (@/dev/null)
    checkpoint_test.go:163: (00.006920) mnt-v2: Inspecting sharing on 587 shared_id 0 master_id 21 (@/dev/random)
    checkpoint_test.go:163: (00.006924) mnt-v2: Inspecting sharing on 594 shared_id 0 master_id 21 (@/dev/full)
    checkpoint_test.go:163: (00.006927) mnt-v2: Inspecting sharing on 595 shared_id 0 master_id 21 (@/dev/tty)
    checkpoint_test.go:163: (00.006929) mnt-v2: Inspecting sharing on 596 shared_id 0 master_id 21 (@/dev/zero)
    checkpoint_test.go:163: (00.006932) mnt-v2: Inspecting sharing on 597 shared_id 0 master_id 21 (@/dev/urandom)
    checkpoint_test.go:163: (00.006935) mnt-v2: Inspecting sharing on 477 shared_id 0 master_id 21 (@/proc/kcore)
    checkpoint_test.go:163: (00.006940) mnt-v2: Detected external slavery for shared group (0, 21) with source /dev/null
    checkpoint_test.go:163: (00.006945) mnt-v2: Detected external slavery for shared group (0, 1) with source /tmp/TestUsernsCheckpoint1975440553/002/test/criu-root
    checkpoint_test.go:163: (00.006948) mnt: Mountpoint 577 (@/) moved to the root yard
    checkpoint_test.go:163: (00.007026) No pidns-9.img image
    checkpoint_test.go:163: (00.007099) Warn  (criu/cr-restore.c:1296): Set CLONE_PARENT | CLONE_NEWPID but it might cause restore problem,because not all kernels support such clone flags combinations!
    checkpoint_test.go:163: (00.007104) Forking task with 1 pid (flags 0x7c028000)
    checkpoint_test.go:163: (00.007108) Creating process using clone3()
    checkpoint_test.go:163: (00.007649) PID: real 98084 virt 1
    checkpoint_test.go:163: (00.007943) Wait until namespaces are created
    checkpoint_test.go:163: (00.008686)      1: timens: monotonic -1 969321594
    checkpoint_test.go:163: (00.008710)      1: timens: boottime -1 969298473
    checkpoint_test.go:163: (00.008802) Running setup-namespaces scripts
    checkpoint_test.go:163: (00.008814) 	RPC
    checkpoint_test.go:163: (00.009075)      1: cg: Move into 2
    checkpoint_test.go:163: (00.009100)      1: cg:   `-> unified//test/integration/cgroup.procs
    checkpoint_test.go:163: (00.009104)      1: uns: calling userns_move (-1, 0)
    checkpoint_test.go:163: (00.009161) uns: daemon calls 0x43c6f0 (98084, -1, 0)
    checkpoint_test.go:163: (00.009253)      1: cg:   `-> blkio//test/integration/cgroup.procs
    checkpoint_test.go:163: (00.009259)      1: uns: calling userns_move (-1, 0)
    checkpoint_test.go:163: (00.009294) uns: daemon calls 0x43c6f0 (98084, -1, 0)
    checkpoint_test.go:163: (00.009356)      1: cg:   `-> cpu,cpuacct//test/integration/cgroup.procs
    checkpoint_test.go:163: (00.009373)      1: uns: calling userns_move (-1, 0)
    checkpoint_test.go:163: (00.009406) uns: daemon calls 0x43c6f0 (98084, -1, 0)
    checkpoint_test.go:163: (00.009485)      1: cg:   `-> cpuset//test/integration/cgroup.procs
    checkpoint_test.go:163: (00.009491)      1: uns: calling userns_move (-1, 0)
    checkpoint_test.go:163: (00.009513) uns: daemon calls 0x43c6f0 (98084, -1, 0)
    checkpoint_test.go:163: (00.009586)      1: cg:   `-> devices//test/integration/cgroup.procs
    checkpoint_test.go:163: (00.009591)      1: uns: calling userns_move (-1, 0)
    checkpoint_test.go:163: (00.009624) uns: daemon calls 0x43c6f0 (98084, -1, 0)
    checkpoint_test.go:163: (00.009668)      1: cg:   `-> freezer//test/integration/cgroup.procs
    checkpoint_test.go:163: (00.009672)      1: uns: calling userns_move (-1, 0)
    checkpoint_test.go:163: (00.009689) uns: daemon calls 0x43c6f0 (98084, -1, 0)
    checkpoint_test.go:163: (00.009726)      1: cg:   `-> hugetlb//test/integration/cgroup.procs
    checkpoint_test.go:163: (00.009730)      1: uns: calling userns_move (-1, 0)
    checkpoint_test.go:163: (00.009747) uns: daemon calls 0x43c6f0 (98084, -1, 0)
    checkpoint_test.go:163: (00.009783)      1: cg:   `-> memory//test/integration/cgroup.procs
    checkpoint_test.go:163: (00.009787)      1: uns: calling userns_move (-1, 0)
    checkpoint_test.go:163: (00.009804) uns: daemon calls 0x43c6f0 (98084, -1, 0)
    checkpoint_test.go:163: (00.009838)      1: cg:   `-> misc///cgroup.procs
    checkpoint_test.go:163: (00.009842)      1: uns: calling userns_move (-1, 0)
    checkpoint_test.go:163: (00.009858) uns: daemon calls 0x43c6f0 (98084, -1, 0)
    checkpoint_test.go:163: (00.009896)      1: cg:   `-> systemd//test/integration/cgroup.procs
    checkpoint_test.go:163: (00.009899)      1: uns: calling userns_move (-1, 0)
    checkpoint_test.go:163: (00.009916) uns: daemon calls 0x43c6f0 (98084, -1, 0)
    checkpoint_test.go:163: (00.009952)      1: cg:   `-> net_cls,net_prio//test/integration/cgroup.procs
    checkpoint_test.go:163: (00.009956)      1: uns: calling userns_move (-1, 0)
    checkpoint_test.go:163: (00.009972) uns: daemon calls 0x43c6f0 (98084, -1, 0)
    checkpoint_test.go:163: (00.010007)      1: cg:   `-> perf_event//test/integration/cgroup.procs
    checkpoint_test.go:163: (00.010011)      1: uns: calling userns_move (-1, 0)
    checkpoint_test.go:163: (00.010028) uns: daemon calls 0x43c6f0 (98084, -1, 0)
    checkpoint_test.go:163: (00.010063)      1: cg:   `-> pids//test/integration/cgroup.procs
    checkpoint_test.go:163: (00.010067)      1: uns: calling userns_move (-1, 0)
    checkpoint_test.go:163: (00.010084) uns: daemon calls 0x43c6f0 (98084, -1, 0)
    checkpoint_test.go:163: (00.010116)      1: Calling restore_sid() for init
    checkpoint_test.go:163: (00.010128)      1: Restoring 1 to 1 sid
    checkpoint_test.go:163: (00.010232)      1: Collecting 44/37 (flags 2)
    checkpoint_test.go:163: (00.010243)      1: No tty-info.img image
    checkpoint_test.go:163: (00.010255)      1:  `- ... done
    checkpoint_test.go:163: (00.010258)      1: Collecting 45/51 (flags 0)
    checkpoint_test.go:163: (00.010264)      1: No tty-data.img image
    checkpoint_test.go:163: (00.010267)      1:  `- ... done
    checkpoint_test.go:163: (00.010270)      1: Restoring namespaces 1 flags 0x7c028000
    checkpoint_test.go:163: (00.010310)      1: uns: calling __userns_sysctl_op (17, 0)
    checkpoint_test.go:163: (00.010373) uns: daemon calls 0x4a8f70 (98084, 14, 0)
    checkpoint_test.go:163: (00.010866) kernel/hostname nr 11
    checkpoint_test.go:163: (00.010961) kernel/domainname nr 6
    checkpoint_test.go:163: (00.011768)      1: Restoring IPC namespace
    checkpoint_test.go:163: (00.011782)      1: Restoring IPC variables
    checkpoint_test.go:163: (00.011824)      1: uns: calling __userns_sysctl_op (7, 0)
    checkpoint_test.go:163: (00.011896) uns: daemon calls 0x4a8f70 (98084, 14, 0)
    checkpoint_test.go:163: (00.013109)      1: Restoring IPC shared memory
    checkpoint_test.go:163: (00.013129)      1: No ipcns-shm-11.img image
    checkpoint_test.go:163: (00.013134)      1: Restoring IPC message queues
    checkpoint_test.go:163: (00.013140)      1: No ipcns-msg-11.img image
    checkpoint_test.go:163: (00.013143)      1: Restoring IPC semaphores sets
    checkpoint_test.go:163: (00.013148)      1: No ipcns-sem-11.img image
    checkpoint_test.go:163: (00.015813)      1: Try to restore a link 10:1:lo
    checkpoint_test.go:163: (00.015823)      1: Restoring link lo type 1
    checkpoint_test.go:163: (00.017070)      1: 	Running ip addr restore  
    checkpoint_test.go:163: RTNETLINK answers: File exists
    checkpoint_test.go:163: RTNETLINK answers: File exists
    checkpoint_test.go:163: (00.019660)      1: 	Running ip route restore  
    checkpoint_test.go:163: (00.021863)      1: 	Running ip route restore  
    checkpoint_test.go:163: (00.024098)      1: 	Running ip rule flush  
    checkpoint_test.go:163: (00.026221)      1: 	Running ip rule delete table local
    checkpoint_test.go:163: (00.028330)      1: 	Running ip rule restore  
    checkpoint_test.go:163: (00.033150)      1: 	Running iptables-restore -w for iptables-restore -w
    checkpoint_test.go:163: (00.038695)      1: 	Running ip6tables-restore -w for ip6tables-restore -w
    checkpoint_test.go:163: (00.051003)      1: No netns-ct-10.img image
    checkpoint_test.go:163: (00.051016)      1: No netns-exp-10.img image
    checkpoint_test.go:163: (00.051322)      1: mnt: Restoring mount namespace
    checkpoint_test.go:163: (00.051694)      1: mnt: Move the root to /tmp/.criu.mntns.j5eNLw
    checkpoint_test.go:163: (00.052117)      1: mnt: Start with 0:(null)
    checkpoint_test.go:163: (00.052138)      1: mnt-v2: can_mount_now_v2: true as 577 is global root
    checkpoint_test.go:163: (00.052143)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000577 for 577
    checkpoint_test.go:163: (00.052156)      1: mnt-v2: 	Mounting unsupported @577 (0)
    checkpoint_test.go:163: (00.052164)      1: uns: calling mount_root (12, 0)
    checkpoint_test.go:163: (00.052233) uns: daemon calls 0x46f630 (98084, 14, 0)
    checkpoint_test.go:163: (00.052513)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000577 for 577
    checkpoint_test.go:163: (00.052581)      1: mnt-v2: Mount 578 is detected as dir-mount
    checkpoint_test.go:163: (00.052587)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000578 for 578
    checkpoint_test.go:163: (00.052608)      1: mnt-v2: 	Mounting proc @578 (0)
    checkpoint_test.go:163: (00.052698)      1: mnt-v2: 		Propagate 578 to 475
    checkpoint_test.go:163: (00.052709)      1: mnt-v2: 		Propagate 578 to 473
    checkpoint_test.go:163: (00.052716)      1: mnt-v2: 		Propagate 578 to 471
    checkpoint_test.go:163: (00.052719)      1: mnt-v2: 		Propagate 578 to 469
    checkpoint_test.go:163: (00.052750)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000578 for 578
    checkpoint_test.go:163: (00.052791)      1: mnt-v2: Mount 469 is detected as dir-mount
    checkpoint_test.go:163: (00.052796)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000469 for 469
    checkpoint_test.go:163: (00.052806)      1: mnt-v2: 	Mounting proc @469 (0)
    checkpoint_test.go:163: (00.052811)      1: mnt-v2: 	Bind /tmp/.criu.mntns.j5eNLw/mnt-0000000578/sys to /tmp/.criu.mntns.j5eNLw/mnt-0000000469
    checkpoint_test.go:163: (00.052850)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000469 for 469
    checkpoint_test.go:163: (00.052881)      1: mnt-v2: Mount 471 is detected as file-mount
    checkpoint_test.go:163: (00.052885)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000471 for 471
    checkpoint_test.go:163: (00.052908)      1: mnt-v2: 	Mounting proc @471 (0)
    checkpoint_test.go:163: (00.052912)      1: mnt-v2: 	Bind /tmp/.criu.mntns.j5eNLw/mnt-0000000578/sysrq-trigger to /tmp/.criu.mntns.j5eNLw/mnt-0000000471
    checkpoint_test.go:163: (00.052951)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000471 for 471
    checkpoint_test.go:163: (00.052984)      1: mnt-v2: Mount 473 is detected as dir-mount
    checkpoint_test.go:163: (00.052989)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000473 for 473
    checkpoint_test.go:163: (00.052998)      1: mnt-v2: 	Mounting proc @473 (0)
    checkpoint_test.go:163: (00.053002)      1: mnt-v2: 	Bind /tmp/.criu.mntns.j5eNLw/mnt-0000000578/irq to /tmp/.criu.mntns.j5eNLw/mnt-0000000473
    checkpoint_test.go:163: (00.053038)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000473 for 473
    checkpoint_test.go:163: (00.053066)      1: mnt-v2: Mount 475 is detected as dir-mount
    checkpoint_test.go:163: (00.053070)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000475 for 475
    checkpoint_test.go:163: (00.053080)      1: mnt-v2: 	Mounting proc @475 (0)
    checkpoint_test.go:163: (00.053084)      1: mnt-v2: 	Bind /tmp/.criu.mntns.j5eNLw/mnt-0000000578/bus to /tmp/.criu.mntns.j5eNLw/mnt-0000000475
    checkpoint_test.go:163: (00.053121)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000475 for 475
    checkpoint_test.go:163: (00.053140)      1: mnt-v2: can_mount_now_v2: true as 477 is external
    checkpoint_test.go:163: (00.053153)      1: mnt-v2: Mount 477 is detected as file-mount
    checkpoint_test.go:163: (00.053156)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000477 for 477
    checkpoint_test.go:163: (00.053170)      1: mnt-v2: 	Mounting devtmpfs @477 (0)
    checkpoint_test.go:163: (00.053174)      1: mnt-v2: 	Bind /dev/null to /tmp/.criu.mntns.j5eNLw/mnt-0000000477
    checkpoint_test.go:163: (00.053196)      1: mnt-v2: 		Propagate 477 to 586
    checkpoint_test.go:163: (00.053226)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000477 for 477
    checkpoint_test.go:163: (00.053254)      1: mnt-v2: Mount 579 is detected as dir-mount
    checkpoint_test.go:163: (00.053258)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000579 for 579
    checkpoint_test.go:163: (00.053267)      1: mnt-v2: 	Mounting tmpfs @579 (0)
    checkpoint_test.go:163: (00.058345)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000579 for 579
    checkpoint_test.go:163: (00.058396)      1: mnt-v2: Mount 582 is detected as dir-mount
    checkpoint_test.go:163: (00.058402)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000582 for 582
    checkpoint_test.go:163: (00.058418)      1: mnt-v2: 	Mounting devpts @582 (0)
    checkpoint_test.go:163: (00.058592)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000582 for 582
    checkpoint_test.go:163: (00.058642)      1: mnt-v2: Mount 584 is detected as dir-mount
    checkpoint_test.go:163: (00.058647)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000584 for 584
    checkpoint_test.go:163: (00.058657)      1: mnt-v2: 	Mounting tmpfs @584 (0)
    checkpoint_test.go:163: (00.063670)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000584 for 584
    checkpoint_test.go:163: (00.063714)      1: mnt-v2: can_mount_now_v2: true as 586 is external
    checkpoint_test.go:163: (00.063725)      1: mnt-v2: Mount 586 is detected as file-mount
    checkpoint_test.go:163: (00.063728)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000586 for 586
    checkpoint_test.go:163: (00.063744)      1: mnt-v2: 	Mounting devtmpfs @586 (0)
    checkpoint_test.go:163: (00.063761)      1: mnt-v2: 	Bind /dev/null to /tmp/.criu.mntns.j5eNLw/mnt-0000000586
    checkpoint_test.go:163: (00.063809)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000586 for 586
    checkpoint_test.go:163: (00.063836)      1: mnt-v2: can_mount_now_v2: true as 587 is external
    checkpoint_test.go:163: (00.063843)      1: mnt-v2: Mount 587 is detected as file-mount
    checkpoint_test.go:163: (00.063846)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000587 for 587
    checkpoint_test.go:163: (00.063860)      1: mnt-v2: 	Mounting devtmpfs @587 (0)
    checkpoint_test.go:163: (00.063864)      1: mnt-v2: 	Bind /dev/random to /tmp/.criu.mntns.j5eNLw/mnt-0000000587
    checkpoint_test.go:163: (00.063899)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000587 for 587
    checkpoint_test.go:163: (00.063921)      1: mnt-v2: can_mount_now_v2: true as 594 is external
    checkpoint_test.go:163: (00.063928)      1: mnt-v2: Mount 594 is detected as file-mount
    checkpoint_test.go:163: (00.063931)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000594 for 594
    checkpoint_test.go:163: (00.063944)      1: mnt-v2: 	Mounting devtmpfs @594 (0)
    checkpoint_test.go:163: (00.063948)      1: mnt-v2: 	Bind /dev/full to /tmp/.criu.mntns.j5eNLw/mnt-0000000594
    checkpoint_test.go:163: (00.063985)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000594 for 594
    checkpoint_test.go:163: (00.064010)      1: mnt-v2: can_mount_now_v2: true as 595 is external
    checkpoint_test.go:163: (00.064017)      1: mnt-v2: Mount 595 is detected as file-mount
    checkpoint_test.go:163: (00.064020)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000595 for 595
    checkpoint_test.go:163: (00.064032)      1: mnt-v2: 	Mounting devtmpfs @595 (0)
    checkpoint_test.go:163: (00.064036)      1: mnt-v2: 	Bind /dev/tty to /tmp/.criu.mntns.j5eNLw/mnt-0000000595
    checkpoint_test.go:163: (00.064069)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000595 for 595
    checkpoint_test.go:163: (00.064091)      1: mnt-v2: can_mount_now_v2: true as 596 is external
    checkpoint_test.go:163: (00.064097)      1: mnt-v2: Mount 596 is detected as file-mount
    checkpoint_test.go:163: (00.064100)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000596 for 596
    checkpoint_test.go:163: (00.064112)      1: mnt-v2: 	Mounting devtmpfs @596 (0)
    checkpoint_test.go:163: (00.064116)      1: mnt-v2: 	Bind /dev/zero to /tmp/.criu.mntns.j5eNLw/mnt-0000000596
    checkpoint_test.go:163: (00.064200)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000596 for 596
    checkpoint_test.go:163: (00.064223)      1: mnt-v2: can_mount_now_v2: true as 597 is external
    checkpoint_test.go:163: (00.064229)      1: mnt-v2: Mount 597 is detected as file-mount
    checkpoint_test.go:163: (00.064232)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000597 for 597
    checkpoint_test.go:163: (00.064245)      1: mnt-v2: 	Mounting devtmpfs @597 (0)
    checkpoint_test.go:163: (00.064249)      1: mnt-v2: 	Bind /dev/urandom to /tmp/.criu.mntns.j5eNLw/mnt-0000000597
    checkpoint_test.go:163: (00.064314)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000597 for 597
    checkpoint_test.go:163: (00.064342)      1: mnt-v2: Mount 585 is detected as dir-mount
    checkpoint_test.go:163: (00.064346)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000585 for 585
    checkpoint_test.go:163: (00.064357)      1: mnt-v2: 	Mounting sysfs @585 (0)
    checkpoint_test.go:163: (00.064559)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000585 for 585
    checkpoint_test.go:163: (00.064611)      1: mnt-v2: Mount 479 is detected as dir-mount
    checkpoint_test.go:163: (00.064616)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000479 for 479
    checkpoint_test.go:163: (00.064627)      1: mnt-v2: 	Mounting tmpfs @479 (0)
    checkpoint_test.go:163: (00.069111)      1: uns: calling apply_sb_flags (12, 0)
    checkpoint_test.go:163: (00.069219) uns: daemon calls 0x46f620 (98084, 12, 0)
    checkpoint_test.go:163: (00.069401)      1: mnt-v2: Create plain mountpoint /tmp/.criu.mntns.j5eNLw/mnt-0000000479 for 479
    checkpoint_test.go:163: (00.069494)      1: mnt: Start with 577:/
    checkpoint_test.go:163: (00.069517)      1: mnt-v2: Move mount 577 from /tmp/.criu.mntns.j5eNLw/mnt-0000000577 to /tmp/.criu.mntns.j5eNLw/13-0000000000/
    checkpoint_test.go:163: (00.069577)      1: mnt-v2: Move mount 578 from /tmp/.criu.mntns.j5eNLw/mnt-0000000578 to /tmp/.criu.mntns.j5eNLw/13-0000000000/proc
    checkpoint_test.go:163: (00.069613)      1: mnt-v2: Move mount 469 from /tmp/.criu.mntns.j5eNLw/mnt-0000000469 to /tmp/.criu.mntns.j5eNLw/13-0000000000/proc/sys
    checkpoint_test.go:163: (00.069636)      1: mnt-v2: Move mount 471 from /tmp/.criu.mntns.j5eNLw/mnt-0000000471 to /tmp/.criu.mntns.j5eNLw/13-0000000000/proc/sysrq-trigger
    checkpoint_test.go:163: (00.069657)      1: mnt-v2: Move mount 473 from /tmp/.criu.mntns.j5eNLw/mnt-0000000473 to /tmp/.criu.mntns.j5eNLw/13-0000000000/proc/irq
    checkpoint_test.go:163: (00.069676)      1: mnt-v2: Move mount 475 from /tmp/.criu.mntns.j5eNLw/mnt-0000000475 to /tmp/.criu.mntns.j5eNLw/13-0000000000/proc/bus
    checkpoint_test.go:163: (00.069710)      1: mnt-v2: Move mount 477 from /tmp/.criu.mntns.j5eNLw/mnt-0000000477 to /tmp/.criu.mntns.j5eNLw/13-0000000000/proc/kcore
    checkpoint_test.go:163: (00.069749)      1: mnt-v2: Move mount 579 from /tmp/.criu.mntns.j5eNLw/mnt-0000000579 to /tmp/.criu.mntns.j5eNLw/13-0000000000/dev
    checkpoint_test.go:163: (00.069774)      1: mnt-v2: Move mount 582 from /tmp/.criu.mntns.j5eNLw/mnt-0000000582 to /tmp/.criu.mntns.j5eNLw/13-0000000000/dev/pts
    checkpoint_test.go:163: (00.069796)      1: mnt-v2: Move mount 584 from /tmp/.criu.mntns.j5eNLw/mnt-0000000584 to /tmp/.criu.mntns.j5eNLw/13-0000000000/dev/shm
    checkpoint_test.go:163: (00.069815)      1: mnt-v2: Move mount 586 from /tmp/.criu.mntns.j5eNLw/mnt-0000000586 to /tmp/.criu.mntns.j5eNLw/13-0000000000/dev/null
    checkpoint_test.go:163: (00.069834)      1: mnt-v2: Move mount 587 from /tmp/.criu.mntns.j5eNLw/mnt-0000000587 to /tmp/.criu.mntns.j5eNLw/13-0000000000/dev/random
    checkpoint_test.go:163: (00.069852)      1: mnt-v2: Move mount 594 from /tmp/.criu.mntns.j5eNLw/mnt-0000000594 to /tmp/.criu.mntns.j5eNLw/13-0000000000/dev/full
    checkpoint_test.go:163: (00.069871)      1: mnt-v2: Move mount 595 from /tmp/.criu.mntns.j5eNLw/mnt-0000000595 to /tmp/.criu.mntns.j5eNLw/13-0000000000/dev/tty
    checkpoint_test.go:163: (00.069889)      1: mnt-v2: Move mount 596 from /tmp/.criu.mntns.j5eNLw/mnt-0000000596 to /tmp/.criu.mntns.j5eNLw/13-0000000000/dev/zero
    checkpoint_test.go:163: (00.069927)      1: mnt-v2: Move mount 597 from /tmp/.criu.mntns.j5eNLw/mnt-0000000597 to /tmp/.criu.mntns.j5eNLw/13-0000000000/dev/urandom
    checkpoint_test.go:163: (00.069955)      1: mnt-v2: Move mount 585 from /tmp/.criu.mntns.j5eNLw/mnt-0000000585 to /tmp/.criu.mntns.j5eNLw/13-0000000000/sys
    checkpoint_test.go:163: (00.069976)      1: mnt-v2: Move mount 479 from /tmp/.criu.mntns.j5eNLw/mnt-0000000479 to /tmp/.criu.mntns.j5eNLw/13-0000000000/sys/firmware
    checkpoint_test.go:163: (00.069994)      1: mnt: Move the root to /tmp/.criu.mntns.j5eNLw/13-0000000000
    checkpoint_test.go:163: (00.070207)      1: uns: calling __move_mount_set_group (-1, 0)
    checkpoint_test.go:163: (00.070264) uns: daemon calls 0x46d3b0 (98084, -1, 0)
    checkpoint_test.go:163: (00.070403)      1: uns: calling __move_mount_set_group (-1, 0)
    checkpoint_test.go:163: (00.070465) uns: daemon calls 0x46d3b0 (98084, -1, 0)
    checkpoint_test.go:163: (00.070577)      1: uns: calling __move_mount_set_group (-1, 0)
    checkpoint_test.go:163: (00.070640) uns: daemon calls 0x46d3b0 (98084, -1, 0)
    checkpoint_test.go:163: (00.070693) Error (criu/mount-v2.c:891): mnt-v2: Failed to copy sharing from 31: to 17: Invalid argument
    checkpoint_test.go:163: (00.070749)      1: Error (criu/mount-v2.c:1019): mnt-v2: Failed to copy sharing from 595 to 477
    checkpoint_test.go:163: (00.071836) uns: calling exit_usernsd (-1, 1)
    checkpoint_test.go:163: (00.071892) uns: daemon calls 0x478610 (98082, -1, 1)
    checkpoint_test.go:163: (00.071901) uns: `- daemon exits w/ 0
    checkpoint_test.go:163: (00.072666) uns: daemon stopped
    checkpoint_test.go:163: (00.072678) Error (criu/cr-restore.c:2536): Restoring FAILED.
    checkpoint_test.go:163: === END ===
    checkpoint_test.go:164: criu failed: type NOTIFY errno 0
        log file: /tmp/TestUsernsCheckpoint1975440553/003/criu/restore.log

To replicate this you can try the following commands:

# git clone https://github.com/opencontainers/runc
# cd runc/libcontainer/integration/
# go test -timeout 3m -tags "seccomp" -run TestUsernsCheckpoint  -v ./...

@Snorch Snorch force-pushed the mount-v2-wa-multiple-external-bindmounts-without-common-root branch from aacedb3 to 0c3edd2 Compare May 19, 2022 21:08
@Snorch
Copy link
Member Author

Snorch commented May 19, 2022

Sorry it was a stupid mistake:

diff --git a/criu/mount-v2.c b/criu/mount-v2.c
index 2b6e48206..3f6a41108 100644
--- a/criu/mount-v2.c
+++ b/criu/mount-v2.c
@@ -1013,11 +1013,11 @@ static int restore_one_sharing_group(struct sharing_group *sg)
                                       other->mnt_id, other->root);
                                return -1;
                        }
-               }
-
-               if (move_mount_set_group(first->mnt_fd_id, NULL, other->mnt_fd_id)) {
-                       pr_err("Failed to copy sharing from %d to %d\n", first->mnt_id, other->mnt_id);
-                       return -1;
+               } else {
+                       if (move_mount_set_group(first->mnt_fd_id, NULL, other->mnt_fd_id)) {
+                               pr_err("Failed to copy sharing from %d to %d\n", first->mnt_id, other->mnt_id);
+                               return -1;
+                       }
                }
        }

After this I have test passing:

[root@ integration]# BUILDTAGS="" go test -timeout 3m -tags "seccomp" -run TestUsernsCheckpoint  -v ./...
=== RUN   TestUsernsCheckpoint
--- PASS: TestUsernsCheckpoint (0.26s)
PASS
ok  	github.com/opencontainers/runc/libcontainer/integration	(cached)
[root@ integration]# 

@Snorch
Copy link
Member Author

Snorch commented May 19, 2022

@Snorch Would it be possible to add a test for this case?

Sure, will do tomorrow. (done)

@adrianreber
Copy link
Member

The current version of the PR works with runc. Thanks.

I also restarted the failing tests.

Copy link
Member

@rst0git rst0git left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

criu/mount-v2.c Outdated Show resolved Hide resolved
Snorch added 3 commits May 31, 2022 18:56
This helper restores master_id and shared_id of first mount in the
sharing group. It first copies sharing from either external source or
internal parent sharing group and makes master_id from shared_id. Next
it creates new shared_id when needed.

All other mounts except first are just copied from the first one.

Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
…root

It's a problem when while restoring sharing group we need to copy
sharing between two mounts with non-intersecting roots, because kernel
does not allow it.

We have a case opencontainers/runc#3442, where
runc adds different devtmpfs file-bindmounts to container and there is
no fsroot mount in container for this devtmpfs, thus mount-v2 faces the
above problem.

Luckily for the case of external mounts which are in one sharing group
and which have non-intersecting roots, these mounts likely only have
external master with no sharing, so we can just copy sharing from
external source and make it slave as a workaround.

checkpoint-restore#1886

Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
@Snorch Snorch force-pushed the mount-v2-wa-multiple-external-bindmounts-without-common-root branch from d13d392 to 24bf471 Compare May 31, 2022 16:07
@Snorch
Copy link
Member Author

Snorch commented May 31, 2022

I've added a more detailed comment in code and also a bit simplified the code by splitting restore_one_sharing function of restoring sharing on first mount of shared group, and reusing it in w/a.

This makes clear that w/a is about restoring "other" mount similar to how we've restored "first" mount in case we can't copy from "first".

upd: And rechecked runc usecase, looks working after this change.

@avagin avagin merged commit 58a2d98 into checkpoint-restore:criu-dev Jun 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants