Correctly restore containers with nested bind mounts #2798

adrianreber · 2021-02-08T13:28:08Z

This tries to fix #2748

This change is motivated by the work in kubernetes/kubernetes#97194

During restore CRIU mounts all mounts as they were during checkpointing. Using nested bind mounts this can lead to situation where a mountpoint in the parent bind mount is missing.

During initial container creation 'runc' mounts a bind mount and then creates the mountpoints for nested mountpoints.

This PR changes runc to also mount all bind mounts before telling CRIU to restore the container to re-create missing mountpoints.

libcontainer/container_linux.go

tests/integration/checkpoint.bats

libcontainer/container_linux.go

tests/integration/checkpoint.bats

kolyshkin

Just a single nit about making temp dirs under CWD, LGTM otherwise, thanks @adrianreber!

I have also checked that the test case indeed fails before the fix.

runc was already re-creating mountpoints before calling CRIU to restore a container. But mountpoints inside a bind mount were not re-created. During initial container creation runc will mount bind mounts and then create the necessary mountpoints for further mounts inside those bind mounts. If, for example, one of the bind mounts is a tmpfs and empty before restore, CRIU will fail re-mounting all mounts because the mountpoints in the bind mounted tmpfs no longer exist. CRIU expects all mount points to exist as during checkpointing. This changes runc to mount bind mounts after mountpoint creation to ensure nested bind mounts have their mountpoints created before CRIU does the restore. Signed-off-by: Adrian Reber <areber@redhat.com>

This adds a checkpoint/restore test to ensure that nested bind mount mountpoints are correctly re-created. Signed-off-by: Adrian Reber <areber@redhat.com>

adrianreber · 2021-02-10T07:57:15Z

Fixed mktemp to use -p . and removed old comment.

kolyshkin

LGTM, and thanks!

kolyshkin · 2021-03-10T22:30:09Z

@cyphar @AkihiroSuda @mrunalp PTAL

AkihiroSuda · 2021-03-17T02:46:21Z

test (1.16.x) Expected — Waiting for status to be reported

CI seems to have got stuck, attempting to restart by closing & reopening this PR

The upstream patch (itself backported for rc93) was backported to rc92. Original cover letter: From 14faf1c20948688a48edb9b41367ab07ac11ca91 Mon Sep 17 00:00:00 2001 From: Aleksa Sarai <cyphar@cyphar.com> Date: Wed, 28 Apr 2021 15:44:36 +1000 Subject: [PATCH 0/5] rootfs: add mount destination validation This is a backport of the fix for CVE-2021-30465 to the v1.0.0-rc93 release. However, because the patch does not apply cleanly it was necessary to backport the following commits (from [1]): * deb8a8dd7767 ("libct/newInitConfig: nit") * 1e476578b6cd ("libct/rootfs: introduce and use mountConfig") * 3826db196d59 ("libct/rootfs/mountCgroupV2: minor refactor") * ff692f289b60 ("Fix cgroup2 mount for rootless case") And the patch itself was modified to remove hardenings for code which didn't exist in v1.0.0-rc93 (in particular, the mount changes in [2]). [1]: opencontainers/runc#2818 [2]: opencontainers/runc#2798 Aleksa Sarai (1): rootfs: add mount destination validation Kir Kolyshkin (4): libct/newInitConfig: nit libct/rootfs: introduce and use mountConfig libct/rootfs/mountCgroupV2: minor refactor Fix cgroup2 mount for rootless case libcontainer/container_linux.go | 11 +- libcontainer/init_linux.go | 1 + libcontainer/rootfs_linux.go | 291 +++++++++++++++++-------------- libcontainer/specconv/example.go | 18 +- libcontainer/utils/utils.go | 54 ++++++ libcontainer/utils/utils_test.go | 35 ++++ 6 files changed, 263 insertions(+), 147 deletions(-) -- 2.31.1

adrianreber force-pushed the 2021-02-08-nested-bind.mounts branch from de92cdc to 155f6d5 Compare February 8, 2021 13:38

cyphar changed the title ~~Correcty restore containers with nested bind mounts~~ Correctly restore containers with nested bind mounts Feb 8, 2021

adrianreber force-pushed the 2021-02-08-nested-bind.mounts branch from 155f6d5 to 884adab Compare February 8, 2021 14:19