libcontainer: remove all mount logic from nsexec #3985

cyphar · 2023-08-17T03:27:34Z

Split out from #3953 to be reviewed separately.

~~Draft until #4124 is merged.~~

~~Draft until #4003 is merged.~~

~~Draft until #3967 is merged.~~

With open_tree(OPEN_TREE_CLONE), it is possible to implement both the
id-mapped mounts and bind-mount source file descriptor logic entirely in
Go without requiring any complicated handling from nsexec.

This allows us to remove the amount of C code we have in nsexec, as well
as simplifying a whole host of places that were made more complicated
with the addition of id-mapped mounts and the bind sourcefd logic. The
one downside of this is that the bind sourcefd feature now depends on
Linux 5.4. (EDIT: We just pass a regular file descriptor, so there's no
change)

In addition, we can easily add support for id-mappings that don't match
the container's user namespace. The approach taken here is to use Go's
officially supported mechanism for spawning a process in a user
namespace, but (ab)use PTRACE_TRACEME to avoid actually having to exec a
different process. The most efficient way to implement this would be to
do clone() in cgo directly to run a function that just does
kill(getpid(), SIGSTOP) -- we can always switch to that if it turns out
this approach is too slow.

This also includes a fix for a fairly sneaky bug caused by commit 9c44407
("Open bind mount sources from the host userns") and further exacerbated by
commit fda12ab ("Support idmap mounts on volumes"), where opening the
mount sources long before we do any of the container rootfs mounts results in
undesirable behaviour with regards to mount propagation and unconventional
mount entries that use the container filesystem as the source. The solution is
to have a thread that joins the container mount namespace and does on-demand
generation of mount fds when requested.

Fixes: fda12ab ("Support idmap mounts on volumes")
Fixes: 9c44407 ("Open bind mount sources from the host userns")
Signed-off-by: Aleksa Sarai cyphar@cyphar.com

kolyshkin · 2023-08-26T01:32:16Z

@cyphar two issues:

squash! libcontainer: remove all mount logic from nsexec probably shouldn be renamed (or merged into other commits).
Not sure what's wrong with Cirrus CI, but it doesn't want to clone the repo.

cyphar · 2023-08-26T02:22:02Z

Will do, I wanted to make sure the rewrite-of-the-rewrite worked before doing the squash.
I pushed the branch to origin by accident and Cirrus has saved the origin branch name and won't re-fetch the PR (probably because the commit is the same). I think a force-push will fix it...

rata · 2023-08-28T11:05:05Z

It seems @cyphar is on vacations for two weeks, does someone wants to picks this up? Would that be okay?

cyphar · 2023-08-28T11:29:41Z

I will push the version that is rebased on top of #3967 (which fixes the test errors here). But if anything else is needed I can't really guarantee that I'll be able to get back to it before I get back in two weeks. Feel free to carry this if necessary @kolyshkin.

rata · 2023-08-28T11:31:07Z

@cyphar can we carry your other open PRs forward too? (if it happens to be needed)

cyphar · 2023-08-28T11:34:57Z

Yeah, of course.

rata · 2023-08-29T14:02:36Z

@kolyshkin can you carry this forward, then?

lifubang · 2023-08-30T12:41:19Z

It's so big that we need time to review and do some corner tests.

libcontainer/apparmor/apparmor_linux.go

libcontainer/cgroups/file.go

lifubang · 2023-08-30T13:20:28Z

libcontainer/cgroups/cgroups_test.go

@@ -5,6 +5,8 @@ import (
 )

 func TestParseCgroups(t *testing.T) {
+	// We don't need to use /proc/thread-self here because runc always runs


I think we could remove these comments in this file and other files.

Yeah, though the unfortunate thing is that knowing whether to use thread-self or not is a little bit delicate and so the comments (while repetitive) are kind of important.

lifubang · 2023-08-30T13:34:10Z

I think there are many refactor codes not about mount logic, I think we can split this PR to some small PRs, then it will be easy to review. I will do it ASAP, if someone wants to do it, it will be appreciated.

cyphar · 2023-08-31T05:57:25Z

Thanks @lifubang! Just be aware that while it would be nice to cherry-pick the thread-self patch, when I tried it became quite ugly to disentangle it from the mount changes.

lifubang · 2023-08-31T11:03:24Z

Part I: #4003

lifubang · 2023-08-31T11:04:15Z

when I tried it became quite ugly to disentangle it from the mount changes.

After #4003 merged, I'll see whether it's worth to do or not.

libcontainer/process_linux.go

AkihiroSuda · 2023-12-13T02:27:19Z

libcontainer/userns/usernsfd_linux.go

+	// We don't need to use /proc/thread-self here because the exe mm of a
+	// thread-group is guaranteed to be the same for all threads by definition.
+	// This lets us avoid having to do runtime.LockOSThread.
+	return os.StartProcess("/proc/self/exe", []string{"runc", "--help"}, &os.ProcAttr{


This can’t be just /bin/true ? (when it exists)

The path doesn't actually matter, we never call execve(2). We can't use "" or some other dummy path because the Go runtime checks that the path exists and is executable. I would prefer to not add code to spawn a random external executable (even if we don't actually execute it), and if /bin/true doesn't exist (or is not executable for some reason) we would need a fallback anyway.

AkihiroSuda

LGTM, but please consider to keep json field names consistent with Go variable names

cyphar · 2023-12-13T22:55:22Z

@AkihiroSuda I'm fixing them up now 😸.

libcontainer/mount_linux.go

With open_tree(OPEN_TREE_CLONE), it is possible to implement both the id-mapped mounts and bind-mount source file descriptor logic entirely in Go without requiring any complicated handling from nsexec. However, implementing it the naive way (do the OPEN_TREE_CLONE in the host namespace before the rootfs is set up -- which is what the existing implementation did) exposes issues in how mount ordering (in particular when handling mount sources from inside the container rootfs, but also in relation to mount propagation) was handled for idmapped mounts and bind-mount sources. In order to solve this problem completely, it is necessary to spawn a thread which joins the container mount namespace and provides mountfds when requested by the rootfs setup code (ensuring that the mount order and mount propagation of the source of the bind-mount are handled correctly). While the need to join the mount namespace leads to other complicated (such as with the usage of /proc/self -- fixed in a later patch) the resulting code is still reasonable and is the only real way to solve the issue. This allows us to reduce the amount of C code we have in nsexec, as well as simplifying a whole host of places that were made more complicated with the addition of id-mapped mounts and the bind sourcefd logic. Because we join the container namespace, we can continue to use regular O_PATH file descriptors for non-id-mapped bind-mount sources (which means we don't have to raise the kernel requirement for that case). In addition, we can easily add support for id-mappings that don't match the container's user namespace. The approach taken here is to use Go's officially supported mechanism for spawning a process in a user namespace, but (ab)use PTRACE_TRACEME to avoid actually having to exec a different process. The most efficient way to implement this would be to do clone() in cgo directly to run a function that just does kill(getpid(), SIGSTOP) -- we can always switch to that if it turns out this approach is too slow. It should be noted that the included micro-benchmark seems to indicate this is Fast Enough(TM): goos: linux goarch: amd64 pkg: github.com/opencontainers/runc/libcontainer/userns cpu: Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz BenchmarkSpawnProc BenchmarkSpawnProc-8 1670 770065 ns/op Fixes: fda12ab ("Support idmap mounts on volumes") Fixes: 9c44407 ("Open bind mount sources from the host userns") Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>

With the rework of nsexec.c to handle MOUNT_ATTR_IDMAP in our Go code we can now handle arbitrary mappings without issue, so remove the primary artificial limit of mappings (must use the same mapping as the container's userns) and add some tests. We still only support idmap mounts for bind-mounts because configuring mappings for other filesystems would require switching our entire mount machinery to the new mount API. The current design would easily allow for this but we would need to convert new mount options entirely to the fsopen/fsconfig/fsmount API. This can be done in the future. Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>

The primary change is a switch to using /proc/thread-self, which is needed for when we add a CLONE_FS thread to runc. Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>

With the idmap work, we will have a tainted Go thread in our thread-group that has a different mount namespace to the other threads. It seems that (due to some bad luck) the Go scheduler tends to make this thread the thread-group leader in our tests, which results in very baffling failures where /proc/self/mountinfo produces gibberish results. In order to avoid this, switch to using /proc/thread-self for everything that is thread-local. This primarily includes switching all file descriptor paths (CLONE_FS), all of the places that check the current cgroup (technically we never will run a single runc thread in a separate cgroup, but better to be safe than sorry), and the aforementioned mountinfo code. We don't need to do anything for the following because the results we need aren't thread-local: * Checks that certain namespaces are supported by stat(2)ing /proc/self/ns/... * /proc/self/exe and /proc/self/cmdline are not thread-local. * While threads can be in different cgroups, we do not do this for the runc binary (or libcontainer) and thus we do not need to switch to the thread-local version of /proc/self/cgroups. * All of the CLONE_NEWUSER files are not thread-local because you cannot set the usernamespace of a single thread (setns(CLONE_NEWUSER) is blocked for multi-threaded programs). Note that we have to use runtime.LockOSThread when we have an open handle to a tid-specific procfs file that we are operating on multiple times. Go can reschedule us such that we are running on a different thread and then kill the original thread (causing -ENOENT or similarly confusing errors). This is not strictly necessary for most usages of /proc/thread-self (such as using /proc/thread-self/fd/$n directly) since only operating on the actual inodes associated with the tid requires this locking, but because of the pre-3.17 fallback for CentOS, we have to do this in most cases. In addition, CentOS's kernel is too old for /proc/thread-self, which requires us to emulate it -- however in rootfs_linux.go, we are in the container pid namespace but /proc is the host's procfs. This leads to the incredibly frustrating situation where there is no way (on pre-4.1 Linux) to figure out which /proc/self/task/... entry refers to the current tid. We can just use /proc/self in this case. Yes this is all pretty ugly. I also wish it wasn't necessary. Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>

Our previous test for whether we can mount on top of /proc incorrectly assumed that it would only be called with bind-mount sources. This meant that having a non bind-mount entry for a pseudo-filesystem (like overlayfs) with a dummy source set to /proc on the host would let you bypass the check, which could easily lead to security issues. In addition, the check should be applied more uniformly to all mount types, so fix that as well. And add some tests for some of the tricky cases to make sure we protect against them properly. Fixes: 331692b ("Only allow proc mount if it is procfs") Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>

If a user specifies a configuration like "rro, rrw", we should have similar behaviour to "ro, rw" where we clear the previous flags so that the last specified flag takes precendence. Fixes: 382eba4 ("Support recursive mount attrs ("rro", "rnosuid", "rnodev", ...)") Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>

ridmap indicates that the id mapping should be applied recursively (only really relevant for rbind mount entries), and idmap indicates that it should not be applied recursively (the default). If no mappings are specified for the mount, we use the userns configuration of the container. This matches the behaviour in the currently-unreleased runtime-spec. This includes a minor change to the state.json serialisation format, but because there has been no released version of runc with commit fbf183c ("Add uid and gid mappings to mounts"), we can safely make this change without affecting running containers. Doing it this way makes it much easier to handle m.IsIDMapped() and indicating that a mapping has been specified. Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>

Our previous implementation of idmapped mounts and bind-mount sources would open all of the source paths before we did any mounts, meaning that mounts using sources from inside the container rootfs would not be correct. This has been fixed with the new on-demand system, and so add some regression tests. Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>

cyphar · 2023-12-14T01:47:30Z

@kolyshkin Did you want to re-review this, or can we merge it as-is?

libcontainer/mount_linux.go

lifubang · 2024-07-03T15:17:31Z

runc/libcontainer/rootfs_linux.go

Line 734 in fa93c8b

    
           // If m is nil, don't stat the filesystem.  This is used for restore of a checkpoint.

Maybe here it should be: If m.Source is nil......
And maybe this line can be removed, because it seems like we don't care about it anymore.

cyphar · 2024-07-04T04:13:32Z

@lifubang Yeah, that comment was obsoleted by cdff09a ("rootfs; fix 'can we mount on top of /proc' check") but I didn't notice this until I did the backport in #4334.

(We can remove it, hopefully this didn't break anything.)

cyphar added this to the 1.2.0 milestone Aug 17, 2023

This comment was marked as resolved.

Sign in to view

This was referenced Aug 20, 2023

nsexec: spring cleaning #3953

Closed

Support for ID map mounts without userns #3943

Closed

This comment was marked as resolved.

Sign in to view

This was referenced Aug 22, 2023

[Merged] features-linux: Expose idmap information opencontainers/runtime-spec#1219

Closed

features: mountExtensions: how best to represent feature support for idmap? opencontainers/runtime-spec#1223

Open

This comment was marked as resolved.

Sign in to view

cyphar marked this pull request as ready for review August 24, 2023 06:12

This comment was marked as resolved.

Sign in to view

lifubang requested a review from kolyshkin August 30, 2023 12:39

lifubang added the status/2-code-review label Aug 30, 2023

lifubang reviewed Aug 30, 2023

View reviewed changes

libcontainer/apparmor/apparmor_linux.go Outdated Show resolved Hide resolved

lifubang reviewed Aug 30, 2023

View reviewed changes

libcontainer/cgroups/file.go Outdated Show resolved Hide resolved

lifubang reviewed Aug 30, 2023

View reviewed changes

lifubang mentioned this pull request Aug 31, 2023

[Carry #3985 Part I] code refactor for process sync #4003

Merged

AkihiroSuda reviewed Dec 13, 2023

View reviewed changes

libcontainer/process_linux.go Show resolved Hide resolved

AkihiroSuda reviewed Dec 13, 2023

View reviewed changes

AkihiroSuda approved these changes Dec 13, 2023

View reviewed changes

AkihiroSuda reviewed Dec 13, 2023

View reviewed changes

libcontainer/mount_linux.go Outdated Show resolved Hide resolved

AkihiroSuda approved these changes Dec 13, 2023

View reviewed changes

cyphar added 8 commits December 14, 2023 11:36

vendor: update to github.com/moby/sys/mountinfo@v0.7.1

a04d88e

The primary change is a switch to using /proc/thread-self, which is needed for when we add a CLONE_FS thread to runc. Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>

AkihiroSuda approved these changes Dec 14, 2023

View reviewed changes

AkihiroSuda added status/4-merge and removed status/2-code-review labels Dec 14, 2023

lifubang reviewed Dec 17, 2023

View reviewed changes

libcontainer/mount_linux.go Show resolved Hide resolved

lifubang approved these changes Dec 18, 2023

View reviewed changes

lifubang merged commit 371ff9c into opencontainers:main Dec 18, 2023
45 checks passed

cyphar deleted the idmap-generic branch December 18, 2023 05:13

lifubang mentioned this pull request Dec 18, 2023

fix a (u|g)IDMappings type value convertion error #4150

Merged

lifubang mentioned this pull request Jan 4, 2024

update spec-conformance.md(rsvd hugetlb cgroup and idmapped-mounts) #4159

Merged

cyphar mentioned this pull request Mar 14, 2024

release 1.2.0-rc.1 #4221

Merged

cyphar mentioned this pull request Jul 2, 2024

[1.1] rootfs: fix 'can we mount on top of /proc' check #4334

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

libcontainer: remove all mount logic from nsexec #3985

libcontainer: remove all mount logic from nsexec #3985

cyphar commented Aug 17, 2023 •

edited

Loading

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

kolyshkin commented Aug 26, 2023

cyphar commented Aug 26, 2023

rata commented Aug 28, 2023 •

edited

Loading

cyphar commented Aug 28, 2023

rata commented Aug 28, 2023 •

edited

Loading

cyphar commented Aug 28, 2023

rata commented Aug 29, 2023

lifubang commented Aug 30, 2023

lifubang Aug 30, 2023

cyphar Aug 31, 2023

lifubang commented Aug 30, 2023

cyphar commented Aug 31, 2023

lifubang commented Aug 31, 2023

lifubang commented Aug 31, 2023

AkihiroSuda Dec 13, 2023

cyphar Dec 13, 2023 •

edited

Loading

AkihiroSuda left a comment

cyphar commented Dec 13, 2023

cyphar commented Dec 14, 2023

lifubang commented Jul 3, 2024

cyphar commented Jul 4, 2024

libcontainer: remove all mount logic from nsexec #3985

libcontainer: remove all mount logic from nsexec #3985

Conversation

cyphar commented Aug 17, 2023 • edited Loading

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

kolyshkin commented Aug 26, 2023

cyphar commented Aug 26, 2023

rata commented Aug 28, 2023 • edited Loading

cyphar commented Aug 28, 2023

rata commented Aug 28, 2023 • edited Loading

cyphar commented Aug 28, 2023

rata commented Aug 29, 2023

lifubang commented Aug 30, 2023

lifubang Aug 30, 2023

Choose a reason for hiding this comment

cyphar Aug 31, 2023

Choose a reason for hiding this comment

lifubang commented Aug 30, 2023

cyphar commented Aug 31, 2023

lifubang commented Aug 31, 2023

lifubang commented Aug 31, 2023

AkihiroSuda Dec 13, 2023

Choose a reason for hiding this comment

cyphar Dec 13, 2023 • edited Loading

Choose a reason for hiding this comment

AkihiroSuda left a comment

Choose a reason for hiding this comment

cyphar commented Dec 13, 2023

cyphar commented Dec 14, 2023

lifubang commented Jul 3, 2024

cyphar commented Jul 4, 2024

cyphar commented Aug 17, 2023 •

edited

Loading

rata commented Aug 28, 2023 •

edited

Loading

rata commented Aug 28, 2023 •

edited

Loading

cyphar Dec 13, 2023 •

edited

Loading