Cannot read mounts in rootless Podman #8205

apyrgio · 2022-11-16T10:58:06Z

Description

I'm trying to run gVisor within rootless Podman. The aim is to use gVisor for its syscall interception capabilities. However, I get the following error:

$ runsc -debug -network none -rootless do echo ok
Error reading mounts file: error unmarshaling mounts: unexpected end of JSON input
JSON bytes:

creating container: cannot create sandbox: cannot read client sync file: waiting for sandbox to start: EOF

In a regular Docker (not rootless) container, this command works:

$ runsc -debug -network none -rootless do echo ok
ok

Steps to reproduce

Install Podman in your system.
Execute into an Alpine container with podman run -it --rm alpine:edge ash
Install runsc based on the installation instructions
Run runsc -debug -network none -rootless do echo ok

runsc version

runsc version release-20221107.0
spec: 1.0.2-dev

docker version (if using docker)

$ podman --version
podman version 4.3.1

uname

Linux 6.0.8-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 10 Nov 2022 21:14:24 +0000 x86_64 GNU/Linux

kubectl (if using Kubernetes)

No response

repo state (if built from source)

No response

runsc debug logs (if available)

Error reading mounts file: error unmarshaling mounts: unexpected end of JSON input
JSON bytes:

The text was updated successfully, but these errors were encountered:

apyrgio · 2022-11-16T12:10:09Z

I used the -debug-log switch, and I'm now seeing more logs. The following logs from the gofer process provide more info:

I1116 12:05:48.419039       1 main.go:216] ***************************
I1116 12:05:48.419097       1 main.go:217] Args: [runsc-gofer --root=/tmp/runsc-do237025772 --debug=true --debug-log=log/ --overlay=true --network=none --strace=true --rootless=true --debug-l
og-fd=3 gofer --bundle /tmp/runsc-do237025772 --spec-fd=4 --mounts-fd=5 --io-fds=6]
I1116 12:05:48.419124       1 main.go:218] Version release-20221107.0
I1116 12:05:48.419139       1 main.go:219] GOOS: linux
I1116 12:05:48.419153       1 main.go:220] GOARCH: amd64
I1116 12:05:48.419168       1 main.go:221] PID: 1
I1116 12:05:48.419183       1 main.go:222] UID: 0, GID: 0
I1116 12:05:48.419198       1 main.go:223] Configuration:
I1116 12:05:48.419214       1 main.go:224]              RootDir: /tmp/runsc-do237025772
I1116 12:05:48.419231       1 main.go:225]              Platform: ptrace
I1116 12:05:48.419246       1 main.go:226]              FileAccess: exclusive, overlay: true
I1116 12:05:48.419262       1 main.go:227]              Network: none, logging: false
I1116 12:05:48.419279       1 main.go:228]              Strace: true, max size: 1024, syscalls: 
I1116 12:05:48.419293       1 main.go:229]              LISAFS: true
I1116 12:05:48.419308       1 main.go:230]              Debug: true
I1116 12:05:48.419322       1 main.go:231]              Systemd: false
I1116 12:05:48.419336       1 main.go:232] ***************************
I1116 12:05:48.419352       1 main.go:254] Failed to set RLIMIT_MEMLOCK: operation not permitted
W1116 12:05:48.420557       1 specutils.go:113] noNewPrivileges ignored. PR_SET_NO_NEW_PRIVS is assumed to always be set.
W1116 12:05:48.420759       1 util.go:64] FATAL ERROR: error mounting proc: operation not permitted
error mounting proc: operation not permitted

apyrgio · 2022-11-16T12:53:21Z

I see that the relevant check in Gofer happens here:

gvisor/runsc/cmd/gofer.go

Line 418 in d8aa09e

if !conf.TestOnlyAllowRunAsCurrentUserWithoutChroot {

Since we are already running in rootless Podman, we may opt to reuse the existing /proc and / paths. My understanding is that this can happen via the -TESTONLY-unsafe-nonroot flag, and it works indeed:

$ runsc -network none -rootless -TESTONLY-unsafe-nonroot do echo ok
ok

Still, this is a pretty scary looking argument. Is it safe using it in a sandbox within a sandbox use case? Else, is there a better way to achieve the original goal?

The aim is to use gVisor for its syscall interception capabilities

github-actions · 2023-09-13T00:16:49Z

A friendly reminder that this issue had no activity for 120 days.

felschr · 2023-10-01T16:32:16Z

@apyrgio have you found another solution that doesn't rely on -TESTONLY-unsafe-nonroot by now?

apyrgio · 2023-10-16T14:08:19Z

Unfortunately not. I haven't managed to work more in this front actually.

ayushr2 · 2023-10-16T14:24:54Z

I think this issue was fixed by c6a1db5. Can you please confirm?

apyrgio · 2023-11-10T09:41:30Z

Doesn't look like it I'm afraid. The original invocation (runsc -debug -network none -rootless do echo ok) still fails with:

I1110 09:32:21.865885       1 main.go:189] ***************************
I1110 09:32:21.865967       1 main.go:190] Args: [runsc-gofer --root=/tmp/runsc-do4154063816 --debug-log=logs/ --overlay2=all:memory --network=none --rootless=true --debug-log-fd=3 gofer --bundle /tmp/runsc-do4154063816 --gofer-mount-conf
s=1 --spec-fd=4 --mounts-fd=5 --io-fds=6]
I1110 09:32:21.865995       1 main.go:191] Version release-20231106.0
I1110 09:32:21.866008       1 main.go:192] GOOS: linux
I1110 09:32:21.866020       1 main.go:193] GOARCH: amd64
I1110 09:32:21.866032       1 main.go:194] PID: 1
I1110 09:32:21.866045       1 main.go:195] UID: 0, GID: 0
I1110 09:32:21.866058       1 main.go:196] Configuration:
I1110 09:32:21.866070       1 main.go:197]              RootDir: /tmp/runsc-do4154063816
I1110 09:32:21.866082       1 main.go:198]              Platform: systrap
I1110 09:32:21.866095       1 main.go:199]              FileAccess: exclusive
I1110 09:32:21.866110       1 main.go:200]              Directfs: true
I1110 09:32:21.866123       1 main.go:201]              Overlay: all:memory
I1110 09:32:21.866137       1 main.go:202]              Network: none, logging: false
I1110 09:32:21.866152       1 main.go:203]              Strace: false, max size: 1024, syscalls: 
I1110 09:32:21.866165       1 main.go:204]              IOURING: false
I1110 09:32:21.866178       1 main.go:205]              Debug: false
I1110 09:32:21.866190       1 main.go:206]              Systemd: false
I1110 09:32:21.866202       1 main.go:207] ***************************
W1110 09:32:21.867751       1 specutils.go:124] noNewPrivileges ignored. PR_SET_NO_NEW_PRIVS is assumed to always be set.
W1110 09:32:21.868016       1 util.go:64] FATAL ERROR: error mounting proc: operation not permitted
error mounting proc: operation not permitted

(note that the line Failed to set RLIMIT_MEMLOCK: operation not permitted is no longer present, in contrast with the previous logs)

Also, the altered invocation (runsc -network none -rootless -TESTONLY-unsafe-nonroot do echo ok) still works.

terenceli · 2023-12-07T04:38:33Z

I think this issue was fixed by c6a1db5. Can you please confirm?

@ayushr2 This commit code is after the bug occurs, so it can't fix this issue.

avagin · 2023-12-07T20:13:40Z

@apyrgio is selinux enabled? If the answer is yes, could you try to temporary disable it and reproduce the issue?

$ sestatus
$ sudo setenforce Permissive

selinux doesn't allow to do that. Fixes #8205 PiperOrigin-RevId: 588891222

avagin · 2023-12-07T23:43:41Z

@apyrgio @terenceli could you try out #9798?

terenceli · 2023-12-08T01:30:48Z

@apyrgio @terenceli could you try out #9798?

@avagin in my case it doesn't work.

I did some analysis, the issue is happen in here:
https://github.com/google/gvisor/blob/master/runsc/cmd/gofer.go#L394C21-L394C21

Your fix code is after it.

This issue seems to be a general issue 'mount procfs from unprivileged container'. It is discussed in runc issue opencontainers/runc#1658

I used Alban's workaround in here
opencontainers/runc#1658 (comment)

and It works.

Following is my Ubuntu test.

The /mnt/proc volume is in Alban's workaround.

# docker run -it --security-opt apparmor:unconfined --security-opt seccomp=unconfined  -v /home:/home -v /mnt/proc:/newproc -v /tmp:/tmp ubuntu
root@561ca9b97a06:/# useradd test
root@561ca9b97a06:/# su test
$ cd /tmp
$ ./runsc -rootless do sh
*** Warning: sandbox network isn't supported with --rootless, switching to host ***
#

So my question, could we find a more elegant fix/workaround for we can use gVisor in unprivileged docker/podman environment? I rely this to build a process-level sandbox.

avagin · 2023-12-11T01:22:41Z

Your fix code is after it.

You are right. I created a fedora vm to reproduce the issue, but runsc failed differently there.

This issue seems to be a general issue 'mount procfs from unprivileged container'. It is discussed in runc issue opencontainers/runc#1658

Thank you for researching the problem. I think 063ee51 should fix the issue. Could you try it out?

terenceli · 2023-12-11T02:02:33Z

Your fix code is after it.

You are right. I created a fedora vm to reproduce the issue, but runsc failed differently there.

This issue seems to be a general issue 'mount procfs from unprivileged container'. It is discussed in runc issue opencontainers/runc#1658

Thank you for researching the problem. I think 063ee51 should fix the issue. Could you try it out?

@avagin Yes, after apply this patch, it works! Thanks

apyrgio · 2023-12-11T17:43:46Z

Oh, that's great! Thanks a lot for the progress on this front.

@terenceli, just to be sure, did this fix work within a rootless Podman container, as described at the top of the issue? If not, I can give it a go as well.

terenceli · 2023-12-12T00:41:50Z

Oh, that's great! Thanks a lot for the progress on this front.

@terenceli, just to be sure, did this fix work within a rootless Podman container, as described at the top of the issue? If not, I can give it a go as well.

@apyrgio Sorry, I don't try it now. I have tried it in Ubuntu and docker env. You can test it in Podman.

terenceli · 2023-12-20T08:58:17Z

@apyrgio I have tried this patch using podman. With selinux set to Permissive, it works, and with selinux enabled, it doesn't work.

@avagin Could you summit a PR to upstream?

[test@debug010000002015 ~]$ podman --version
podman version 4.8.1
[test@debug010000002015 ~]$ podman run -it --rm alpine ash
/ # cd /tmp
/tmp # ls
runsc
/tmp # ./runsc  do ls
Error setting up network: failed to run "/sbin/ip link add ve-runsc-387325 type veth peer name vp-runsc-387325": exit status 2
/tmp # ./runsc  --rootless do ls
*** Warning: sandbox network isn't supported with --rootless, switching to host ***
runsc               runsc-do3020874715

apyrgio · 2023-12-21T19:30:47Z

Thanks a lot @terenceli. Unfortunately I didn't have time to test it on my own, but the results look promising! For our use case (Dangerzone) we are spinning up containers that don't have a network device, so it could very well be that nested gVisor will work, even with SELinux enabled. That's a great development!

If the existing proc instance has over-mounted areas, it can be impossible to mount a new /proc instance (look at SB_I_USERNS_VISIBLE for more details). Actually, runsc-gofer needs /proc just to open /proc/self/fd and to read a few generic files, so it doesn't need a proc instance of the target pid namespace. Fixes google#8205

If the existing proc instance has over-mounted areas, it can be impossible to mount a new /proc instance (look at SB_I_USERNS_VISIBLE for more details). Actually, runsc-gofer needs /proc just to open /proc/self/fd and to read a few generic files, so it doesn't need a proc instance of the target pid namespace. Fixes #8205 FUTURE_COPYBARA_INTEGRATE_REVIEW=#10028 from avagin:proc-vs-unserns 805fb9b PiperOrigin-RevId: 608729405

apyrgio added the type: bug Something isn't working label Nov 16, 2022

blechschmidt mentioned this issue May 6, 2023

can't change an owner of /dev/stdin: chown /dev/stdin: operation not permitted #8921

Closed

github-actions bot added the stale-issue This issue has not been updated in 120 days. label Sep 13, 2023

ayushr2 assigned avagin Sep 13, 2023

avagin added type: enhancement New feature or request and removed type: bug Something isn't working stale-issue This issue has not been updated in 120 days. labels Sep 13, 2023

apyrgio mentioned this issue Oct 16, 2023

Is it Possible to add gVisor Support to minimize the risk of an container escape exploit? freedomofpress/dangerzone#126

Closed

apyrgio mentioned this issue Oct 16, 2023

Sandbox all document processing in gVisor freedomofpress/dangerzone#590

Merged

copybara-service bot mentioned this issue Dec 7, 2023

gofer: don't create bind-mounts in /proc/self #9798

Closed

copybara-service bot pushed a commit that referenced this issue Dec 7, 2023

gofer: don't create bind-mounts in /proc/self

43d1baa

selinux doesn't allow to do that. Fixes #8205 PiperOrigin-RevId: 588891222

avagin mentioned this issue Feb 20, 2024

runsc:gofer: don't mount a new proc instance #10028

Closed

copybara-service bot mentioned this issue Feb 20, 2024

runsc:gofer: don't mount a new proc instance #10029

Closed

copybara-service bot mentioned this issue Feb 20, 2024

runsc:gofer: don't mount a new proc instance #10031

Closed

copybara-service bot closed this as completed in 88f7bb6 Feb 23, 2024

EtiennePerot mentioned this issue Sep 23, 2024

Operation not permitted when mounting /proc to /tmp/proc #10944

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot read mounts in rootless Podman #8205

Cannot read mounts in rootless Podman #8205

apyrgio commented Nov 16, 2022 •

edited

Loading

apyrgio commented Nov 16, 2022

apyrgio commented Nov 16, 2022 •

edited

Loading

github-actions bot commented Sep 13, 2023

felschr commented Oct 1, 2023

apyrgio commented Oct 16, 2023

ayushr2 commented Oct 16, 2023

apyrgio commented Nov 10, 2023

terenceli commented Dec 7, 2023 •

edited

Loading

avagin commented Dec 7, 2023

avagin commented Dec 7, 2023

terenceli commented Dec 8, 2023 •

edited

Loading

avagin commented Dec 11, 2023

terenceli commented Dec 11, 2023 •

edited

Loading

apyrgio commented Dec 11, 2023

terenceli commented Dec 12, 2023 •

edited

Loading

terenceli commented Dec 20, 2023 •

edited

Loading

apyrgio commented Dec 21, 2023

Cannot read mounts in rootless Podman #8205

Cannot read mounts in rootless Podman #8205

Comments

apyrgio commented Nov 16, 2022 • edited Loading

Description

Steps to reproduce

runsc version

docker version (if using docker)

uname

kubectl (if using Kubernetes)

repo state (if built from source)

runsc debug logs (if available)

apyrgio commented Nov 16, 2022

apyrgio commented Nov 16, 2022 • edited Loading

github-actions bot commented Sep 13, 2023

felschr commented Oct 1, 2023

apyrgio commented Oct 16, 2023

ayushr2 commented Oct 16, 2023

apyrgio commented Nov 10, 2023

terenceli commented Dec 7, 2023 • edited Loading

avagin commented Dec 7, 2023

avagin commented Dec 7, 2023

terenceli commented Dec 8, 2023 • edited Loading

avagin commented Dec 11, 2023

terenceli commented Dec 11, 2023 • edited Loading

apyrgio commented Dec 11, 2023

terenceli commented Dec 12, 2023 • edited Loading

terenceli commented Dec 20, 2023 • edited Loading

apyrgio commented Dec 21, 2023

apyrgio commented Nov 16, 2022 •

edited

Loading

apyrgio commented Nov 16, 2022 •

edited

Loading

terenceli commented Dec 7, 2023 •

edited

Loading

terenceli commented Dec 8, 2023 •

edited

Loading

terenceli commented Dec 11, 2023 •

edited

Loading

terenceli commented Dec 12, 2023 •

edited

Loading

terenceli commented Dec 20, 2023 •

edited

Loading