Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Rootless] Permission denied: unknown on image pull via nerdctl #590

Open
inklesspen1rus opened this issue Apr 21, 2024 · 6 comments
Open

Comments

@inklesspen1rus
Copy link

Issue

Tried to pull ubuntu:20.04 via nerdctl using nydus-snapshotter, but got permission denied: unknown:

$ nerdctl --snapshotter nydus image pull ubuntu:20.04
docker.io/library/ubuntu:20.04:                                                   resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:71b82b8e734f5cd0b3533a16f40ca1271f28d87343972bb4cd6bd6c38f1bd38e:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:39e6324487ef503ef36c38bf0b57935d639398ca0d6081fd20a17f90b956a7a4: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:33985b2ba010a084175876629b280ed9ae49965e9ee5d30b79896cad707bf350:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:43cfb69dbb464ebad014cd4687bf02ee4f5011d540916c658af36faafbfd3481:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 8.2 s                                                                    total:  26.2 M (3.2 MiB/s)                                       
FATA[0008] failed to commit snapshot extract-160833661-2ZUy sha256:106e8431b412f51ccd75ea46a2d5cb4343b23273cbcf50188377cb93aa9a6d82: open /home/inklesspen/.local/share/containerd-nydus/snapshots/3/fs/var/cache/apt/archives/partial: permission denied: unknown 

Expected result

alpine:3 pulls fine:

$ nerdctl --snapshotter nydus image pull alpine:3
docker.io/library/alpine:3:                                                       resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 4.9 s                                                                    total:  3.3 Mi (680.0 KiB/s)

Environment

containerd in rootless via user systemd
config.tar.gz from $HOME/.config

$ inxi
CPU: 6-core AMD Ryzen 5 5625U with Radeon Graphics (-MT MCP-)
speed/min/max: 1091/400/4388 MHz Kernel: 6.7.12-1-MANJARO x86_64 Up: 4d 6h 13m
Mem: 6.64/15.01 GiB (44.2%) Storage: 476.94 GiB (66.7% used) Procs: 422
Shell: Zsh inxi: 3.3.33
$ containerd --version
containerd github.com/containerd/containerd v1.7.13 7c3aca7a610df76212171d200ca3811ff6096eb8.m
$ nerdctl --version
nerdctl version 1.7.2
$ nydusd --version

Version:        v2.3.0-alpha.1
Git Commit:     93ef71db793ae36b12b0e9e6e08d1b4e9566b498
Build Time:     2023-12-06T01:10:03.515180463Z
Profile:        release
Rustc:          rustc 1.68.2 (9eb3afe9e 2023-03-27)
$ containerd-nydus-grpc --version
Version:     v0.13.11
Revision:    7835988d383d591d4f4b1e0e3a1f0c71f6ac8a77
Go version:  go1.19.6
Build time:  2024-03-22T11:10:30
@imeoer
Copy link
Collaborator

imeoer commented Apr 23, 2024

Any error logs are output from nydus-snapshotter for the ubuntu:20.04 image? The problem doesn't seem to be nydus related (ubuntu:20.04 is not a nydus image), have you tried removing --snapshotter nydus ?

@inklesspen1rus
Copy link
Author

Thank you for reply!

Yes, it works fine with --snapshotter overlayfs (Currently I have default snapshotter - stargz):

$ nerdctl --snapshotter=overlayfs pull ubuntu:20.04
docker.io/library/ubuntu:20.04:                                                   resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:71b82b8e734f5cd0b3533a16f40ca1271f28d87343972bb4cd6bd6c38f1bd38e:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:39e6324487ef503ef36c38bf0b57935d639398ca0d6081fd20a17f90b956a7a4: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:33985b2ba010a084175876629b280ed9ae49965e9ee5d30b79896cad707bf350:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:43cfb69dbb464ebad014cd4687bf02ee4f5011d540916c658af36faafbfd3481:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 180.0s                                                                   total:  26.2 M (149.3 KiB/s)

With --snapshotter=nydus

$ nerdctl --snapshotter=nydus pull ubuntu:20.04
docker.io/library/ubuntu:20.04:                                                   resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:71b82b8e734f5cd0b3533a16f40ca1271f28d87343972bb4cd6bd6c38f1bd38e:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:39e6324487ef503ef36c38bf0b57935d639398ca0d6081fd20a17f90b956a7a4: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:33985b2ba010a084175876629b280ed9ae49965e9ee5d30b79896cad707bf350:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:43cfb69dbb464ebad014cd4687bf02ee4f5011d540916c658af36faafbfd3481:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 31.9s                                                                    total:  26.2 M (842.3 KiB/s)                                     
FATA[0032] failed to commit snapshot extract-111208692-sqvL sha256:106e8431b412f51ccd75ea46a2d5cb4343b23273cbcf50188377cb93aa9a6d82: open /home/inklesspen/.local/share/containerd-nydus/snapshots/1/fs/var/cache/apt/archives/partial: permission denied: unknown

Here's nydus logs:
nydus-snapshotter.log

@inklesspen1rus
Copy link
Author

Is it would be simpler if I share qemu virtual machine image with that issue?

So you won't worry about reproducing bug

@imeoer
Copy link
Collaborator

imeoer commented Apr 25, 2024

There are no exceptions in the nydus snapshotter logs, please check if it is related to the access permissions of the directory where /home/inklesspen/.local/share/containerd-nydus/snapshots/1/fs/var/cache/apt/archives/partial is located, e.g., the access perm of the directory /home/inklesspen/.local/share/containerd-nydus are not configured correctly.

@inklesspen1rus
Copy link
Author

Chmodded 777, still doesn't work

$ cd /home/inklesspen/.local/share/containerd-nydus
$ ls -lah
total 80K
drwx------ 1 inklesspen inklesspen   74 апр 24 16:08 .
drwxr-xr-x 1 inklesspen inklesspen 1,4K апр 25 02:04 ..
drwxr-xr-x 1 inklesspen inklesspen    0 апр 21 16:00 cache
drwxr-xr-x 1 inklesspen inklesspen   42 апр 21 16:00 logs
-rw------- 1 inklesspen inklesspen  64K апр 24 16:08 metadata.db
-rw------- 1 inklesspen inklesspen  64K апр 24 16:01 nydus.db
drwx------ 1 inklesspen inklesspen    0 апр 24 16:08 snapshots
$ ls -lah snapshots
total 0
drwx------ 1 inklesspen inklesspen  0 апр 24 16:08 .
drwx------ 1 inklesspen inklesspen 74 апр 24 16:08 ..
$ chmod -R 777 .
$ ls -lah
total 88K
drwxrwxrwx 1 inklesspen inklesspen   74 апр 24 16:08 .
drwxr-xr-x 1 inklesspen inklesspen 1,4K апр 25 10:59 ..
drwxrwxrwx 1 inklesspen inklesspen    0 апр 21 16:00 cache
drwxrwxrwx 1 inklesspen inklesspen   42 апр 21 16:00 logs
-rwxrwxrwx 1 inklesspen inklesspen  64K апр 25 11:00 metadata.db
-rwxrwxrwx 1 inklesspen inklesspen  64K апр 24 16:01 nydus.db
drwxrwxrwx 1 inklesspen inklesspen    2 апр 25 11:00 snapshots
$ nerdctl --snapshotter=nydus pull ubuntu:20.04
docker.io/library/ubuntu:20.04:                                                   resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:71b82b8e734f5cd0b3533a16f40ca1271f28d87343972bb4cd6bd6c38f1bd38e:    exists         |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:39e6324487ef503ef36c38bf0b57935d639398ca0d6081fd20a17f90b956a7a4: exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:33985b2ba010a084175876629b280ed9ae49965e9ee5d30b79896cad707bf350:   exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:43cfb69dbb464ebad014cd4687bf02ee4f5011d540916c658af36faafbfd3481:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 14.1s                                                                    total:  26.2 M (1.9 MiB/s)                                       
FATA[0014] failed to commit snapshot extract-313934468-QzOP sha256:106e8431b412f51ccd75ea46a2d5cb4343b23273cbcf50188377cb93aa9a6d82: open /home/inklesspen/.local/share/containerd-nydus/snapshots/4/fs/var/cache/apt/archives/partial: permission denied: unknown 

@changweige
Copy link
Member

@inklesspen1rus Looks like you are running rootless container. Nydus-snapshotter is not expected to run in such environment yet. For rootless container, nydus-snapshotter has to help containerd do to UIDMAP mount, however, it is not implemented yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants