Official Ubuntu 24.04 can not pull certain images

[coesensbert]

https://hub.grid.tf/tf-official-vms/ubuntu-24.04-full.flist
Also tested, same result: https://hub.grid.tf/petep.3bot/ubuntu-24.04_fullvm.flist

root@hubbcdeptst:~/grid_deployment/grid-hub# docker pull caddy:2.8.4
2.8.4: Pulling from library/caddy
c6a83fedfae6: Pull complete 
8f1190437381: Pull complete 
04f2b318172a: Pull complete 
025823b4426f: Extracting [==================================================>]  14.64MB/14.64MB
4f4fb700ef54: Download complete 
failed to register layer: lsetxattr security.capability /usr/bin/caddy: operation not supported

Docker installed with: https://github.com/threefoldtech/grid_deployment/blob/development/apps/prep-env-prereq.sh

[LeeSmet]

What is the storage driver docker is using? This seems like its not using btrfs but rather something which does not support setting extended attributes

[coesensbert]

root@blabla:~# docker info
Client: Docker Engine - Community
 Version:    27.2.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 27.2.1
 Storage Driver: vfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.21
 Operating System: Ubuntu 24.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.928GiB
 Name: blabla
 ID: 1143ad7c-c74a-4eff-9b4d-1346925d4887
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[coesensbert]

root@blabla:~# mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=1004144k,nr_inodes=251036,mode=755,inode64)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
none on /sys/fs/cgroup type cgroup2 (rw,relatime)
tmpfs on /dev/shm type tmpfs (rw,relatime,inode64)
vroot on / type virtiofs (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=33,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13091)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=202144k,nr_inodes=50536,mode=700,inode64)

root@blabla:~# cat /proc/filesystems
nodev   sysfs
nodev   tmpfs
nodev   bdev
nodev   proc
nodev   cgroup
nodev   cgroup2
nodev   cpuset
nodev   devtmpfs
nodev   binfmt_misc
nodev   configfs
nodev   debugfs
nodev   tracefs
nodev   securityfs
nodev   sockfs
nodev   bpf
nodev   pipefs
nodev   ramfs
nodev   hugetlbfs
nodev   devpts
        ext3
        ext2
        ext4
        vfat
        msdos
nodev   autofs
        fuseblk
nodev   fuse
nodev   fusectl
nodev   virtiofs
nodev   overlay
nodev   mqueue
        btrfs
nodev   pstore

[LeeSmet]

This issue is likely caused because the root fs is mounted as a virtiofs, which presumably does not have extended attributes enabled.

[PeterNashaat]

@LeeSmet this is how i create the ubuntu 24 full vm flist
https://github.com/threefoldtech/tf-images/blob/ub24full/tfgrid3/ubuntu24.04/fullvm/ubuntu24-fullvm-flist-builder.sh#L23

[LeeSmet]

After some more looking, the problem is the fact that the root file system is served over virtio-fs, and the virtiofsd-rs process on the host does not have the --xattr flag set. As a result, the virtio fs host process does not announce the availability of extended attributes to the guest, and the virtiofs filesystem driver in the guest, thinking the host does not support them, replies with ENOTSUPP when extended attributes are used in the guest. I'm guessing that the docker vfs driver just tries to use them, since vfs is just a "generic" driver without knowledge of specifics of the underlying fs.

I think it is reasonable that zos adds the --xattr flag on the virtiofsd process, since any mordern filesystem should support that. On top of that, there are quite some options on virtiofsd, some others might also be interesting to take a look at (e.g. xattr remapping from guest to host in case the guest wants to use SELinux).

For the above reason I'll move the issue to the zos repo

[ashraffouda]

tried it on mainnet and devnet both works
and this screenshot from mainnet

[LeeSmet]

The following is from a deployment I made just now on node 129 on devnet:

After using the dashboard to deploy a full VM, which now by default uses the ubuntu 24.04 image in question from the original comment ("flist": "https://hub.grid.tf/tf-official-vms/ubuntu-24.04-full.flist")

[ashraffouda]

which docker version r u using? I installed it like

apt update
apt install docker.io

[LeeSmet]

root@vmt1tp4:~# docker -v
Docker version 27.2.1, build 9e34c9b

[Omarabdul3ziz]

Verified

deployed a full VM from the dashboard with the flist https://hub.grid.tf/tf-official-vms/ubuntu-24.04-full.flist, installed docker from the mentioned script https://github.com/threefoldtech/grid_deployment/blob/development/apps/prep-env-prereq.sh

docker version

root@vmkqjtp:~# docker version
Client: Docker Engine - Community
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:40:59 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:40:59 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.22
  GitCommit:        7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc:
  Version:          1.1.14
  GitCommit:        v1.1.14-0-g2c9f560
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker can pull images

root@vmkqjtp:~# docker pull caddy:2.8.4
2.8.4: Pulling from library/caddy
43c4264eed91: Pull complete
02040ba779ee: Pull complete
c257707c9719: Pull complete
06ce39c94b8d: Pull complete
4f4fb700ef54: Pull complete
Digest: sha256:63d8776389cc6527e4a23bd9750489dc661923cffc3b9d7e0c20e062fa0325ec
Status: Downloaded newer image for caddy:2.8.4
docker.io/library/caddy:2.8.4

also checking the virtiofsd start command on the node 12 on devnet where my machine is deployed.
it started my deployment with --xattr flag

13027 root      3:00 virtiofsd-rs --xattr --socket-path /var/run/virtio-9558-165076-vmkqjtp-0.socket --shared-dir /var/cache/modules/flistd/mountpoint/9558-165076-vmkqjtp --shared-dir-stats /usr/share/btrfs/volstat.sh /var/cache/modules/flistd/mountpoint/9558-165076-vmkqjtp