lxd-to-incus may not be handling daemon storage volumes correctly

[kienanstewart]

Required information

• Distribution: Debian
• Distribution version: 12 (Bookworm)
• The output of "incus info" or if that fails:
  • Kernel version:
  • LXC version: 5.0.2-1+deb12u2
  • Incus version: 6.0.2-1~bpo12+1
  • Storage backend in use: dir, lvm, lvmcluster, zfs

Issue description

On our deployed hosts, we keep a relatively small root fs (~80-100G). When launching new incus images, images with a larger footprint will fail to launch with an error such as:

Error: Failed transferring image "962948505414268fda3520e51582bd45df907bb03f003ba1bb1cab6f734ff971" from "172.x.y.z:8443": write /var/lib/incus/images/incus_build_474263561/incus_tar_2678206713: no space left on device

Due to the relatively small root setup, all our cluster members are configured with different backups and images volumes, e.g.

$ incus config get --target X storage.backups_volume
default/storage-backups

$ incus config get --target X storage.images_volume
default/storage-images

The storage.images_volume is used for final storage, but the /var/lib/incus/images directory is still used for transient operations.

E.g.

Image export:

incus/cmd/incusd/images.go

Line 399 in 3c2b356

finalName := internalUtil.VarPath("images", info.Fingerprint)

Image building:

incus/cmd/incusd/images.go

Line 1022 in 3c2b356

builddir, err := os.MkdirTemp(internalUtil.VarPath("images"), "incus_build_")

Image unpacking:

incus/internal/server/storage/utils.go

Line 745 in 3c2b356

tempDir, err := os.MkdirTemp(internalUtil.VarPath("images"), "incus_image_unpack_")

Would it be reasonable to have the images storage directory used when set instead of $INCUS_DIR/images?

In the immediate, my work-around will be to mount a storage pool over /var/lib/incus/images to ensure that there is enough disk space for these operations.

[stgraber]

/var/lib/incus/images becomes a symlink to your configure storage.images_volume when set.
That's how we handle that feature without having to make every single code path within Incus aware of that configuration.

So the fact that you're getting this error suggests that you either have some machines on which the config key wasn't actually set, that your volume is somehow full or that for some odd reason, the symlink wasn't put in place on the system.

[stgraber]

stgraber@dakara:~$ sudo ls -lh /var/lib/incus/
total 56K
lrwxrwxrwx  1 root root          60 Aug 16  2023 backups -> /var/lib/incus/storage-pools/default/custom/default_backups/
drwx--x--x  2 root root        4.0K Nov 26 14:53 containers
drwx------  2 root root        4.0K Oct 13 22:51 containers-snapshots
drwx------  5 root root        4.0K Nov 25 20:43 database
drwx--x--x 76 root root        4.0K Nov 26 14:52 devices
drwx------  2 root root        4.0K Sep  6 20:32 disks
drwxr-xr-x  2 root root          60 Nov 25 20:43 guestapi
lrwxrwxrwx  1 root root          59 Aug 16  2023 images -> /var/lib/incus/storage-pools/default/custom/default_images/
drwx--x--x 11 root root        4.0K Oct 18 14:13 networks
srwx------  1 root root           0 Jan 23  2024 seccomp.socket
drwx------  4 root root        4.0K Aug 16  2023 security
-rw-r--r--  1 root root         757 Aug 19  2021 server.crt
-rw-------  1 root root         288 Aug 19  2021 server.key
drwx--x--x 15 root root         300 Nov 26 14:53 shmounts
drwx--x--x  4 root root        4.0K Sep  6 20:32 storage-pools
srw-rw----  1 root incus-admin    0 Nov 15 16:05 unix.socket
srw-rw----  1 root incus          0 Nov 15 16:05 unix.socket.user
drwx------  2 root root        4.0K Sep  5 19:12 users
drwx--x--x  2 root root        4.0K Nov 26 14:52 virtual-machines
drwx------  2 root root        4.0K Nov 18 02:30 virtual-machines-snapshots

[kienanstewart]

Oh interesting, I didn't know that that was the expected state.

The newly on-lined nodes in the cluster are all fine, but all those legacy nodes (migrated from LXD) don't have the symlinks configured for neither backups nor images.

I think that gives me a clear path on how to regularize the deployment.

Thanks so much for your help!

E.g.

$ while read -r line ; do echo $line ; incus config get --target $line storage.backups_volume ; incus config get --target $line storage.images_volume ; ssh -n root@$line 'ls -l /var/lib/incus' | grep -E 'backups|images' ; echo ; done < <(incus cluster list -f csv | cut -d',' -f1)

host-amd64-1a
default/storage-backups
default/storage-images
drwx------  4 root root        4096 Aug  8 14:01 backups
drwx------  2 root root        4096 Nov 26 14:32 images

host-amd64-1b
default/storage-backups
default/storage-images
drwx------  4 root root        4096 Aug  8 14:02 backups
drwx------  2 root root        4096 Nov 26 15:04 images

...

host-arm64-2a
default/storage-backups
default/storage-images
lrwxrwxrwx 1 root root          67 Aug 14 10:53 backups -> /var/lib/incus/storage-pools/default/custom/default_storage-backups
lrwxrwxrwx 1 root root          66 Aug 14 10:53 images -> /var/lib/incus/storage-pools/default/custom/default_storage-images

host-arm64-2b
default/storage-backups
default/storage-images
lrwxrwxrwx 1 root root          67 Aug 14 10:53 backups -> /var/lib/incus/storage-pools/default/custom/default_storage-backups
lrwxrwxrwx 1 root root          66 Aug 14 10:53 images -> /var/lib/incus/storage-pools/default/custom/default_storage-images

[stgraber]

I'll take a look at lxd-to-incus, we may be missing some logic in there.

[kienanstewart]

It's certainly possible that the problem existed before the migration to incus and I didn't know or didn't notice

[stgraber]

Confirmed that it's an issue in the migration tool.