Incus volume storage copy with refresh copies all snapshots when using ZFS

[ciph3rm0nk]

Required information

• Distribution: Ubuntu
• Distribution version: 24.04.1 LTS
• The output of "incus info" or if that fails:
  • Kernel version: 6.8.0-48-generic
  • LXC version: 6.7
  • Incus version: 6.6
  • Storage backend in use: ZFS
  • ZFS Version: zfs-2.2.2-0ubuntu9.1

I also had the same behavior with a Ubuntu 22.04.X (I don't remember what it was, but it should have been latest) when issuing the command to a 24.04.1 destination.

Issue description

When using storage / volumes on ZFS, and using copy to another host with refresh, it seams that incus is copying all previous snapshots as well.

Steps to reproduce

incus storage volume snapshot create example-storage-pool example-volume snap-manual-2024-08-11
incus storage volume snapshot create example-storage-pool example-volume snap-manual-2024-09-11
incus storage volume snapshot create example-storage-pool example-volume snap-manual-2024-10-11

So we have three snapshots separated by a month. With data written between each snapshot. So we copy this to our remote server.

incus storage volume copy example-storage-pool/example-volume remote-incus:example-storage-pool/example-volume --refresh

Its the first copy so lots of data is copied over. All looks well.

However, next month, we issue our snapshot again.

incus storage volume snapshot create example-storage-pool example-volume snap-manual-2024-11-11

incus storage volume copy example-storage-pool/example-volume remote-incus:example-storage-pool/example-volume --refresh

However all previous snapshots are again copied over. Issuing something like ps aux|grep zfs shows that its running zfs send on the old snapshots that were already copied over. My understanding was that this wasn’t supposed to be the behavior and that incus was supposed to only zfs send only sends missing snapshots. Only happens with storage volumes, not containers themselves.

Information to attach

I'm not sure if anything else is relevant, let me know. This was part of my migrations off of LXD and hoping to reduce downtime. I still have the interim server available, or i can setup another lab to produce logs, but this was the second time this happened so I am reasonably sure I could reproduce the problem.