reflink by default (if possible) when copying between containers-storage: instances

[cgwalters]

Editing this older issue to make clear it's about a more general problem. Today when doing a skopeo copy or a podman push between two different containers-storage: instances, we end up with full physical copies of all of the files, even if the two storages are on the same physical filesystem.

The copy code is generic and operates in terms of tarballs.

(Note this issue also relates to #197 and it may make sense really to fix both of them at once)

Now this was originally an issue about copying between a root-owned containers-storage and a rootful which is I think something we also want.

In the truly general case, we likely want to handle having different configurations for each containers-storage: instance (e.g. one might be set up with zstd:chunked, another not).

On the implementation side given all of these things it may make sense to handle this via two communicating processes. Basically we'd have an RPC API that's a bit of a variant of something like https://github.com/containers/skopeo/blob/main/docs-experimental/skopeo-experimental-image-proxy.1.md but that allows accessing the underlying objects directly va fd passing too (cc #98 ).

Original issue follows

---

(Sorry if this is a duplicate, I tried to search here and didn't find it...tried to search in podman issues but there are soooo many issues and these are generic search terms. Maybe someday github will have an LLM integrated for search instead)

An rpm-ostree user wanted to copy an image built in their rootless podman storage to the host root storage, doing

$ rpm-ostree rebase ostree-unverified-image:containers-storage:[/home/swick/.local/share/containers/storage]localhost/my-silverblue-toolbox-image:latest

And reported

one more thing: it creates the directory ~/.local/share/containers/storage/overlay/l/ owned by root, making podman images fail to chown it.

I think in general the c/image stack would need to learn to do one of a few options here:

• Fork into two processes, where the process operating on the unprivileged store calls setuid() to the owner of that directory
• setfsuid - but this one being process global just heavily conflicts with goroutine threads and seems like a nonstarter
• Just avoid any writes to the target directory at all (no locking) and accepting the possibility of race conditions

[cgwalters]

Fork into two processes,

Tangentially related to this, I think a mechanism that would make sense for copying here would be quite similar conceptually to what landed in the skopeo experimental-image-proxy; basically, an in-band primary channel for metadata, but instead of passing tarballs over a pipe, we pass readonly file descriptors for filesystem objects. In the case of a privileged caller copying from an unprivileged store, this would allow us to use reflinks (if available) which would be a giant speed and disk space improvement.

[cgwalters]

.oO(thinking more) - maybe we should think of this like a "composefs clone" operation really, with support for extra metadata (tar-split stuff).

[cgwalters]

cc ostreedev/ostree-rs-ext#642 which definitely wants a better interface to c/storage than passing tarballs around.

And really the more I think about things like this, the more strongly I feel that composefs is the right core to things. For example, when copying a container image from rootful podman to rootless (or vice versa), because fsverity is implemented in the kernel (a common shared trust domain) and enforces read-only state for content we can efficiently provide a file descriptor opened from a user's home directory and reflink (or copy if we have to) into the rootful container storage, while knowing that:

• There's no possibility that the user could concurrently mutate the file contents
• We don't need to inefficiently recalculate the checksum for files