podman run: add ability to store files written inside a container in host's tmpfs.

[socketpair]

Feature request description

Software I run in a container scatters different files everywhere. Everything works, but slow because these changes are stored in the same FS where layers of the image are (disk). I need podman to make final overlayfs with workdir and upperdir in tmpfs. I don't need any files stored in the container after its death.

[rhatdan]

@giuseppe @vrothberg WDYT?

Using tmpfs for this could be risky since you have limited disk space. Specifying where to store the upper/Container directory might be useful.

[socketpair]

Yes, I know. I have plenty of memory on the host. The software I run writes many files inside the container in random places (actually installs packages) and then run some process reading these files. I can not preinstall these packages before podman run. I need to make everyting to run as fast as possible, and tmpfs is only option. Right now, I have to run --privileged in order to mount overlayfs+tmpfs inside container for root dirs like /usr, /bin, /etc and so on

[giuseppe]

could you try if --transient-store is what you are looking for?

$ podman --transient-store run ....

[socketpair]

@giuseppe
inside container:

# mount
overlay on / type overlay (rw,relatime,
lowerdir=
/var/lib/containers/storage/overlay/l/TVA2BXPFKGVD54KOM7JNLTNIK6:
/var/lib/containers/storage/overlay/l/ABGGV3KRCWNXXQFARGPMDGM22Y:
/var/lib/containers/storage/overlay/l/R72KYFYKHAQMOLTBI5WEEJGOYE,
upperdir=/var/lib/containers/storage/overlay/d5967391751856893e2618aa54f7223ec557b91a3029796bc99f8f38e543a189/diff,
 workdir=/var/lib/containers/storage/overlay/d5967391751856893e2618aa54f7223ec557b91a3029796bc99f8f38e543a189/work,
metacopy=on,volatile)

I added newlines for readability.
As you can see, upperdir and workdir on HDD. Note, podman is run under root user.

[socketpair]

podman --graphroot might help, but seems this command line disappeared.

$ podman --version
podman version 4.4.1

[giuseppe]

thanks for verifying that. Another option could be to pull the images in your store on /var/lib then use a different store to create these containers. You need to configure the store in /var/lib as an additional store for that work.

# podman pull fedora
# podman --root /tmp/root --storage-opt AdditionalImageStore=/var/lib/containers/storage run --rm fedora grep "overlay overlay" /proc/self/mountinfo
2636 2598 0:98 / / rw,relatime - overlay overlay rw,context="system_u:object_r:container_file_t:s0:c433,c999",lowerdir=/var/lib/containers/storage/overlay/l/R4BMMC7ZTAKM7US6IT53BTBBCY,upperdir=/tmp/root/overlay/97a79d68c590dbeab36cf9988e661fc86d82fea36a3f6dd7beaa4ba423532105/diff,workdir=/tmp/root/overlay/97a79d68c590dbeab36cf9988e661fc86d82fea36a3f6dd7beaa4ba423532105/work,volatile

[rhatdan]

I like that idea, Perhaps a small blog on it.

[socketpair]

Unbelievable! It works!

So, I suggest some standard option that do this. Because it's really non-trivial. For example, I still don't understand why it stops working if I remove AdditionalImageStore=.

It will be very useful for podman to mount tmpfs for this case (in separate mount namespace?) + some tmpfs options like size= and others.

[giuseppe]

if you drop the AdditionalImageStore= option it won't try to use the existing store to read images.

I wouldn't make it too easy for users to end up in this setup, we can document or better as @rhatdan suggested have a blog but users must be sure of what they do. Using tmpfs for the upper layer is an unusual configuration, you can easily run out of memory and the entire container storage is lost on a power cycle.

One thing I'd like to mention is that if you specify --rm to your container, then overlay is mounted with volatile=on as it seems to be in your case, and it already speeds up the I/O in some cases, since any sync/fsync syscall is inhibited. If you play around with the writeback timeouts, you can then force Linux to keep dirty pages longer in memory, getting similar performance as using tmpfs as the backing store.

[socketpair]

Hmm. What approach/method is used to inhibit fsyncs ?!

[giuseppe]

the volatile=on mount option to the overlay file system

[socketpair]

@giuseppe

1. do you know how to achieve the same (or similar) on containerd?
2. why not to add new command line key/config parameter to podman - where to create runtime layer ? By default – in main storage.

[giuseppe]

1. no idea
2. we are investigating that as part of containers/storage. Although it is not tracked upstream. One risk compared to the solution I've proposed is that the layers will still be referenced from the main storage but they could have been dropped from tmpfs causing all sort of corruption in the storage.

Since there is a working solution for the problem, I am closing the issue.