RUN true in Containerfile touches mtime of /run, creates layer

[allisonkarlitskaya]

I found containers/podman#14577 and containers/podman#14582 related issues when searching for duplicates.

tl;dr: Running a read-only command in a RUN unexpectedly modifies the container image and creates a new layer. Even if you add a workaround to prevent the modification, the layer will be created but it will be empty (the layer ID is equal to the SHA256 of an empty .tar file). Each RUN creates an additional layer.

Consider this Containerfile:

FROM fedora

RUN sleep 1
RUN stat /run
RUN sleep 1
RUN stat /run

You can see that each stat command reports a different mtime for /run. I think that's because when we go to mount /run/.containerenv (for the duration of each RUN command) we need to create/delete the temporary file for the bindmount point in /run, which results in the mtime changing.

That also means that these presumably do-nothing commands are creating layers, as can be seen in the output of podman image inspect on the created container: you get 5 distinct layers added. If you look at the first layer .tar (after the fedora base image) you'll see that it modifies /etc/hostname and /etc/resolv.conf along with /run but all of the other layers contain only /run with an updated timestamp, which I consider unexpected and undesirable.

In my particular usecase I'm running RUN rm, so I'm actually trying to modify the container image, but am getting /run changed as a side-effect, which is undesired.

As a workaround for my usecase, something like this does the trick:

FROM fedora

RUN --mount=type=tmpfs,target=/run sleep 1
RUN --mount=type=tmpfs,target=/run stat /run
RUN --mount=type=tmpfs,target=/run sleep 1
RUN --mount=type=tmpfs,target=/run stat /run

and then /run won't get modified.

This does do something else strange, though: each of those RUN commands will create a new layer, but all layers will have the same ID: 5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef. That's the SHA-256 of 1024 nul characters. That's the tar end-of-stream marker, so effectively this is creating 3 empty layers. That should probably also be fixed.

[allisonkarlitskaya]

See #4242 about the mentioned files in /etc. That's definitely a different issue, but it's good to know that it's being tracked as well.

[allisonkarlitskaya]

I forgot to mention:

lis@x1:~$ podman --version
podman version 5.3.2
lis@x1:~$ buildah --version
buildah version 1.38.1 (image-spec 1.1.0, runtime-spec 1.2.0)
lis@x1:~$ cat ~/.config/containers/storage.conf 
[storage]
driver="btrfs"