Automate updating images that have infrequent base image updates

[lbussell]

Let's take CentOS 7 for example: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/main/src/centos/7/Dockerfile

CentOS 7's EOL is June 30, 2024. Its package feeds are still being updated daily, see here. However, our base image for that OS, centos:7 was last updated in 2021 (!).

docker inspect centos:7 | grep Created
        "Created": "2021-09-15T18:20:23.99863383Z"

By our current auto-re-building standards, this image is "up-to-date". However, there's no doubt the packages we don't install as part of the Dockerfile above are way out of date. We should come up with a system that lets us use the up-to-date packages in our buildtools-prereqs containers so that the images are more secure.

This could possibly be taken care of by dotnet/dotnet-docker#1455, but that is a more general solution.

Off the top of my head, this could be accomplished by:

1. Architecting the (at-risk) buildtools-prereqs Dockerfiles to have a multi-stage build, where all packages on the base system are updated in a separate layer and used as a base layer. The base image layer can be "squashed" so that the image doesn't contain any unnecessary filesystem duplication (not an explicit Docker feature any more, but see https://stackoverflow.com/a/77052183/13771378 for how this can be done now).
2. Make some adjustments to the auto-builder so that it re-builds images that haven't been built in a given amount of time (a week, say).

With the above two features in place, if there aren't frequent updates to a base image, we'd get automatic rebuilds with updated packages.

[lbussell]

[Triage] We should make the Dockerfile changes as described above, and then instead of making changes to the auto-rebuilding functionality, just run the centos pipeline on a schedule while CentOS 7 is still in support. Rebuilding too frequently and rebuilding Stream 8 and 9 on a schedule is not a huge concern since these images are used in CI and are usually pulled every time they are run. This also applies to Alma Linux.

[mthalman]

The almalinux:8 tag is still supported. See https://hub.docker.com/_/almalinux. It was last updated just a week ago. So I'm not doing anything special for that.