Incorporate apt upgrades for security updates

[rarkins]

Ubuntu base images are only updated periodically, and may have vulnerabilities in packages which are fixed in the apt repository.

How can we balance such upgrading while also retaining some caching ability?

Example:

ubuntu:20.04 (ubuntu 20.04)
===========================
Total: 2 (UNKNOWN: 0, LOW: 1, MEDIUM: 1, HIGH: 0, CRITICAL: 0)

+-------------+------------------+----------+-------------------+------------------+
|   LIBRARY   | VULNERABILITY ID | SEVERITY | INSTALLED VERSION |  FIXED VERSION   |
+-------------+------------------+----------+-------------------+------------------+
| libgcrypt20 | CVE-2021-40528   | MEDIUM   | 1.8.5-5ubuntu1    | 1.8.5-5ubuntu1.1 |
+             +------------------+----------+                   +                  +
|             | CVE-2021-33560   | LOW      |                   |                  |
+-------------+------------------+----------+-------------------+------------------+

Meanwhile the same image with RUN apt-get update && apt-get -y upgrade && rm -rf /var/lib/apt/lists/* results in no vulnerabilities.

Ref: https://pythonspeed.com/articles/security-updates-in-docker/

[viceice]

😕 Maybe we should create a ubuntu base image, which we rebuild without cache weekly, so we get weekly securitxy updates. Ubuntu base image is updates once a month or so. 🤔

[wwuck]

@viceice would this rebuild without cache include running apt-get -y upgrade?

Is there any downside to running apt-get upgrade when building the renovate image? Docker, OWASP, and hadolint all no longer recommend not upgrading packages in Dockerfiles.

docker/docs#12571
OWASP/CheatSheetSeries#614
hadolint/hadolint#562

Until renovate gains support for updating from APT repositories (this might not ever be feasible), then we are stuck with running apt-get upgrade in our images. I don't see any easy way to have apt package version pinning in Dockerfiles without a lot of manual effort in tracking updates. Debian/Ubuntu only keep the latest version in their repositories anyway, so any existing version pins would be quickly broken after a new package version is released.

[viceice]

even i add a apt upgrade line to the docker file, it would be cached until the parent digest is changed. So it's required to rebuild without cache to force a apt upgrade run.

[viceice]

we currently don't do any upgrade to suppress too may dupicated files, as docker would save all those new files the a new layer. so worst case it you fully duplicate the base layer

[rarkins]

BTW this affects both the base image as well as tools we install such as curl

[viceice]

yes, we install latest version which is available at the day, when renovate updates the base image or we change some of our base files.

[rarkins]

Maybe we could inject a "week number" into our build files so that it invalidates the cached layers weekly? But we might also want the ability to manually trigger it somehow

[viceice]

We need to add that value as fix commit to the repo so semantic release will do a release and we can use that file as cache buster.

[wwuck]

For our images, we are using multi-stage builds with a single RUN command in the intermediate stage to avoid docker caching the apt-get commands.

[rarkins]

Caching until now has been a desirable feature, not a bug..

[viceice]

what about weekly lockfile maintenance as fix release? 😏

[wwuck]

There is another CVE (ReDoS in chalk/ansi-regex) detected in the renovate slim image, https://www.cve.org/CVERecord?id=CVE-2021-3807.

I don't think it's likely to be a real problem for renovate though as renovate images not usually used in a public-facing environment.

[chrisfarnham]

@wwuck Could you provide a small example of the multi-stage build with the single RUN that avoides docker caching?

[wwuck]

https://github.com/hadolint/hadolint/wiki/DL3009
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#apt-get

Deleting rm -rf /var/lib/apt/lists/* in the same RUN statement as apt-get install means the package index will always be refreshed on install. I would be also pinning apt package versions, as recommended by hadolint, if it were possible to automate with renovate. Unfortunately that is not possible from what I've read in some issues here.

I thought I remember reading somewhere that intermediate stages in a multi-stage build are not cached (I could be wrong here though)? In any case, I use multi-stage builds to prevent temporary build credentials from appearing in the final image layers.

[wwuck]

Hmmm, after some more reading, it looks like the only way to really avoid the cache (if you're not version pinning with ARG) is to either include ADD/COPY statements, or just run the build with docker build --no-cache.

https://docs.docker.com/engine/reference/builder/#run
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#leverage-build-cache

[reitzig]

FWIW, maintaining pinned apt package versions with Renovate is very possible now.

Posting this here since I see Renovate mentioned above (oh hi, @rarkins! 😬 ) and we're seeing dozens of mid-level vulnerabilities in renovate/renovate, which is based on this here image (cf. renovatebot/docker-renovate:Dockerfile).

I can contribute a PR, if you would accept that solution.

Never mind, there are hardly any explicit package installs in Dockerfile here.

Then again, it's containerbase/base:src/usr/local/bin/install-containerbase that would have to be annotated, right? 🤔

[viceice]

that would not help, as most vulnerabilities are not solved, so no upgrade at all.