using buildpack-deps to build images - bigger images, increased attack surface

[lethargosapatheia]

I'm just wondering, maybe I'm missing something, but is using buildpack-deps actually a good idea?
I've done a short test to compare the image sizes of the official golang versus installing golang starting from plan debian and the difference is quite significant (756MB vs 460MB).
What also surprised me is that python is also installed - which is actually a dependency of mercurial. Is this actually necessary when running go? Is there a reasoning behind this?

Thanks!

[wglambert]

#305 (comment)

I think the general consensus for usage of this repo is that it typically wouldn't be your runtime base and you'd use it more for building your project and then copying the resulting binary into a separate image that's FROM debian:xxx-slim, for example. (Using Docker's multi-stage builds make this especially easy to accomplish.)

Python is from buildpack-deps. Having a consistent base image across all the language-stack images streamlines the build process for updates and changes

[lethargosapatheia]

Yes, I've already thought of the multi-staging setup with go where you can simply copy your binary. And that solves most of the issues, indeed.

I wonder though where this general consensus comes from. I bet people use these images directly in production as such and don't worry too much about it, because they know they're official and they think it's the best they get directly for production purposes. I'm sure I'm not the only one who assumes this. I and many others have used postgresql, mysql, python and other images directly as such. And I might not have enough experience, but many others that I know do, and they're still using it. So I'd assume someone isn't really getting their message across :)

I'm guessing postgresql, for instance, doesn't fall into the same category as golang and other programming languages in this context?

[wglambert]

Yeah the other repo's don't fall into the same category as this particular one. Obviously there's a tremendous number of great usecases for using the Official Images in production and testing, but for this repo we see that the general consensus with is this typically wouldn't be used as a runtime base. That's why it's based on buildpack-deps and there's no slim variant.

All the images you listed have slim variants and valid usecases for deployment in production. Such as php which comes with an apache variant for use in production, with an example in its docs https://github.com/docker-library/docs/tree/master/php#configuration

[lethargosapatheia]

Thanks for pointing me to the docs. This is probably getting out of scope, but regarding nodejs, for instance, I don't think people should use the defacto image with python preinstalled as described in the latter case (production image based on it, so not as a multi-staged setup), as the author suggests here. I think it kind of borders on bloatware in my humble opinion.

This is the defacto image. If you are unsure about what your needs are, you probably want to use this one. It is designed to be used both as a throw away container (mount your source code and start the container to start your app), as well as the base to build other images off of.

and:

Unless you are working in an environment where only the node image will be deployed and you have space constraints, we highly recommend using the default image of this repository.

But I see how this can get ambiguous and interpretable. I guess I simply expected something different here.
As far as golang is concerned, I'm now pretty much convinced that it's ok, given that you basically never have to use go as such in a container.

Thanks for your replies.

[Later edit:] Only afterwards did I see that the same text (first paragraph) is used also for the golang image.

[tianon]

In this case specifically, we're only FROM buildpack-deps:XXX-scm (not the full "batteries included" variant) which is just curl+wget+ca-certificates+gpg+common SCM tools (like git), so if python3 is in the image, it's coming as a dependency of one of those packages 😇

https://github.com/docker-library/buildpack-deps/blob/b135e963f9f664e70b6ec8491fd33578398841b6/debian/bullseye/curl/Dockerfile

plus

https://github.com/docker-library/buildpack-deps/blob/b135e963f9f664e70b6ec8491fd33578398841b6/debian/bullseye/scm/Dockerfile

[tianon]

If you want a slimmer alternative (that's still Debian-based or even Ubuntu-based), something like the following should work pretty reasonably:

FROM debian:bullseye-slim
ENV PATH /usr/local/go/bin:$PATH
COPY --from=golang:1.18-bullseye /usr/local/go /usr/local/go

[lethargosapatheia]

Yeah, I got all that, I mentioned in the first post that python is a dependency of mercurial.
At this point I was basically talking about what people expect to get from these images and I was questioning this general consensus which seems to be confined to a group of people who are in the know, which seems to be contradicting to some extent the documentation.
It's not normal to have python in a container image along with whatever you're actually trying to run in production directly, if that isn't already perfectly clear. But, as I've already said, this applies now more to other images than golang (because of the multi-staging setup). So maybe this topic makes more sense within the docs project, but I'm rather sceptical that it would make any difference.

[yosifkit]

Yeah, I think this is more a documentation adjustment than a change to the image. Users should be using the golang image primarily to compile and should not be using the image directly in production without understanding the implications of that.

The Alpine-based images could maybe be used for production since they include less tools in the final images, but even then the go compiler itself could be considered a potential attack tool (or sh or apk). But also note that Alpine/musl isn't really supported by go upstream development:

This variant is highly experimental, and not officially supported by the Go project (see golang/go#19938 for details).

- https://github.com/docker-library/docs/tree/898112921cc7c4d410c63d10a3d24752f1dc420f/golang#golangversion-alpine

[fgimian]

I think it would be great to have a slim variant of the official Go image; because while it's not required at runtime, it'll still be something that we use as part of our CI setup and based on other images I've checked out, the size will reduce very significantly which may be ideal.

For teams such as ours who, are running their own CI agents, the Go image will be something we need to pull down and store on our CI servers and may also be used as a base for other build images that include further tools and such.

Here are some examples comparing size differences between normal and slim variants for some other languages:

REPOSITORY   TAG                             IMAGE ID       CREATED       SIZE
elixir       1.14.1                          b34a627ea5b9   4 hours ago   1.47GB
elixir       1.14.1-slim                     70742f7ac35f   4 hours ago   305MB
golang       1.19.2-bullseye                 8827cedaa309   2 weeks ago   992MB
python       3.10.8-bullseye                 f05c8762fe15   11 days ago   921MB
python       3.10.8-slim-bullseye            3dea7661ddf9   2 hours ago   126MB
perl         5.36.0-bullseye                 4357c0fae13b   2 weeks ago   894MB
perl         5.36.0-slim-bullseye            790940080cda   2 weeks ago   172MB
perl         5.36.0-slim-threaded-bullseye   ed5ad6a85bd7   2 weeks ago   173MB
perl         5.36.0-threaded-bullseye        2bde646ec13a   2 weeks ago   895MB

The Alpine variant is not optimal for projects that prefer Debian (which is more consistently the base OS choice for all official Docker images).

Huge thanks
Fotis

[mrwormhole]

popping up here, this doesn't look good from buildpacks, it is introducing tons of vulnerabilities from debian/linux 5.10.162-1 to official golang:latest image

https://hub.docker.com/layers/library/golang/latest/images/sha256-bd4a3e7eee6d6ea30b2e27d6c1ac3c56809e78e08c7e44ddf91f8c741091f5ad?context=explore

I couldn't figure out how FROM buildpack-deps:bullseye-scm, scm, stable-scm is introducing old linux kernel to images, because buildpacks is not using old kernel on those

but docker UI points out to this command;

/bin/sh -c set -eux; apt-get update; apt-get install -y --no-install-recommends g++ gcc libc6-dev make pkg-config ; rm -rf /var/lib/apt/lists/*

Note: nvm, it is coming from debian image and from buildpacks, yikes

same issue on https://hub.docker.com/layers/library/golang/1.19.6/images/sha256-699d222b61fb23503d3d5f27599dd0dc2f79a037a5e09f74c285045ef5ce2812?context=explore

might be worth convincing buildspack people to use debian 11.6 (kernel 6) instead of debian 11 (kernel 5)

[yosifkit]

Background:

Tags in the [official-images] library file[s] are only built through an update to that library file or as a result of its base image being updated (ie, an image FROM debian:buster would be rebuilt when debian:buster is built).

-https://github.com/docker-library/official-images/tree/2f086314307c04e1de77f0a515f20671e60d40bb#library-definition-files

Official Images FAQ:

Though not every CVE is removed from the images, we take CVEs seriously and try to ensure that images contain the most up-to-date packages available within a reasonable time frame

- https://github.com/docker-library/faq/tree/0ad5fd60288109c875a54a37f6581b2deaa836db#why-does-my-security-scanner-show-that-an-image-has-cves

Since our build system makes heavy use of Docker build cache, just rebuilding the all of the Dockerfiles won't cause any change. So we rely on periodic base image updates.

We strive to publish updated images at least monthly for Debian. We also rebuild earlier if there is a critical security need. Many Official Images are maintained by the community or their respective upstream projects, like Ubuntu, Alpine, and Oracle Linux, and are subject to their own maintenance schedule.

- from the same FAQ link

So, buildpack-deps images are built using the latest build of each respective Linux distribution and are rebuilt often (approximately every 30 days). The current buildpack-deps Debian images were built/pushed just 7 days ago (e.g. :bullseye-scm). Any of the listed CVE's on Docker Hub from the kernel are incorrect (the package isn't installed in the image); we've reported it the team.

There is one package update available and it will be automatically applied when the images are rebuilt with the next Debian base image update (likely about 2-3 weeks from now):

$ docker run -it --rm --pull=always buildpack-deps:bullseye-scm
bullseye-scm: Pulling from library/buildpack-deps
Digest: sha256:ca9b375648e50b4c26007c99cb7dca68626fc42863c01841d7232f41e166130e
Status: Image is up to date for buildpack-deps:bullseye-scm
root@2ebb96764ffc:/# apt update
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:4 http://deb.debian.org/debian bullseye/main amd64 Packages [8183 kB]
Get:5 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [234 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [14.6 kB]
Fetched 8640 kB in 1s (6932 kB/s)                         
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.
root@2ebb96764ffc:/# apt list --upgradable
Listing... Done
libapr1/stable-security 1.7.0-6+deb11u2 amd64 [upgradable from: 1.7.0-6+deb11u1]
N: There is 1 additional version. Please use the '-a' switch to see it
root@2ebb96764ffc:/# 

No kernel is installed in the image:

root@2ebb96764ffc:/# apt-cache pkgnames | grep linux-image
root@2ebb96764ffc:/# apt install linux-image-amd64
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  apparmor busybox cpio firmware-linux-free initramfs-tools
  initramfs-tools-core klibc-utils kmod libklibc libkmod2 linux-base
  linux-image-5.10.0-21-amd64 pigz udev
Suggested packages:
  apparmor-profiles-extra apparmor-utils libarchive1 bash-completion
  linux-doc-5.10 debian-kernel-handbook grub-pc | grub-efi-amd64 | extlinux
The following NEW packages will be installed:
  apparmor busybox cpio firmware-linux-free initramfs-tools
  initramfs-tools-core klibc-utils kmod libklibc libkmod2 linux-base
  linux-image-5.10.0-21-amd64 linux-image-amd64 pigz udev
0 upgraded, 15 newly installed, 0 to remove and 1 not upgraded.
Need to get 58.9 MB of archives.
After this operation, 333 MB of additional disk space will be used.
Do you want to continue? [Y/n] ^C

[tianon]

This has been fixed in Docker Scout's results. 👍

[tianon]

I think the discussion here is probably as sufficiently "resolved" as this issue is going to be. To summarize, use of this image as a base is a case of the classic "security vs usability teeter-totter" and leaning more towards the right-hand side of that equation (which is also why most of our images also include a slimmer variant to avoid it, such as python:X.Y-slim). Go is a notable exception because it is not intended to be an actual runtime environment.