Document Signing Procedure for Official Images

[NathanMcCauley]

Docker's official images should all be signed using Docker's Content Trust feature. Right now they're all signed, but it'd be good to ensure that remains in effect and a process is followed to make sure that happens for all future builds.

[tianon]

As we were discussing over in moby/moby#20844 (comment), I think it would be extremely useful if we could implement some of what @yosifkit outlined in moby/moby#20844 (comment) to ensure that we're not only creating signed, trustable artifacts, but also that we're using the features of Notary to their fullest in order to do so (eating our own dogfood and ensuring that the system is being tested appropriately, given that my understanding was that signing our images was to be a test of Notary and Content Trust within the engine in general):

It should not be necessary to need a second signing. From all of the documentation I have read, there are two related avenues of approach that would keep the root keys safe while allow the build server to directly sign images.

1. Use the tagging key or create a "build-server" tagging key.

   The Tagging Key is generated for each new repository the publisher owns. They can be exported and shared with any person/system that needs the ability to sign content for this repository.

   • https://blog.docker.com/2015/08/content-trust-docker-1-8/

2. It is possible to have a signing delegation so that none of the "global" docker keys (tagging or offline) are required on the official images build server, but that it can delegate trust to another key. If needed, that trust could be made as a new key for each official image repo (ie, a key for Debian, a key for Python, etc.).

   Signing Delegation: To allow for flexible publishing of trusted collections, a content publisher can delegate part of their collection to another signer. This delegation is represented as signed metadata so that a consumer of the content can verify both the content and the delegation.

   • https://github.com/docker/notary#goals

[tianon]

The timing on today's blog post (https://blog.docker.com/2016/03/notary-0-2/) is fantastic 😄 👍

[endophage]

We've been having a lot of discussion about how to best handle official image signing and there is a strong feeling there should be an opportunity to review a built image. It feels like the right way to handle this is to take @toli's suggestion on the docker/docker PR that you should sign and push to officialstaging. Docker can than apply any manual and/or automated review we feel is useful before promoting the image to the official repos.

I think there are 2 phases of official image creation at play that are being conflated and we should look at separating them. This repo handles review and collation of the input of the build. However there is a separate, increasingly useful step, of validating the output of the build. It is desirable to do this output validation before finally stamping the image as "official".

[wol4max]

@endophage @NathanMcCauley : I spoke to @tianon and he is in agreement to this approach. We will be setting up official-staging and move this forward. We are actually waiting for another issue (repolinks) to get resolved (hopefully today).

[tianon]

To clarify a little, I'm fine with that plan as long as the validation remains as simple as signing; I think it's a bit overblown to make image availability contingent on security scanning results being available, as we have discussed.

[daghack]

As of now, the current process is as follows: official images are pushed to the library organization, and Docker Inc signs them in place. The introduction of windows images necessitated either a change from what was being discussed here, or running the process in duplicate, and we opted for the former.

Currently, the exact process isn't well documented, so we're keeping this issue open to point people to--this is just a small update as to where we are currently at with this.

[abeluck]

This is still rather unclear in 2019.

official images are pushed to the library organization, and Docker Inc signs them in place.

I assume that is supposed to read "pushed by the library organization"?

There are still more questions:

• Docker Inc is running the notary server and signer, but who owns the root/snapshot/delegation keys for the images? For example, do the Debian developers pushing releases to the debian docker repo control the signing keys or does Docker Inc?

• How can one manually verify that an image is correctly signed? Using the notary client's verify command fails:

  # de8b49d4b0b3 is the latest debian:stretch
  $ docker save de8b49d4b0b3 > stretch.tar
  $ notary -s https://notary.docker.io -d ~/.docker/trust verify docker.io/library/debian stretch < stretch.tar
  * fatal: data not present in the trusted collection, sha256 checksum for stretch did not
             match: expected 58a80e0b6aa4d960ee2a5452b0230c406c47ed30a66555ba753c8e1710a434f5
  

• How does one configure their ~/.notary/config.json as described in the docs to connect to the Docker Hub Notary service in order to pin the certs?

• If the Notary server is down or unreachable (e.g., in an air-gapped network), how can we verify images? (Looking for some sort of GnuPG style detached sig)

[tianon]

I assume that is supposed to read "pushed by the library organization"?

No, although I can definitely see the confusion (especially since the library part of the official images isn't exactly a detail you're expected to know or even use). What was meant here is the library organization on Docker Hub, ala https://hub.docker.com/u/library (in other words, when you docker pull debian, what actually gets pulled is library/debian:latest, or more specifically, docker.io/library/debian:latest, so there's a lot of implied bits there).

Docker Inc is running the notary server and signer, but who owns the root/snapshot/delegation keys for the images? For example, do the Debian developers pushing releases to the debian docker repo control the signing keys or does Docker Inc?

All signing of all images under https://hub.docker.com/u/library happens at/by Docker Inc -- neither the image maintainers or the official images maintainers have any access or control over that process (although we're definitely open to being more involved in that / signing closer to the build source if any Docker Inc folks are reading this and want to make that process more interesting ❤️).

For image maintainers to be involved in that signing flow, however, the image maintainers would need to be responsible for building their own images, which for most of our maintainers would be a huge hurdle and would remove a huge portion of the value proposition of the program in the first place (automated rebuilds, multiple architectures, image source review, etc). See https://github.com/docker-library/official-images#architectures-other-than-amd64 and https://github.com/docker-library/faq#how-are-images-built-especially-multiarch for more details around the image build process if you're curious about that.

---

As for questions on Notary (and usage of the notary client), the best I can do is refer you to the Notary project (perhaps @justincormack can provide some pointers as the notary image maintainer 😇❤️).

[abeluck]

Thanks @tianon for the comprehensive reply.

I just discovered that enforcing content trust on a docker daemon is only a Docker EE feature. This means docker content trust is wholly infeasible to use with Docker CE and docker-compose, that is, the vast majority of the open source ecosystem.

So for most of us there is no way to run verified trusted docker images in production.

Very unfortunate.

[konstan]

@abeluck You may want to check this docker engine plugin https://github.com/SixSq/img-authz-plugin. It enforces DCT in DE on image and container creation operations. Internally, it simply uses docker pull with DCT env set.
This implementation is an updated fork of this plugin https://github.com/crosslibs/img-authz-plugin, which was initially for whitelisting registries.

[justincormack]

@abeluck Docker compose is client side, so client side enforcement is fine. The engine side enforcement is for some very specific use cases, and it is not the interface we really want to see long term, and looking to design a new interface.

[abeluck]

@justincormack The issue is, afaict, there is no way to tell docker-compose to use DCT (such as the docker flag --disable-content-trust=false). According to the DCT docs it seems the "right way" is to use engine-side enforcement. Have I missed something?

@konstan Thanks for the link, I'll check it out.

[marekciupak]

Hi 👋

All signing of all images under https://hub.docker.com/u/library happens at/by Docker Inc -- neither the image maintainers or the official images maintainers have any access or control over that process (although we're definitely open to being more involved in that / signing closer to the build source if any Docker Inc folks are reading this and want to make that process more interesting heart).

Thank you for clarifying this. I think it is very important that people have a clear understanding who they trust when they see an image has been signed (by using DOCKER_CONTENT_TRUST=1 or docker trust inspect, etc.).

Where do you think is a good place to document it? Perhaps https://github.com/docker-library/faq would be good? What do you think? Unfortunately, to guarantee credibility, this information should be included in public materials controlled by Docker Inc. However, the mere information that the keys do not belong to the image maintainers, neither the maintainers of Docker Official Images, is very useful.

For example, I always try to pin the certificates when I provision a new environment, and I found quite easily (kudos for Michael) that for pulling Docker images I can do it by $HOME/.docker/trust/tuf/docker.io/library/<NAME>/metadata/root.json file. Unexpectedly, it was difficult for me to find out who the keys used in Docker Library images belong to - it took my a few days to find this thread and Tianon's comment. I was looking for this information because I was hoping that I might be able to confirm that it is the image maintainers (which now I know is not true) who sign these images and it will provide me with a stronger chain of trust. To me it seems a very important piece of information which is currently missing from the README / FAQ.

Best,
Maro