Support manifest lists / manifest-tool

[tianon]

Ideally we'd like to start experimenting with manifest lists (likely using Phil's tool; https://github.com/estesp/manifest-tool), initially for allowing for latest and other similar tags to redirect to either Linux or Windows based images automatically as appropriate, but eventually for multi-architecture support as well.

I'm filing this issue so we can discuss obstacles and work on overcoming them. 😄

• library file changes
  • Add initial support for an explicit "ManifestListTags" field go-dockerlibrary#2
• bashbrew changes (new subcommand for assembling and pushing manifest list objects)
  • Add an initial "manifest-tool" implementation #3058
• signing system (Document Signing Procedure for Official Images #1516) support for manifest lists
  • scanning system support for manifest lists (decision was made to separate this to be a separate non-blocking task)
• unexpected manifest-tool bug pushing to library/ (Build manifest-tool from source to apply a minor patch #3437)
• update Jenkins to build/push amd64 (https://github.com/docker-library/oi-janky-groovy/blob/7516f1609f1260f94e70192238c588bfb4f8a88b/multiarch/vars.groovy#L4)
• update Jenkins to push manifest lists (https://github.com/docker-library/oi-janky-groovy/tree/06a6a7de82617c024fdd36f958316857b48e268d/put-shared), taking care with "heavy hitters"

[friism]

@tianon thanks for getting this started.

Note that tags are not actually redirecting - the manifest list is served to the engine on pull, and then the engine chooses what to get.

I think it's probably fine to still have linux, debian, winddows and so on tags, as long as latest and other tags that don't specifically include OS are manifest lists and do the right thing. Once there's good UX for this, and if we get a way in the engine to overide what manifest list entry to pull, we should consolidate.

[tianon]

Right, I don't mean that they redirect per-se, but we do have to push something somewhere first in order to include it in a manifest list (and can't push the full list until all the relevant artifacts it needs to reference are pushed), so pushing becomes a bit complex. Sorry for not being more clear on that point. 😇

[StefanScherer]

I haven't used the manifest-tool yet. But without knowing all the details I would like to have an easy "the best tools go out of your way" style approach for multi os/architecture:

1. Travis/Circle/Autobuild do a Linux build with
   • docker build -t foo/bar:version and
   • docker push foo/bar:version
2. AppVeyor does a Windows build with the same commands
   • docker build -t foo/bar:version and
   • docker push foo/bar:version
3. I build on a Linux ARM machine and do the same, and so on...

And I'm done.

My build agents does not depend on each other or have to be called in a specific order.
All the magic happens while pushing into Docker Hub.

[tianon]

@StefanScherer yeah, I guess by delaying, I was kinda hoping Docker would grow support in docker push itself to append to an existing manifest list, because that'd definitely simplify this problem, but as-is the tooling isn't quite there yet, so each of those pushes will overwrite the existing manifest and you'll be left with only the final push's contents available 😇 😞

[tianon]

Here's an initial teaser of integrating docker-library/go-dockerlibrary#2, at least for generating the doc block I outlined in docker-library/go-dockerlibrary#2 (comment):

$ bashbrew cat hello-manifest
Maintainers: Tianon Gravi <admwiggin@gmail.com> (@tianon), Joseph Ferguson <yosifkit@gmail.com> (@yosifkit)
GitRepo: https://github.com/docker-library/hello-world.git

Tags: 7.6-linux, 7-linux, linux
SharedTags: 7.6, 7, latest
GitCommit: bdee60d7ff6b98037657dc34a10e9ca4ffd6785f
Directory: hello-world

Tags: 7.7-nanoserver, 7-nanoserver, nanoserver
SharedTags: 7.7, 7, latest
GitCommit: 1f13a5bc3b787747eeefb3b0051d8d29f851260d
Directory: hello-world/nanoserver
Constraints: nanoserver

$ bashbrew cat --format '{{ range .Manifest.GetSharedTagGroups }}- `{{ .SharedTags | join "`, `" }}`:{{ "\n" }}{{ range .Entries }}  - `{{ .Tags | first }}`{{ "\n" }}{{ end }}{{ end }}' hello-manifest
- `7.6`:
  - `7.6-linux`
- `7`, `latest`:
  - `7.6-linux`
  - `7.7-nanoserver`
- `7.7`:
  - `7.7-nanoserver`

[friism]

@tianon have you made more progress on this? Asking for friend. (@dmp42 actually)

[tianon]

This has been a little bit side-tracked by officializing the multi-architecture work (which is definitely highly related): docker-library/go-dockerlibrary#10 😇

[tianon]

Filed #3009 to start bringing some bits of this together -- there's a stub in there for where we plan to implement manifest lists, but for now we wanted to get at least this much in so we can start moving from https://github.com/docker-library/oi-janky-groovy/blob/2c031cae0bd68ed7679e952c7aeffeb7cb081044/multiarch/vars.groovy as the source-of-truth for our current multiarchitecture support to the image manifest files themselves being the source-of-truth, as they should be (and as they are for the rest of what we do). 👍

[tianon]

See https://hub.docker.com/u/trollin/ for where I've temporarily been pushing manifest-list-ified versions of the official images. 👍

[vielmetti]

@tianon - is there a straightforward way to inspect the Docker hub to see what architectures these sample images point to?

[tianon]

@vielmetti the best I've seen so far is manifest-tool inspect <image>; I think Docker Cloud might show arches on hover, but it's hard to explore on Docker Cloud, and the hover text isn't very granular

[StefanScherer]

@vielmetti another way is to use curl + jq if you just want to see the os/arch pairs: curl -sL https://registry.hub.docker.com/v2/repositories/trollin/golang/tags/latest | jq .

[StefanScherer]

@tianon what about extending trollin/golang (and others where applicable) for windows/amd64 with nanoserver as the default? 😄

[tianon]

Updated the OP checklist with a bit of news! The signing system is now updated to handle manifest lists appropriately, and we've applied a minor patch to our builds of manifest-tool to account for estesp/manifest-tool#47.

We're really close to finally supporting this properly! 🤘 👍 🎂

[tianon]

So it begins: docker-library/oi-janky-groovy@477c8aa

As long as all goes well, we should be able to finally close this issue very shortly!

[tianon]

This is pretty much done now! Just a few stragglers left. Success!

root@rpi3:~# docker run -it --rm debian:jessie-slim dpkg --print-architecture
Unable to find image 'debian:jessie-slim' locally
jessie-slim: Pulling from library/debian
01c8fcecd171: Pull complete 
Digest: sha256:01f3ae3155dcdd3c86470cbcb81b1215dff357329e6750b1b4cf663e3c00a099
Status: Downloaded newer image for debian:jessie-slim
arm64

[friism]

@tianon wait, so this is rolled out to all the library images?

[tianon]

@friism yessir! There are a few still in queue (which still includes ubuntu, among a couple others, since it's a "heavy hitter"), but the majority have been pushed and the rest will be pushed before EOD!

[friism]

@tianon including windoze?

[tianon]

@friism well, we're not using SharedTags yet, but the functionality is implemented and manifest lists for multiarch was the prerequisite, so now we can update hello-world:latest to run on Windows as well too (this was all groundwork for that 👍)

[friism]

ah, got it - really looking forward to golang et.al. supporting this

[mkumatag]

hurray..! @friism

root@wave0m:~# docker run -ti golang:1.8.3 go version
go version go1.8.3 linux/ppc64le
root@wave0m:~#

[SvenDowideit]

@tianon congrats!