Export metadata about a packaged docker image

[huonw]

This has the docker target output a $name.docker-info.json file when packaged, to end up in dist/ or for use as a dependency. The file contains the full specification of the packaged image, including each of the registry/repository:tag names it was tagged with. This stops users from having to parse the pants output (or query docker itself) if they need this information, such as to place into a cloud deploy template, or to run the image locally.

This is achieved by defining dataclasses to set the schema of the JSON file (as discussed in #16999 and https://pantsbuild.slack.com/archives/C0D7TNJHL/p1664329466314179), which are then serialised via asdict/json.dump. The schema is relatively nested (not just a flat list of image names), so a user can easily look up a specific tag, even in the presence of templating, and different registries using different tags. This requires refactoring the image_refs function to return all the information.

For example:

# pants.toml
[docker.registries.demo-reg]
address = "example.invalid"
repository = "prefix/{name}"
use_local_alias = true

# BUILD
docker_image(
    name="docker",
    instructions=["FROM python:3.9.10", "RUN echo 1 > /file.txt"],
    image_tags=["pants-{pants.hash}", "latest"],
    registries=["other-example.invalid", "@demo-reg"],
)

Packaging the docker image results in JSON like the following:

{
  "version": 1,
  "image_id": "sha256:51e2d6c9b17c114696e2f4bf5ccfc41ee6886f50a3c0615a29ee20f71f030786",
  "registries": [
    {
      "alias": "demo-reg",
      "address": "example.invalid",
      "repository": "prefix/docker",
      "tags": [
        {
          "template": "latest",
          "tag": "latest",
          "uses_local_alias": true,
          "name": "demo-reg/prefix/docker:latest"
        },
        {
          "template": "pants-{pants.hash}",
          "tag": "pants-baee6c62401aae07329ab65c6d39ae9be558b5d4ab2feae89e1495d7bd1f6764",
          "uses_local_alias": true,
          "name": "demo-reg/prefix/docker:pants-baee6c62401aae07329ab65c6d39ae9be558b5d4ab2feae89e1495d7bd1f6764"
        },
        {
          "template": "latest",
          "tag": "latest",
          "uses_local_alias": false,
          "name": "example.invalid/prefix/docker:latest"
        },
        {
          "template": "pants-{pants.hash}",
          "tag": "pants-baee6c62401aae07329ab65c6d39ae9be558b5d4ab2feae89e1495d7bd1f6764",
          "uses_local_alias": false,
          "name": "example.invalid/prefix/docker:pants-baee6c62401aae07329ab65c6d39ae9be558b5d4ab2feae89e1495d7bd1f6764"
        }
      ]
    },
    {
      "alias": null,
      "address": "other-example.invalid",
      "repository": "docker",
      "tags": [
        {
          "template": "latest",
          "tag": "latest",
          "uses_local_alias": false,
          "name": "other-example.invalid/docker:latest"
        },
        {
          "template": "pants-{pants.hash}",
          "tag": "pants-baee6c62401aae07329ab65c6d39ae9be558b5d4ab2feae89e1495d7bd1f6764",
          "uses_local_alias": false,
          "name": "other-example.invalid/docker:pants-baee6c62401aae07329ab65c6d39ae9be558b5d4ab2feae89e1495d7bd1f6764"
        }
      ]
    }
  ]
}

Someone who wants to pull out the example.invalid/prefix/docker:pants-baee6c62401aae07329ab65c6d39ae9be558b5d4ab2feae89e1495d7bd1f6764 full name for @demo-reg tag can query (in JS/TS):

fileContents
  .registries.find(r => r.alias == "demo-reg")
  .tags.find(t => t.template == "pants-{pants.hash}" && !t.uses_local_alias)
  .name

In #16999, I flagged including the 'digest', which allows referring to the image in a content-addressed manner: registry/repository@digest (rather than just via a tag, like registry/repository:tag, which are mutable). However, the digest is not available until the image is pushed (to a v2 registry), and thus is omitted.

Fixes #16999

[ci skip-rust]

[ci skip-build-wheels]

[benjyw]

Re writing generated data out to a file: you create a Digest with something like:

digest = await Get(Digest, CreateDigest([FileContent("relpath/to/file.json", binary_data)]))

and then materialize that under dist/. To do that you request Workspace and DistDir in a @goal_rule (only goal rules can cause side effects, such as writing to disk). The latter gives the location of the dist dir, the former has a write_digest method for materializing the data.

[huonw]

Thanks. I've added that.

It seems like setting relpath in BuiltDockerImage does what I want, without needing a separate @goal_rule (just checking that this is expected?).

Thus, I think this is now ready: with the example above, running ./pants package ...:docker now includes a reference to writing a file (and it has the expected contents):

... [INFO] Wrote dist/.../docker.docker-info.json

(Where ... is a path like path.to.target if the BUILD file is in path/to/target)

And, a Python test that depends on the image passes too: (and similarly for experimental_shell_command)

python_tests(name="docker-tests", runtime_package_dependencies=[":docker"])

import pathlib
import json

def test_whatever() -> None:
    with pathlib.Path(".../docker.docker-info.json").open() as f:
        data = json.load(f)
    assert data["registries"].keys() == {"other-example.invalid", "@demo-reg"}

[kaos]

Sorry for the belated review. This is some cool stuff, thanks!

[kaos]

Thank you!

This is looking great. I'll wait for some more feedback from the other maintainers a few days. If nothing comes up I think we can merge this early next week.

[huonw]

Woohoo, thanks. Now that we've agreed on the approach, I've written a paragraph of documentation too.

[kaos]

ping @benjyw and perhaps @stuhood if you have any remarks on this?

[benjyw]

This looks good to me! Thanks!!

[calleo]

Will this be available in 2.15 or how does things work? 🙂

[benjyw]

Looks like it's slated for 2.16.

[kaos]

Maybe we could keep a record of cut-dates (and shas) for the various release branches.. :) (or I'll write a little script that automates the answer to questions like these.. 🤔 )

[benjyw]

git tag --contains <sha> already exists to answer questions like this.

[huonw]

The Github's commit view can answer that too, although I suspect neither that nor git tag --contains works with cherry picking. E.g. for the squash-and-merge commit 23cc363

[kaos]

The Github's commit view can answer that too, although I suspect neither that nor git tag --contains works with cherry picking. E.g. for the squash-and-merge commit 23cc363

Ah, cherry picks. Good point. 🤔 again...