Build Docker image and push to GHCR

[br3ndonland]

Description

Closes #58

Up to this point, the project has been set up as a Docker action referencing the Dockerfile.

gh-action-pypi-publish/action.yml

Lines 86 to 88 in 3fbcf7c

	runs:
	using: docker
	image: Dockerfile

The downside to using the Dockerfile for the action is that the Docker image must be built every time the action is used (#58).

This PR will set up the project to build the Docker image and push it to GitHub Container Registry (GHCR). This change will speed up user workflows every time the action is used because the workflows will simply pull the Docker image from GHCR instead of building again.

Changes

Build container image with GitHub Actions

This PR will build Docker images with the Docker CLI (docker build). Builds will include inline cache metadata so layers can be reused by future builds.

This PR only proposes to build container images for x86_64 (linux/amd64) because GitHub Actions Linux runners currently only support x86_64 CPU architectures (actions/runner-images#5631), and this project only supports GitHub Actions Linux runners. The README explains:

Since this GitHub Action is docker-based, it can only be used from within GNU/Linux based jobs in GitHub Actions CI/CD workflows. This is by design and is unlikely to change due to a number of considerations we rely on.

Push container image to GHCR

The workflow will log in to GHCR using the built-in GitHub token and push the Docker image. Workflow runs triggered by pull requests will build the Docker image and run the smoke tests but will not push the Docker image.

Update action to pull container image from GHCR

Docker actions support pulling in pre-built Docker images by supplying a registry address to the image: key. The downside to this syntax is that there's no way to specify the correct Docker tag because the GitHub Actions image: and uses: keys don't accept any context. For example, if a user's workflow has uses: pypa/gh-action-pypi-publish@release/v1.8, then the action should pull in a Docker image built from the release/v1.8 ref, something like ghcr.io/pypa/gh-action-pypi-publish:release-v1.8 (Docker tags can't have /).

# this works but the image tag can't be customized
runs:
  using: docker
  image: docker://ghcr.io/pypa/gh-action-pypi-publish:release-v1.8

# this doesn't work because `image:` doesn't support context
runs:
  using: docker
  image: docker://ghcr.io/pypa/gh-action-pypi-publish:${{ github.action_ref }}

The workaround is to switch the top-level action.yml to a composite action that then calls the Docker action, substituting the correct image name and tag.

Related

• GitHub docs: Actions - Creating actions - Creating a composite action
• GitHub docs: Packages - Working with a GitHub Packages registry - Container registry (GHCR)
• github.action_repository and github.action_ref are empty in run for composite actions actions/runner#2473
• make reference accessible in reusable workflow  actions/toolkit#1264
• https://github.com/orgs/community/discussions/26212
• https://github.com/orgs/community/discussions/27048
• https://github.com/orgs/community/discussions/41927
• https://github.com/orgs/community/discussions/63675

[webknjaz]

This looks.. intriguing! I don't remember if I ever considered combining composite+docker actions (I did play with having two composites in the same repo in the past, though).

I'll need to take some time to think about it and look through the patch more closely. Please, don't expect an immediate review, however it does look very promising at glance!

Originally I thought that I'd have a workflow where I trigger a release, that release adds a commit that hardcodes an update to action.yml with the "future" version tag, tags that commit and pushes it (post docker publish). It wouldn't be on the main branch, the tags would be on the orphaned leaves.

This looks like a better idea so far. Thanks again!

[br3ndonland]

That sounds great. Take your time. Thanks for your consideration.

If you do decide to accept this change, I'm happy to help maintain the workflows in the future. Feel free to mention me @br3ndonland and I will help address any issues that come up.

[webknjaz]

Thanks! I've hit "rebase" on the UI to get this on top of the recent changes/linting/lockfile bumps but haven't yet looked into it deeper.

[webknjaz]

Two separate workflow runs is often hard to track. Instead, I adopted a practice of modularizing the workflow pieces as reusable workflows having the reusable- prefix in their names. This allows embedding everything in all the right places. Let's try this, WDYT?

[br3ndonland]

I will return to this suggestion at a later time.

[webknjaz]

I think we should be able to circle back to this already...

[br3ndonland]

Two separate workflow runs is often hard to track. Instead, I adopted a practice of modularizing the workflow pieces as reusable workflows having the reusable- prefix in their names. This allows embedding everything in all the right places. Let's try this, WDYT?

@webknjaz I removed workflow_call and switched to a reusable workflow instead.

Note that the required status checks for the repo will need to be updated.

[webknjaz]

I'm done with the initial review. More is needed, but I'd rather accept what I can through separate PRs to make this one smaller. And the suggested refactoring could be done in parallel. I think that generating the file is a good idea. It should be possible to write the file without bringing in the PyYAML dependency. But it's not that easy for reading it. Can we make use of yq somehow, and convert YAML to JSON this way, maybe?

[br3ndonland]

@webknjaz thank you for your detailed review. I've addressed most of your comments so far.

[webknjaz]

Commits like 213c885ac41d769527ac150e2e633bb1ccd886d4 aren't really necessary since Git would automatically absorb changes applied separately. FYI. In fact, this one may have a harmful effect — when the label change is merged into the default branch, and this one rebased on top, Git will keep the removal commit and attempt deleting the label 🤷‍♂️

Anyway, we'll address this later on.

[webknjaz]

@br3ndonland I'm sure my review is incomplete, but hopefully it gives you enough ideas to try out until the next time.

I think it'd be nice to get this in before #236 if at all possible.

[webknjaz]

@br3ndonland could you also rebase this branch locally? The GH button doesn't work, meaning there's going to be some conflicts to resolve.

[webknjaz]

Is it possible to pass the image between containers? It would be a good idea to pass it to the tests job and then to another one that would actually publish it, as opposed to publishing and then testing whether it should've been published in the first place...

[br3ndonland]

Do you mean, "pass the image between workflow jobs?" Not directly. A different workflow job is a different virtual machine. We could upload and download artifacts, but that seems like unnecessary complexity.

What about just running the smoke tests before the Docker builds?

[webknjaz]

Yeah, I was imagining using the artifacts. In the past there was one project where I was publishing to GHCR from one job with some unique tag and then pulling from the following one. But using artifacts is better.

It's indeed more complex, but that's how I build testing these days. OTOH, since you asked to cut the scope, I'll consider this a future improvement idea for now.

[webknjaz]

@br3ndonland when running in another repo, wouldn't github.ref_name point to that repo instead of ours? Is it safe to use this var as a fallback? Same question for github.event.pull_request. Is this to cover contributions to the action itself?

[br3ndonland]

If another repo is using the marketplace action (uses: pypa/gh-action-pypi-publish), env.ACTION_REF will be set from github.action_ref.

If the action is being run on a PR to this repo, env.PR_REF will be set from github.event.pull_request.head.ref.

If the action is being run from a push, release, or other event in this repo, github.ref_name will be used.

[webknjaz]

Alright. Thanks for clarifying. So I'll need to be careful in the release process to make an image before tagging…

[webknjaz]

@br3ndonland with https://github.com/pypa/gh-action-pypi-publish/pull/230/files#r1787026278 in mind, if I switch to the process where I trigger the GHA workflow, wait for the container to show up and then manually tag the same commit, would that cover everything necessary post this patch? If yes, I think it'd fine to stop making the changes and merge this. I'll still want to see the clarifications on that question about reference fallbacks, though. And the PR needs rebasing to resolve the conflicts.

[webknjaz]

@br3ndonland so I was almost ready to merge this and realized that I don't understand a few things. Could you help me get some clarity?

1. When workflow_dispatch is triggered, and I type in 1.12.0, I understand that it'll publish ghcr.io/pypa/gh-action-pypi-publish:v1.12.0, ghcr.io/pypa/gh-action-pypi-publish:v1.12 and ghcr.io/pypa/gh-action-pypi-publish:v1. But how about ghcr.io/pypa/gh-action-pypi-publish:release-v1 and ghcr.io/pypa/gh-action-pypi-publish:unstable-v1? What's the process for those?
2. Should the case of uses: pypa/gh-action-pypi-publish@unstable/v1 not hit the GHCR image and build the image like it does now? Does set_image() need to be modified to implement this?
3. Should the case of uses: pypa/gh-action-pypi-publish@release/v1 hit the :v1 image, or should the image generation workflow produce one more tag? It looks like these two bits aren't cohesive: reading the diff once again, it looks like if I enter release/v1.11.0, it'll produce ghcr.io/pypa/gh-action-pypi-publish:release-v1, ghcr.io/pypa/gh-action-pypi-publish:release-v1.11 and ghcr.io/pypa/gh-action-pypi-publish:release-v1.11.0 which would then not match references like uses: pypa/gh-action-pypi-publish@v1.11.0.

Am I missing anything?