Build multi-arch images for commits on main #7900

ddelange · 2022-12-15T06:21:49Z

On commits to main, push:

prefecthq/prefect-dev:main-python3.X
prefecthq/prefect-dev:sha-424242-python3.X
prefecthq/prefect-dev:main-python3.X-conda
prefecthq/prefect-dev:sha-424242-python3.X-conda

Example

Steps 1-3 ref #7740 (comment) for multi arch docker images, towards #5388

Checklist

This pull request references any related issue by including "closes <link to issue>"
- If no issue exists and your change is not a small fix, please create an issue first.
This pull request includes tests or only affects documentation.
This pull request includes a label categorizing the change e.g. fix, feature, enhancement

netlify · 2022-12-15T06:22:00Z

✅ Deploy Preview for prefect-orion ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`9b29971`
🔍 Latest deploy log	https://app.netlify.com/sites/prefect-orion/deploys/63bc33d95957ca0009273f99
😎 Deploy Preview	https://deploy-preview-7900--prefect-orion.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

.github/workflows/docker-builds.yaml

zanieb · 2022-12-15T16:03:45Z

.github/workflows/docker-builds.yaml

+          images: prefecthq/prefect-dev
+          # first tag generated here will be used in CACHE_IMAGE below
+          tags: |
+            type=raw,enable=${{ github.event_name == 'release' }},value=${{ github.ref_name }},suffix=-python${{ matrix.python-version }}


This will be skipped since this is on push/dispatch for now, right?

thanks for pointing this one out, this is not intended. it should be negated, because ref_name points to a tag on release event and not a branch. The git rev-parse setup from before would return HEAD because a release event is a detached branch so is also unusable. it seems to be hard to get out which branch the release was made from.

Suggested change

type=raw,enable=${{ github.event_name == 'release' }},value=${{ github.ref_name }},suffix=-python${{ matrix.python-version }}

type=raw,enable=${{ github.event_name != 'release' }},value=${{ github.ref_name }},suffix=-python${{ matrix.python-version }}

this means there will be no cache hit on a release event, because the CACHE_IMAGE would point to the sha pattern instead of the branch pattern, and we dont have buildcache for the sha pattern. fix would be to hardcode a buildcache tag in the cache-from and cache-to based on github.matrix. so not getting it from metadata-action.

I don't think we need to know what branch releases are made from, we can just always pull from the cache on main?

yes fine by me, will hardcode 👍

zanieb · 2022-12-15T16:04:20Z

Thanks for splitting this up! It's much easier to review.

zanieb · 2022-12-15T16:09:25Z

.github/workflows/docker-builds.yaml

+      - name: Checkout
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0


This reminds me of the issue with retagging images build from main on release — the image will have the wrong version information. versioneer pulls information from git into a static file on Python package build.

we solved this problem with a mount directive: https://github.com/pypa/setuptools_scm#usage-from-docker

although that does void all subsequent layer caches, so we have to do it as late as possible 😅

Hm we just copy the .git folder in during the build step which works fine. The issue is that we cannot retag these images on release, we'll need to rebuild them.

ah gotcha. yes copying .git might be unwanted layer size if you have a lot of history. setuptools_scm also needs fetch-depth: 0 btw... ref pypa/setuptools-scm#480

they do however provide a 'pretend' environment variable mechanic you could combine with ${{ github.ref_name}} to 'solve' the problem

.github/workflows/docker-builds.yaml

zanieb · 2022-12-28T22:00:29Z

@jawnsy would you be able to take a look at this and investigate Docker credentials that can only push to prefect-dev?

ddelange · 2023-01-04T11:41:11Z

fwiw I can see no real security concerns with using the same token for all image pushes without manual approval. the action only pushes to production on a release event (#7901), and that can only be changed if a malicious PR was merged into main

zanieb · 2023-01-06T18:32:38Z

I've now added DockerHub credentials scoped to prefect-dev to the "dev" GitHub Actions Environment.

zanieb · 2023-01-07T22:03:36Z

.github/workflows/docker-builds.yaml

+    name: Build Docker images
+    runs-on: ubuntu-latest


I believe you'll need "environment: dev" like we do for the prod environment https://github.com/PrefectHQ/prefect/blob/main/.github/workflows/release.yaml#L186

good catch! d6defda

zanieb

Sweet thanks! Let's give this a try :)

ddelange · 2023-01-09T17:05:43Z

fix for failing main incoming https://github.com/ddelange/prefect/actions/runs/3875949170/jobs/6609182214#step:7:1

…r-builds-release * 'main' of https://github.com/ddelange/prefect: (131 commits) Fix docker-builds.yaml templating syntax (PrefectHQ#8114) Rename Worker pools -> work pools (PrefectHQ#8107) Build multi-arch images for commits on main (PrefectHQ#7900) [Issue PrefectHQ#456] Change example of `infra_override` in docs/concepts/deployment (PrefectHQ#8101) Bump @playwright/test from 1.29.1 to 1.29.2 in /orion-ui (PrefectHQ#8105) Bump @prefecthq/orion-design from 1.1.53 to 1.1.54 in /orion-ui (PrefectHQ#8104) Update deployment docs to include tag and idempotency key (PrefectHQ#7771) Add `BaseWorker` and `ProcessWorker` (PrefectHQ#7996) Add Peyton and Serina as global code owners (PrefectHQ#8098) Add release notes for 2.7.7 (PrefectHQ#8091) Add youtube badge (PrefectHQ#8089) Adds `MAX_RRULE_LENGTH` (PrefectHQ#7762) Limit task run cache key size (PrefectHQ#7275) Add --match flag to work queues documentation (PrefectHQ#7768) Modify disable ssl setting tests to allow any for headers and timeout (PrefectHQ#8086) Add test for allow_failure and quote (PrefectHQ#8055) Adds `experimental_field` decorator (PrefectHQ#8066) add docs on migrating block documents (PrefectHQ#8085) Add Redoc documentation for REST API reference (PrefectHQ#7503) Allow disabling SSL verification (PrefectHQ#7850) ...

…refect into docker-builds-consolidation * 'docker-builds-release' of https://github.com/ddelange/prefect: (131 commits) Fix docker-builds.yaml templating syntax (PrefectHQ#8114) Rename Worker pools -> work pools (PrefectHQ#8107) Build multi-arch images for commits on main (PrefectHQ#7900) [Issue PrefectHQ#456] Change example of `infra_override` in docs/concepts/deployment (PrefectHQ#8101) Bump @playwright/test from 1.29.1 to 1.29.2 in /orion-ui (PrefectHQ#8105) Bump @prefecthq/orion-design from 1.1.53 to 1.1.54 in /orion-ui (PrefectHQ#8104) Update deployment docs to include tag and idempotency key (PrefectHQ#7771) Add `BaseWorker` and `ProcessWorker` (PrefectHQ#7996) Add Peyton and Serina as global code owners (PrefectHQ#8098) Add release notes for 2.7.7 (PrefectHQ#8091) Add youtube badge (PrefectHQ#8089) Adds `MAX_RRULE_LENGTH` (PrefectHQ#7762) Limit task run cache key size (PrefectHQ#7275) Add --match flag to work queues documentation (PrefectHQ#7768) Modify disable ssl setting tests to allow any for headers and timeout (PrefectHQ#8086) Add test for allow_failure and quote (PrefectHQ#8055) Adds `experimental_field` decorator (PrefectHQ#8066) add docs on migrating block documents (PrefectHQ#8085) Add Redoc documentation for REST API reference (PrefectHQ#7503) Allow disabling SSL verification (PrefectHQ#7850) ...

ddelange requested a review from zanieb as a code owner December 15, 2022 06:21

ddelange changed the title ~~Build multi-arch prefecthq/prefect-dev:main-python3.X images~~ Build multi-arch images for commits on main Dec 15, 2022

ddelange mentioned this pull request Dec 15, 2022

Build multi-arch images for releases #7901

Merged

3 tasks

ddelange force-pushed the docker-builds-dev-main branch from ea6f99f to c64d46a Compare December 15, 2022 11:24

Build multi-arch images for commits on main

2db6388

ddelange force-pushed the docker-builds-dev-main branch from c64d46a to 2db6388 Compare December 15, 2022 11:37

zanieb reviewed Dec 15, 2022

View reviewed changes

.github/workflows/docker-builds.yaml Show resolved Hide resolved

zanieb reviewed Dec 15, 2022

View reviewed changes

Deprecate CACHE_IMAGE

157dd9b

ddelange commented Dec 16, 2022

View reviewed changes

.github/workflows/docker-builds.yaml Outdated Show resolved Hide resolved

Only update buildcache on commits to main

54c8469

zanieb reviewed Dec 19, 2022

View reviewed changes

.github/workflows/docker-builds.yaml Show resolved Hide resolved

Merge branch 'main' into docker-builds-dev-main

fff5b5b

zanieb reviewed Jan 7, 2023

View reviewed changes

Add environment to build job

d6defda

ddelange requested review from serinamarie and peytonrunyan as code owners January 8, 2023 08:02

Merge branch 'main' into docker-builds-dev-main

976f9bc

zanieb approved these changes Jan 8, 2023

View reviewed changes

Merge branch 'main' into docker-builds-dev-main

9b29971

zanieb merged commit 7c92aca into PrefectHQ:main Jan 9, 2023

ddelange mentioned this pull request Jan 9, 2023

Fix docker-builds.yaml templating syntax #8114

Merged

3 tasks

masonmenges pushed a commit that referenced this pull request Jan 10, 2023

Build multi-arch images for commits on main (#7900)

2adb4bc

github-actions bot pushed a commit that referenced this pull request Jan 12, 2023

Build multi-arch images for commits on main (#7900)

aad511e

github-actions bot pushed a commit that referenced this pull request Jan 12, 2023

Build multi-arch images for commits on main (#7900)

38e741f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build multi-arch images for commits on main #7900

Build multi-arch images for commits on main #7900

ddelange commented Dec 15, 2022

netlify bot commented Dec 15, 2022 •

edited

Loading

zanieb Dec 15, 2022

ddelange Dec 15, 2022

zanieb Dec 15, 2022

ddelange Dec 15, 2022

ddelange Dec 16, 2022

zanieb commented Dec 15, 2022

zanieb Dec 15, 2022

ddelange Dec 15, 2022

ddelange Dec 15, 2022

zanieb Dec 15, 2022

ddelange Dec 15, 2022 •

edited

Loading

zanieb commented Dec 28, 2022

ddelange commented Jan 4, 2023 •

edited

Loading

zanieb commented Jan 6, 2023

zanieb Jan 7, 2023

ddelange Jan 8, 2023

zanieb left a comment

ddelange commented Jan 9, 2023

	type=raw,enable=${{ github.event_name == 'release' }},value=${{ github.ref_name }},suffix=-python${{ matrix.python-version }}
	type=raw,enable=${{ github.event_name != 'release' }},value=${{ github.ref_name }},suffix=-python${{ matrix.python-version }}

Build multi-arch images for commits on main #7900

Build multi-arch images for commits on main #7900

Conversation

ddelange commented Dec 15, 2022

Example

Checklist

netlify bot commented Dec 15, 2022 • edited Loading

✅ Deploy Preview for prefect-orion ready!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zanieb commented Dec 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddelange Dec 15, 2022 • edited Loading

Choose a reason for hiding this comment

zanieb commented Dec 28, 2022

ddelange commented Jan 4, 2023 • edited Loading

zanieb commented Jan 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zanieb left a comment

Choose a reason for hiding this comment

ddelange commented Jan 9, 2023

netlify bot commented Dec 15, 2022 •

edited

Loading

ddelange Dec 15, 2022 •

edited

Loading

ddelange commented Jan 4, 2023 •

edited

Loading