-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CI: cache syntest
docker image
#16
Commits on Oct 31, 2022
-
Cache the syntest Docker image
Because it takes 3-5 minutes to build. I cobbled this config together from several incomplete sources: https://stackoverflow.com/questions/71348621/docker-re-use-container-image-by-caching https://github.com/docker/build-push-action/blob/master/docs/advanced/cache.md#github-cache https://github.com/docker/buildx/blob/master/docs/guides/cache/gha.md https://github.com/moby/buildkit#github-actions-cache-experimental https://docs.github.com/en/rest/actions/cache https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache Also, the GitHub Actions cache exporter is experimental...
Configuration menu - View commit details
-
Copy full SHA for 5704f0e - Browse repository at this point
Copy the full SHA 5704f0eView commit details -
(Try to) fix the cache not working
Just troubleshooting, don't mind me. Buildkit is disabled in Docker, by default. You have to enable it with that env. Because the cache is associated to buildkit, of course it should be enabled in the docker build of syntest.
Configuration menu - View commit details
-
Copy full SHA for b4aeec4 - Browse repository at this point
Copy the full SHA b4aeec4View commit details -
(Try to) fix the cache not working 2
Try with this install parameter to setup-buildx, that should achieve the same as DOCKER_BUILDKIT=1 and is referenced in the documentation for that action: https://github.com/docker/setup-buildx-action/blob/master/docs/advanced/install-default.md Also comments the scope param to build-push-action because the runner was complaining about it.
Configuration menu - View commit details
-
Copy full SHA for 3815bae - Browse repository at this point
Copy the full SHA 3815baeView commit details -
(Try to) fix the cache not working 3
So syntax.py has an escape hatch to not build the image if it already exists. (Which is probably bad because it won't be rebuilt on syntect version changes but whatever.) By tagging the image with the name syntax.py expects, we can probably avoid the rebuild?
Configuration menu - View commit details
-
Copy full SHA for 084c63a - Browse repository at this point
Copy the full SHA 084c63aView commit details
Commits on Nov 1, 2022
-
(Try to) fix the cache not working 4
By setting an explicit image tag, with version (latest) and all, on both the python script and build-push-action.
Configuration menu - View commit details
-
Copy full SHA for 0c13c49 - Browse repository at this point
Copy the full SHA 0c13c49View commit details -
Use builtctl to build syntest when in CI (cache 5)
After reviewing [^1]. Because it seems that only buildctl can import the cache, so let's use that command. [^1] https://github.com/moby/buildkit/tree/master#github-actions-cache-experimental
Configuration menu - View commit details
-
Copy full SHA for ceb2040 - Browse repository at this point
Copy the full SHA ceb2040View commit details -
Add a syntax test runner specific for CI (cache 6)
Instead of performing acrobatics / architecture astronautics to have the original script (syntax.py) fit both use cases (feedback loop in the dev's local + automated tests in CI). It's probably the right way to use build-push-action. Build, then run, instead of re-building again in a later step. I tried placing `syntax_ci.sh` in `.github/workflows/` but then it's not visible to the Github Actions runtime.
Configuration menu - View commit details
-
Copy full SHA for 156e5be - Browse repository at this point
Copy the full SHA 156e5beView commit details -
Tell buildkit to load its output to Docker (cache 7)
buildkit seems to be built for versatility, agnostic of the underlying engine / runtime / client / whatever Docker is supposed to be. So you need to explicitly tell it where you want the output of the build to be saved. load = true "is a shorthand for --output=type=docker"
Configuration menu - View commit details
-
Copy full SHA for 3f92d3f - Browse repository at this point
Copy the full SHA 3f92d3fView commit details -
Go back to running syntax.py in CI (cache 8)
The previous commit worked! The syntest image is properly generated and registered with `docker images`! But the job still fails because we need the part of syntax.py where it downloads the man syntax. Some syntax tests depend on that syntax being available in syntaxes/. Now, as the syntest image is already built when we hit syntax.py, it skips rebuilding the image. Which is par for the course: caching, based on docker/build-push-action. ETA: indeed, this version passes 🎉 🚀🚀
Configuration menu - View commit details
-
Copy full SHA for f7fcb03 - Browse repository at this point
Copy the full SHA f7fcb03View commit details -
Cleanup of caching the syntest docker image in CI
Remove failed/unneeded experiments from all the troubleshooting. The `with: install: true` in docker/setup-buildx-action is not needed because it aliases `docker build` to `docker buildx` which we don't use (not explicitly).
Configuration menu - View commit details
-
Copy full SHA for c38fd7c - Browse repository at this point
Copy the full SHA c38fd7cView commit details -
Share syntest cache across branches
So the Github Actions cache prevents sharing a cache across branches, with some exceptions: Only the cache for the current branch, the base branch and the default branch is accessible by a workflow. I believe that by setting the cache scope to "main", all workflows, including those that run on main (default & usually base branch) will use the same scope. And such a scope should be accessible to every workflow because it's the default branch's also. ETA: yup, this works. After the first workflow run for the main branch, every other branch (in a PR) can use its cached syntest image. This fucking works!! 🚀🚀🚀 References: https://github.com/docker/buildx/blob/eab3f704f53577310c2d05700ab663f8a7ad4c8e/docs/guides/cache/gha.md#scope https://github.com/docker/buildx/blob/eab3f704f53577310c2d05700ab663f8a7ad4c8e/docs/guides/cache/gha.md#synopsis https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#matching-a-cache-key
Configuration menu - View commit details
-
Copy full SHA for d4f1cc0 - Browse repository at this point
Copy the full SHA d4f1cc0View commit details -
Configuration menu - View commit details
-
Copy full SHA for f8d593a - Browse repository at this point
Copy the full SHA f8d593aView commit details