Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Properly save and restore the Go build cache in our CI #552

Merged
merged 2 commits into from
Nov 9, 2023
Merged

Properly save and restore the Go build cache in our CI #552

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4bd465b
Properly save and restore the Go build cache
jgiannuzzi Nov 4, 2023
9949dae
Merge branch 'main' into go-cache
jgiannuzzi Nov 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .github/actions/setup-go/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: "Setup Go"
description: "Setup Go with caching"
inputs:
cache-prefix:
description: "Prefix for the cache key"
required: true
outputs:
go-version:
description: "The installed Go version"
value: ${{ steps.setup-go.outputs.go-version }}
runs:
using: "composite"
steps:
- id: setup-go
uses: actions/setup-go@v4
with:
go-version: "1.21"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can parametrize the go version and keep this as a default.

  go-version:
    description: "Go version to install"
    required: false
    default: "1.21"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes but I don't want to have to change it in multiple places, which is why I "hard-coded" it here 😄

check-latest: true
cache: false
- id: cache-info
shell: bash
run: echo path=$(go env GOCACHE) >> $GITHUB_OUTPUT
- uses: actions/cache@v3
with:
path: ${{ steps.cache-info.outputs.path }}
key: ${{ inputs.cache-prefix }}-go-${{ steps.setup-go.outputs.go-version }}-mod-${{ hashFiles('.go-build-tags', 'go.sum') }}
58 changes: 49 additions & 9 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,29 @@ jobs:
uses: actions/checkout@v4

- name: Setup Go
uses: actions/setup-go@v4
id: setup-go
uses: ./.github/actions/setup-go
with:
go-version: "1.21"
cache-prefix: go-lint

- name: Compute tools cache info
id: tools-cache-info
run: |
echo path=$(go env GOPATH)/bin >> $GITHUB_OUTPUT
echo make-hash=$(make -n install-tools | sha256sum | cut -d' ' -f1) >> $GITHUB_OUTPUT

- name: Setup tools cache
uses: actions/cache@v3
id: tools-cache
with:
path: ${{ steps.tools-cache-info.outputs.path }}
key: tools-go-${{ steps.setup-go.outputs.go-version }}-make-${{ steps.tools-cache-info.outputs.make-hash }}

- name: Install tools
if: steps.tools-cache.outputs.cache-hit != 'true'
env:
GOCACHE: /tmp/tools/go-build
GOMODCACHE: /tmp/tools/go-mod
run: make install-tools

- name: Check formatting
Expand Down Expand Up @@ -92,9 +110,9 @@ jobs:
uses: actions/checkout@v4

- name: Setup Go
uses: actions/setup-go@v4
uses: ./.github/actions/setup-go
with:
go-version: "1.21"
cache-prefix: go-unit-tests

- name: Run Go Unit Tests
run: make test-go-unit
Expand All @@ -120,12 +138,33 @@ jobs:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Compute cache info
id: cache-info
run: |
image=$(yq .services.service.image < tests/integration/docker-compose.yml)
echo go-version=$(skopeo inspect docker://${image} | jq -r .Digest | cut -d: -f2) >> $GITHUB_OUTPUT

- name: Setup cache
id: cache
uses: actions/cache@v3
with:
path: /tmp/go-cache/go-build.tar
key: go-integration-tests-image-${{ steps.cache-info.outputs.go-version }}-mod-${{ hashFiles('.go-build-tags', 'go.sum') }}

- name: Restore cache
if: steps.cache.outputs.cache-hit == 'true'
run: docker run --rm -v fml-integration-tests_go-cache:/cache -v /tmp/go-cache:/src alpine tar xf /src/go-build.tar -C /cache

- name: Run Integration Tests
run: make service-test
env:
DOCKER_BUILDKIT: 1
FML_DATABASE_URI: ${{ matrix.database-uri }}

- name: Save cache
if: steps.cache.outputs.cache-hit != 'true'
run: docker run --rm -v fml-integration-tests_go-cache:/cache -v /tmp/go-cache:/dst alpine tar cf /dst/go-build.tar -C /cache go-build

python-integration-tests:
if: github.event_name == 'schedule' || github.event_name == 'push' || github.event.pull_request.head.repo.id != github.event.pull_request.base.repo.id
name: Python Integration Tests
Expand Down Expand Up @@ -153,16 +192,17 @@ jobs:
uses: actions/checkout@v4

- name: Setup Go
uses: actions/setup-go@v4
id: setup-go
uses: ./.github/actions/setup-go
with:
go-version: "1.21"
cache-prefix: python-integration-tests

- name: Run ${{ matrix.api }} integration tests
run: ./tests/integration/python/${{ matrix.api }}/test.sh

build:
if: github.event_name == 'schedule' || github.event_name == 'push' || github.event.pull_request.head.repo.id != github.event.pull_request.base.repo.id
name: Build software distribution for ${{ matrix.os }}/${{ matrix.arch }}
name: Build (${{ matrix.os }}/${{ matrix.arch }})
strategy:
matrix:
os: [darwin, linux, windows]
Expand All @@ -187,9 +227,9 @@ jobs:
fetch-depth: 0
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you need all git data in this job?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we do in order to compute the version (we need tags + amount of commits since the tag)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aha, I missed that part


- name: Setup Go
uses: actions/setup-go@v4
uses: ./.github/actions/setup-go
with:
go-version: "1.21"
cache-prefix: build-${{ matrix.os }}-${{ matrix.arch }}

- name: Setup Python
uses: actions/setup-python@v4
Expand Down