Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add setup-envtest in Makefile #404

Merged
merged 2 commits into from
Aug 2, 2021
Merged

Conversation

darkowlzz
Copy link
Contributor

Use setup-envtest for installing the binaries required for using
envtest.
The latest envtest binaries for k8s v1.21.2 are not compatible with
kubebuilder < v3, which results in envtest suite test tear down
failure. envtest fails to stop the kube-apiserver.

timeout waiting for process kube-apiserver to stop

Pin to k8s v1.19.2 envtest binaries that work with kubebuilder v2.

Copy link
Member

@hiddeco hiddeco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that combined with this PR we should be able to move over to the actions/envtest from fluxcd/pkg?

https://github.com/fluxcd/source-controller/blob/main/.github/workflows/e2e.yaml#L27-L30

Use setup-envtest for installing the binaries required for using
envtest.
The latest envtest binaries for k8s v1.21.2 are not compatible with
kubebuilder < v3, which results in envtest suite test tear down
failure. envtest fails to stop the kube-apiserver.

```
timeout waiting for process kube-apiserver to stop
```

Pin to k8s v1.19.2 envtest binaries that work with kubebuilder v2.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
@hiddeco hiddeco added the area/ci CI related issues and pull requests label Jul 30, 2021
@darkowlzz darkowlzz force-pushed the setup-envtest branch 15 times, most recently from 7aa83df to c975d37 Compare July 31, 2021 00:40
@darkowlzz
Copy link
Contributor Author

darkowlzz commented Jul 31, 2021

I've added a local github action for envtest to avoid installing go on the host for installing setup-envtest (GOROOT from the host is passed on to the run-test container and the builds in the container fails because the GOROOT passed from host doesn't exist inside the container). This will be moved to the pkg repo.
Observed some weird build failures due to the cached build files populated during setup-envtest install. Example of the build failure logs can be found here https://github.com/fluxcd/source-controller/pull/404/checks?check_run_id=3206245478#step:7:256.

Pasting some log lines below for future reference(the github actions logs will get deleted after a few months):

go: creating new go.mod: module tmp
go get: added sigs.k8s.io/controller-tools v0.5.0
cd api; /github/home/go/bin/controller-gen object:headerFile="../hack/boilerplate.go.txt" paths="./..."
-: go build internal/unsafeheader: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/32/322c1475077dbb3e3f149e186e0358a72016917dda03d9910e1abe3663d9ca04-a: no such file or directory
go build runtime/internal/atomic: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/bc/bcb9b2083df4553336339c1815cdfa18452f134067b1bd07acd1af3728d3e75c-a: no such file or directory
go build runtime/internal/sys: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/0a/0a4477cefe4c3522430b00940f0feb4ab86b0ac8c1fbb5bd17f7ff2656d5378d-a: no such file or directory
go build math/bits: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/d6/d681605cd480d72bfde50daf8de9b94205937f387746ac37eff39926be19b412-a: no such file or directory
go build unicode/utf8: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/5a/5ae44ff5b0dfd5092df2fc8641b04ad517b22012d699186e623dd84974ee1906-a: no such file or directory
go build internal/race: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/21/21204b9a9044eb2a686dfceb86034e5d753ab9105badb6f68c22358ee50bc236-a: no such file or directory
go build sync/atomic: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/2b/2ba2f289dec5650e6b5b4a6ffbc468be7e3d5aa6126a5abec1d33bc1880b2d42-a: no such file or directory
go build unicode: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/2e/2e4643207fcfbb3eb71a486abe55c0be4b140a44619554bc6d3b8413b2854609-a: no such file or directory
go build encoding: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/9b/9ba6ed76bb4a3c72b7d20f738df06c561d0b5851eb2c541c0f4a177c97bb65a8-a: no such file or directory
go build unicode/utf16: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/3a/3a210046015812087dc3ba9aa6b94b8dba3db0e9cce74fdb7426904bbab93c3c-a: no such file or directory
go build k8s.io/apimachinery/pkg/selection: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/5d/5dd2b10e90116d63cb703679741c0f45b36825521836d72a42b12ed0030a6a16-a: no such file or directory
go build github.com/google/go-cmp/cmp/internal/flags: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/8d/8d02e0eac337c439bac8d5a15b786dd122ff2b3603a03be16894144bf11cebbc-a: no such file or directory
go build internal/nettrace: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/25/255a069585d6033a295b8c4d07f3e064a1d3bbec510cfe78eb914d53523ede41-a: no such file or directory
go build internal/cpu: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/53/5354e9041207a1bfa5bc2e1f3293695eec315fd91708b5ffac90f80dcfed1594-a: no such file or directory
go build container/list: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/fd/fd12ac769da13df49f0202488b6b9962826a51d3fdf38be29efd70f440da6f94-a: no such file or directory
go build crypto/internal/subtle: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/f9/f90ccd3ff003c82b45bb3a96f1f67aef825d0a2ed33931e58cb8e447a6376424-a: no such file or directory
go build crypto/subtle: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/bb/bbe666d2773635afcfb138458060bda6ec9e79dcd9eac266247d81a24fb6da5a-a: no such file or directory
go build vendor/golang.org/x/crypto/cryptobyte/asn1: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/12/12836403574948506e35e893abf19a9e105b824030a0165b96f57bb213edbf29-a: no such file or directory
go build vendor/golang.org/x/crypto/internal/subtle: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/7b/7bc37a84c61287ab8d9481f8c770a5f0652f1adf4b0914bae8621c33500c1316-a: no such file or directory
go build k8s.io/apimachinery/pkg/types: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/b2/b2b76e52be31a92c3cc8ccf1af6b91061104f578473462bb2d245d15c0105520-a: no such file or directory
go build runtime/cgo: loading compiled Go files from cache: reading srcfiles list: cache entry not found: open /github/home/.cache/go-build/4b/4b3cf5cfa10e9b0116674cfcf5d43e9687d01359858b063815ae23d09e960345-a: no such file or directory
Error: not all generators ran successfully
run `controller-gen object:headerFile=../hack/boilerplate.go.txt paths=./... -w` to see all available markers, or `controller-gen object:headerFile=../hack/boilerplate.go.txt paths=./... -h` for usage
make: *** [Makefile:77: generate] Error 1

A workaround for this was to clean /github/home/.cache at the end of the action.

More details about the changes are in the commit messages.

Copy link
Member

@stefanprodan stefanprodan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, please open a PR for updating the action in fluxcd/pkg.

Thanks @darkowlzz 🏅

@darkowlzz
Copy link
Contributor Author

Created fluxcd/pkg#138 to update the action in fluxcd/pkg. In fluxcd/pkg, since the tests are run on host, it uncovered an issue with the KUBEBUILDER_ASSETS value, which was being set as $GITHUB_WORKSPACE/<repo-dir>/envtest. This works inside a container only. Using $RUNNER_WORKSPACE/<repo-dir>/envtest works on host and inside a container.
Once the envtest action in fluxcd/pkg is merged, I'll update this PR to use that and remove the local action.

Use fluxcd/pkg/actions/envtest to setup envtest.

Since the run-test action runs in a container, the host $PATH is not
passed to the test container. The ENV directive in actions/run-tests
Dockerfile is used to append the PATH with envtest binary path. This
helps avoid redownload of the envtest binaries when it's not found in
the path via the Makefile targets.

Signed-off-by: Sunny <darkowlzz@protonmail.com>
@darkowlzz
Copy link
Contributor Author

Updated to use fluxcd/pkg/actions/envtest 🙂

Copy link
Member

@stefanprodan stefanprodan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Thanks @darkowlzz

@stefanprodan stefanprodan merged commit aef56d8 into fluxcd:main Aug 2, 2021
@darkowlzz darkowlzz deleted the setup-envtest branch August 2, 2021 09:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/ci CI related issues and pull requests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants