Fix integration tests #7827

sagor999 · 2022-01-25T23:46:53Z

Description

Fixes integrations tests. Made sure it runs manually via go or docker.
Added optional parameters to test additional tests (for example one that requires enterprise license or gitlab integration).

Related Issue(s)

Fixes #7675

How to test

Spin up new cluster via workspace-preview and follow steps in README.md file for manual testing via running go test directly or via docker container that werft builds.

Release Notes

Fix integration tests

Documentation

codecov · 2022-01-26T00:31:11Z

Codecov Report

Merging #7827 (ec61d39) into main (2fb5775) will decrease coverage by 2.22%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #7827      +/-   ##
==========================================
- Coverage   12.42%   10.20%   -2.23%     
==========================================
  Files          42       18      -24     
  Lines        4248     1009    -3239     
==========================================
- Hits          528      103     -425     
+ Misses       3679      905    -2774     
+ Partials       41        1      -40

Flag	Coverage Δ
components-blobserve-app	`?`
components-blobserve-lib	`?`
components-gitpod-cli-app	`10.20% <ø> (ø)`
components-local-app-app-darwin-amd64	`?`
components-local-app-app-darwin-arm64	`?`
components-local-app-app-linux-amd64	`?`
components-local-app-app-linux-arm64	`?`
components-local-app-app-windows-386	`?`
components-local-app-app-windows-amd64	`?`
components-local-app-app-windows-arm64	`?`
components-registry-facade-app	`?`
components-registry-facade-lib	`?`
installer-raw-app	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
installer/pkg/components/ws-manager/role.go
components/registry-facade/pkg/registry/metrics.go
components/blobserve/pkg/blobserve/blobserve.go
...omponents/registry-facade/pkg/registry/manifest.go
installer/pkg/common/storage.go
components/local-app/pkg/auth/auth.go
...components/ws-manager/unpriviledged-rolebinding.go
installer/pkg/common/objects.go
...onents/registry-facade/pkg/registry/layersource.go
installer/pkg/components/ws-manager/tlssecret.go
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9f70a0c...ec61d39. Read the comment docs.

utam0k

Awesome! I left a few comments.

test/entrypoint.sh

test/README.md

test/pkg/integration/setup.go

utam0k · 2022-01-26T23:43:00Z

/approve

kylos101 · 2022-01-26T23:44:43Z

/hold I am actively reviewing.

kylos101

Just a few questions, nice work, @sagor999 !

test/README.md

kylos101 · 2022-01-26T22:12:40Z

test/README.md

+3. To test gitlab integration, add `-gitlab=true`
+4. All other tests.
+
+To run the tests:


Are you able to run the tests against a preview environment namespace? Like for this branch, as an example?

I know the goal was to get it running against workspace-preview, but it would be interesting to see how it fairs here, too.

Ideally we'd also be able to use these tests against core-dev (a set of namespaces in a single GKE cluster). Pretty soon core-dev will be replaced with Hertzner, where each preview environment (branch) gets a single node K3s cluster.

Hm. There is something with permissions when trying to run go tests in core-dev. when test calls kubectl it cannot access kubeconfig.

And also you cannot run it via docker container due to permissions:

kubectl apply -f integration.yaml serviceaccount/integration-svc created job.batch/integration-job created Error from server (Forbidden): error when creating "integration.yaml": roles.rbac.authorization.k8s.io is forbidden: User "pavel@gitpod.io" cannot create resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "staging-kyleb-installer-integration": requires one of ["container.roles.create"] permission(s). Error from server (Forbidden): error when creating "integration.yaml": rolebindings.rbac.authorization.k8s.io is forbidden: User "pavel@gitpod.io" cannot create resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "staging-kyleb-installer-integration": requires one of ["container.roleBindings.create"] permission(s).

I'm sorry I missed your response from a couple days ago! 🙏

I rebased with main and did ec61d39 to get the tests runnable in werft against core-dev. May I ask for you to review?

I just tried kubectl apply and had similar trouble in core-dev, which may be intentional (in hindsight) so that we do not "break" core-dev for other namespaces. In other words, kubectl apply is probably best for us (in hindsight) in single node clusters dedicated to ourselves (workspace preview, harvester, etc.)

Looks good, thank you!

test/pkg/integration/integration.go

test/tests/components/image-builder/builder_test.go

kylos101 · 2022-01-30T00:27:48Z

@sagor999 This run seems fast, like almost too good to be true. 🤔 WDYT? Did you see similar speeds with workspace-preview, after downloading images?

First test: 'Test succeeded at Sun Jan 30 00:16:10 UTC 2022'
Last test: 'Test succeeded at Sun Jan 30 00:18:11 UTC 2022'

sagor999 · 2022-01-31T15:17:13Z

This run seems fast, like almost too good to be true. thinking WDYT? Did you see similar speeds with workspace-preview, after downloading images?

@kylos101
yeah, that seems fine. It only ran a few tests, and to run more extensive testing you need to provide -username param, those additional tests usually run longer.

kylos101 · 2022-01-31T15:19:11Z

/unhold

Excellent work @sagor999 ! Super stoked that we can run tests again.

kylos101 · 2022-01-31T15:19:46Z

/LGTM

roboquat · 2022-01-31T15:19:52Z

LGTM label has been added.

Git tree hash: 003cc00ee0c1d28a42017c05fc921c514cf8f26c

roboquat · 2022-01-31T15:19:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kylos101, utam0k

Associated issue: #7675

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [kylos101,utam0k]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sagor999 added the do-not-merge/work-in-progress label Jan 25, 2022

roboquat added do-not-merge/release-note-label-needed size/XL and removed do-not-merge/work-in-progress labels Jan 25, 2022

sagor999 marked this pull request as draft January 25, 2022 23:47

roboquat added the do-not-merge/work-in-progress label Jan 25, 2022

sagor999 force-pushed the kyleb/installer-integration branch from c373e3b to f481e40 Compare January 26, 2022 01:42

sagor999 changed the title ~~Draft: Fix integration tests~~ Fix integration tests Jan 26, 2022

sagor999 force-pushed the kyleb/installer-integration branch from f481e40 to b7ec552 Compare January 26, 2022 01:52

roboquat added release-note and removed do-not-merge/release-note-label-needed labels Jan 26, 2022

sagor999 marked this pull request as ready for review January 26, 2022 01:56

roboquat removed the do-not-merge/work-in-progress label Jan 26, 2022

sagor999 requested a review from a team January 26, 2022 01:56

utam0k reviewed Jan 26, 2022

View reviewed changes

test/entrypoint.sh Show resolved Hide resolved

test/README.md Show resolved Hide resolved

test/pkg/integration/setup.go Outdated Show resolved Hide resolved

sagor999 force-pushed the kyleb/installer-integration branch from b7ec552 to 30a9c96 Compare January 26, 2022 19:03

sagor999 requested a review from aledbf January 26, 2022 19:28

roboquat added the approved label Jan 26, 2022

roboquat added the do-not-merge/hold label Jan 26, 2022

kylos101 reviewed Jan 27, 2022

View reviewed changes

kylos101 added the team: workspace Issue belongs to the Workspace team label Jan 27, 2022

sagor999 force-pushed the kyleb/installer-integration branch from 30a9c96 to 95b0747 Compare January 27, 2022 23:26

sagor999 requested a review from kylos101 January 27, 2022 23:39

kylos101 and others added 4 commits January 29, 2022 16:06

[test] Run tests via a vanilla pod

b6e4704

[test] Avoid randomization which breaks tests

6af99c5

add which test is being executed

f36b942

[tests] fix integration tests

d0c1544

kylos101 force-pushed the kyleb/installer-integration branch 4 times, most recently from 28713df to 661f27a Compare January 29, 2022 23:30

[tests] run in core-dev again

ec61d39

kylos101 force-pushed the kyleb/installer-integration branch from 661f27a to ec61d39 Compare January 30, 2022 00:09

roboquat removed the do-not-merge/hold label Jan 31, 2022

roboquat assigned kylos101 Jan 31, 2022

roboquat added the lgtm label Jan 31, 2022

roboquat merged commit a82a23f into main Jan 31, 2022

roboquat deleted the kyleb/installer-integration branch January 31, 2022 15:20

roboquat added deployed: workspace Workspace team change is running in production deployed Change is completely running in production labels Jan 31, 2022

Fix integration tests #7827

Fix integration tests #7827

Uh oh!

Conversation

sagor999 commented Jan 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

How to test

Release Notes

Documentation

Uh oh!

codecov bot commented Jan 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

utam0k left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

utam0k commented Jan 26, 2022

Uh oh!

kylos101 commented Jan 26, 2022

Uh oh!

kylos101 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kylos101 Jan 26, 2022

Choose a reason for hiding this comment

Uh oh!

sagor999 Jan 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kylos101 Jan 30, 2022

Choose a reason for hiding this comment

Uh oh!

sagor999 Jan 31, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kylos101 commented Jan 30, 2022

Uh oh!

sagor999 commented Jan 31, 2022

Uh oh!

kylos101 commented Jan 31, 2022

Uh oh!

kylos101 commented Jan 31, 2022

Uh oh!

roboquat commented Jan 31, 2022

Uh oh!

roboquat commented Jan 31, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sagor999 commented Jan 25, 2022 •

edited

Loading

codecov bot commented Jan 26, 2022 •

edited

Loading

sagor999 Jan 27, 2022 •

edited

Loading