tests: Stop and Wait for workspace at the end of each tests #12572

utam0k · 2022-09-01T02:51:40Z

Description

Stop and Wait for workspace at the end of each tests to be stabilization

Related Issue(s)

How to test

Run the test

Release Notes

NONE

Documentation

Werft options:

/werft with-preview

jenting · 2022-09-01T14:40:00Z

FYI https://werft.gitpod-dev.com/job/gitpod-custom-to-stablization-test.18

kylos101

What is the total time it takes to run tests assuming no failures? I ask because with increased timeouts, we want to make sure the test run doesn't take too long.

FYI, @jenting shared a test run that had failed tests:
https://werft.gitpod-dev.com/job/gitpod-custom-to-stablization-test.18#integration%20test:test-workspace

Also, I just ran a separate test run in werft, but had trouble with the environment, and notified the platform team:
https://gitpod.slack.com/archives/C03E52788SU/p1662051454479299

I added some questions to the PR, thanks a lot for putting this together, @utam0k !

kylos101 · 2022-09-01T15:21:55Z

.werft/workspace-run-integration-tests.sh

@@ -137,6 +137,8 @@ args=()
 args+=( "-kubeconfig=/home/gitpod/.kube/config" )
 args+=( "-namespace=default" )
 [[ "$USERNAME" != "" ]] && args+=( "-username=$USERNAME" )
+args+=( "-timeout=60m" )


Is this enough time? I ask because the werft job ran for 67 minutes.
https://werft.gitpod-dev.com/job/gitpod-custom-to-stablization-test.17

Yes, it is, because these args pass each test component.

kylos101 · 2022-09-01T15:24:53Z

.werft/workspace-run-integration-tests.sh

@@ -137,6 +137,8 @@ args=()
 args+=( "-kubeconfig=/home/gitpod/.kube/config" )
 args+=( "-namespace=default" )
 [[ "$USERNAME" != "" ]] && args+=( "-username=$USERNAME" )
+args+=( "-timeout=60m" )
+args+=( "-p=1" )


Is this to run each test binary in parallel?

No, opposite. this makes the test serial.

kylos101 · 2022-09-01T15:48:38Z

test/pkg/integration/integration.go

 	select {
 	case err := <-execErrs:
 		if err != nil {
 			return nil, closer, err
 		}
 		return nil, closer, fmt.Errorf("agent stopped unexepectedly")
-	case <-time.After(1 * time.Second):
+	case <-time.After(30 * time.Second):


My understanding is, this is the wait time in-between test binaries. Yes? In other words, this is not the wait time each individual test that we run.

I accused this time means waiting time for the start-up of the agent

utam0k · 2022-09-02T00:10:22Z

FYI https://werft.gitpod-dev.com/job/gitpod-custom-to-stablization-test.18

@jenting @kylos101
This PR improves stabilization, not complete fixing the integration test. I have already found the below issues

port-forwarding sometimes fails The flakiness of workspace integration test when port-forwarding #10671
sometimes hit [ws-manager] The container could not be located when the pod was terminated #12021 when trying to stop a workspace

jenting · 2022-09-02T00:49:07Z

FYI https://werft.gitpod-dev.com/job/gitpod-custom-to-stablization-test.18

@jenting @kylos101 This PR improves stabilization, not complete fixing the integration test. I have already found the below issues

port-forwarding sometimes fails The flakiness of workspace integration test when port-forwarding #10671

sometimes hit [ws-manager] The container could not be located when the pod was terminated #12021 when trying to stop a workspace

@utam0k I add the port-forwarding issue link 😉

utam0k · 2022-09-02T00:50:48Z

FYI https://werft.gitpod-dev.com/job/gitpod-custom-to-stablization-test.18

@jenting @kylos101 This PR improves stabilization, not complete fixing the integration test. I have already found the below issues

port-forwarding sometimes fails The flakiness of workspace integration test when port-forwarding #10671

sometimes hit [ws-manager] The container could not be located when the pod was terminated #12021 when trying to stop a workspace

@utam0k I add the port-forwarding issue link 😉

Oh, I missed it. But I'm trying to fix it on #12248 so that I will close your issue. Thanks

jenting · 2022-09-02T01:00:23Z

Thanks for your effort @utam0k

I thought all the integration test cases failed for the same reason #10671
But with this PR introduced changes, it seems no. And I summarize two reasons for the flaky integration test

we can't have multiple workspaces when running the integration test (I think we could support it, no?)
the timeout issue (probably due to preview env CPU/Memory/Disk are slow...)

utam0k · 2022-09-02T01:02:44Z

Thanks for your effort @utam0k

I thought all the integration test cases failed for the same reason #10671 But with this PR introduced changes, it seems no. And I summarize two reasons for the flaky integration test

we can't have multiple workspaces when running the integration test (I think we could support it, no?)

the timeout issue (probably due to preview env CPU/Memory/Disk are slow...)

Yes, I thought both were the same reason, lack of resources. Ideally, we can run all integration tests in parallel, but first I think we should focus on passing the integration; next, we can try it.

jenting

LGTM

Let's see how it improves the integration test robustness 💪

kylos101 · 2022-09-02T12:03:46Z

Thanks for your effort @utam0k
I thought all the integration test cases failed for the same reason #10671 But with this PR introduced changes, it seems no. And I summarize two reasons for the flaky integration test

we can't have multiple workspaces when running the integration test (I think we could support it, no?)

the timeout issue (probably due to preview env CPU/Memory/Disk are slow...)

Yes, I thought both were the same reason, lack of resources. Ideally, we can run all integration tests in parallel, but first I think we should focus on passing the integration; next, we can try it.

Indeed, @utam0k , great point. In other words, get tests to pass (so we have confidence), then make test runs faster (so we can have shorter feedback loops). Some related thoughts.

utam0k added 2 commits August 31, 2022 10:43

tests: remove unnecessary codes.

2368146

tests: Stop and Wait workspace at the end of each tests.

85ebebd

roboquat added the release-note-none label Sep 1, 2022

utam0k requested a review from a team September 1, 2022 02:51

roboquat added the size/XL label Sep 1, 2022

github-actions bot added the team: workspace Issue belongs to the Workspace team label Sep 1, 2022

utam0k force-pushed the to/stablization-test branch 4 times, most recently from 6c3a9de to 35b08fa Compare September 1, 2022 09:18

tests: expand timeout to 20m from 10m

dc026d0

utam0k force-pushed the to/stablization-test branch from 35b08fa to 5c4e0d7 Compare September 1, 2022 10:43

test: retry if get a error of EOF.

f3d954e

utam0k force-pushed the to/stablization-test branch from 5c4e0d7 to f3d954e Compare September 1, 2022 11:49

kylos101 reviewed Sep 1, 2022

View reviewed changes

jenting approved these changes Sep 2, 2022

View reviewed changes

roboquat merged commit f04a405 into main Sep 2, 2022

roboquat deleted the to/stablization-test branch September 2, 2022 01:25

roboquat added deployed: workspace Workspace team change is running in production deployed Change is completely running in production labels Sep 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests: Stop and Wait for workspace at the end of each tests #12572

tests: Stop and Wait for workspace at the end of each tests #12572

utam0k commented Sep 1, 2022

jenting commented Sep 1, 2022

kylos101 left a comment

kylos101 Sep 1, 2022

utam0k Sep 1, 2022

kylos101 Sep 1, 2022

utam0k Sep 1, 2022

kylos101 Sep 1, 2022

utam0k Sep 2, 2022

utam0k commented Sep 2, 2022 •

edited by jenting

Loading

jenting commented Sep 2, 2022 •

edited by werft-gitpod-dev-com bot

Loading

utam0k commented Sep 2, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jenting commented Sep 2, 2022

utam0k commented Sep 2, 2022

jenting left a comment

kylos101 commented Sep 2, 2022

tests: Stop and Wait for workspace at the end of each tests #12572

tests: Stop and Wait for workspace at the end of each tests #12572

Conversation

utam0k commented Sep 1, 2022

Description

Related Issue(s)

How to test

Release Notes

Documentation

Werft options:

jenting commented Sep 1, 2022

kylos101 left a comment

Choose a reason for hiding this comment

kylos101 Sep 1, 2022

Choose a reason for hiding this comment

utam0k Sep 1, 2022

Choose a reason for hiding this comment

kylos101 Sep 1, 2022

Choose a reason for hiding this comment

utam0k Sep 1, 2022

Choose a reason for hiding this comment

kylos101 Sep 1, 2022

Choose a reason for hiding this comment

utam0k Sep 2, 2022

Choose a reason for hiding this comment

utam0k commented Sep 2, 2022 • edited by jenting Loading

jenting commented Sep 2, 2022 • edited by werft-gitpod-dev-com bot Loading

utam0k commented Sep 2, 2022 • edited by werft-gitpod-dev-com bot Loading

jenting commented Sep 2, 2022

utam0k commented Sep 2, 2022

jenting left a comment

Choose a reason for hiding this comment

kylos101 commented Sep 2, 2022

utam0k commented Sep 2, 2022 •

edited by jenting

Loading

jenting commented Sep 2, 2022 •

edited by werft-gitpod-dev-com bot

Loading

utam0k commented Sep 2, 2022 •

edited by werft-gitpod-dev-com bot

Loading