E2E test steps should exit with non zero exit code if test fails

[jlewi]

In our E2E workflows, the steps that run the E2E tests e.g. test_runner should exit with non-zero exit code
if the tests don't pass.

This way test failures will be immediately evident from the Argo UI.

I think I originally didn't exit with non-zero exit code because I wasn't sure whether this would cause the workflow to fail immediately or only prevent dependent steps from running.

Either way I think its better to make the test failures more obvious.

This will be very useful for our release pipelines because we won't be dependent on prow to show job status.

[zjj2wry]

@jlewi test/e2e/main.go Is it still executed in ci? it seems can not run in local. I think we can use ginkgo to add some e2e tests, and ginkgo is heavily used in kubernetes. more infor https://github.com/onsi/ginkgo

[jlewi]

We should remove e2e/main.go from CI. We currently rely on Argo to write E2E tests.

If we need to use gingko as well we can but its not clear to my what problem that would solve.

[gaocegege]

ginkgo is a BDD style test framework and it makes the e2e test easy to understand and maintain, IMO.

[jlewi]

But its not distributed like Argo workflows which is one of the reasons I picked Argo over Gingko for our E2E tests.

[gaocegege]

Yeah, I agree.

I think what @zjj2wry means is that we could use ginkgo to implement e2e test cases. Argo and ginkgo have no conflict and we could use them together.

[jlewi]

/cc @ankushagarwal

[ankushagarwal]

@gaocegege @zjj2wry ginkgo looks like a good framework for writing E2E tests, but right now we have all of our test code in python modules which are run by argo workflows. If we were to move to ginkgo, it would require non-trivial effort to move the ~15-20 tests that we have in python. I am not sure if we have the capacity to do that right now.

I am going to go ahead and close this issue because the original concern is no longer an issue. The process running the tests does not have to exit with a non-zero code to indicate a test failure. The junit xml files generated have all the information required to understand the failure. With the addition of test_helper.py, we can see all the failure logs in gubernator ui without relying on argo workflow ui.