The error logs for different tests run with Go 1.14 are interleaved in junit xml files

[chizhg]

Go 1.14 has a new feature that "go test -v now streams t.Log output as it happens, rather than at the end of all tests."

This make go-junit-report that we use to generate junit xml files unhappy, which can possibly show the error messages for one test under another one on Spyglass.

See one example in https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/knative_eventing/3034/pull-knative-eventing-integration-tests/1263214476346265603. If you expand one test like TestChannelChain/Channel-messaging.knative.dev/v1alpha1, you can see the error messages contain messages from other tests.

We should either fix go-junit-report or check if switching to another tool like gotestsum can fix this issue.

[chaodaiG]

/assign

[chaodaiG]

The example above has:

TestChannelChain/Channel-messaging.knative.dev/v1alpha1: test_runner.go:99: namespace is : "test-channel-chain-channel-messaging-knative-dev-v1alpha1-5t5ff" 

TestApiServerSourceV1Alpha2EventTypes: tracker.go:130: Cleaning resource: "e2e-apiserver-source-eventtypes" TestApiServerSourceV1Alpha2EventTypes: tracker.go:136: Waiting for e2e-apiserver-source-eventtypes to be deleted TestApiServerSourceV1Alpha2EventTypes: tracker.go:130: Cleaning resource: "event-watcher-sa-event-watcher-r" 

TestApiServerSourceV1Alpha2EventTypes: tracker.go:136: Waiting for event-watcher-sa-event-watcher-r to be deleted TestApiServerSourceV1Alpha2EventTypes: tracker.go:130: Cleaning resource: "event-watcher-r" TestApiServerSourceV1Alpha2EventTypes: tracker.go:136: Waiting for event-watcher-r to be deleted TestApiServerSourceV1Alpha2EventTypes: tracker.go:130: 

Cleaning resource: "event-watcher-sa" TestApiServerSourceV1Alpha2EventTypes: tracker.go:136: Waiting for event-watcher-sa to be deleted

I was an old pull request that's not easy to repro, this also happens in latest serving test:
https://prow.knative.dev/view/gcs/knative-prow/logs/ci-knative-serving-continuous/1269932327077679104

[chaodaiG]

Tested with latest gotestsum, didn't help:

knative/serving#8274
https://prow.knative.dev/view/gcs/knative-prow/pr-logs/pull/knative_serving/8274/pull-knative-serving-integration-tests/1270397344797429763

[chaodaiG]

gotestsum community seems active, created a bug there: gotestyourself/gotestsum#128

[dnephin]

This is an issue with go1.14, see golang/go#38458. It sounds like that issue was fixed recently, so it should be resolved for go1.15.

[chizhg]

Thanks for the pointer @dnephin !
It's likely that they are planning to backport this fix to Go 1.14 - golang/go#39308

[chizhg]

The issue has been fixed after updating Go to 1.14.6, see https://prow.knative.dev/view/gcs/knative-prow/logs/ci-knative-serving-gloo-0.17.1/1285367134993518592