Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dump junit xml #1634

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Dump junit xml #1634

wants to merge 9 commits into from
+166 −8

Conversation

dandavison
Copy link
Contributor

@dandavison dandavison commented Sep 12, 2024 •
edited
Loading

Closes #1633

As in temporalio/sdk-python#617 and temporalio/cli#650, I propose to modify the test suite to output junit XML. The motivation for this is that, given appropriate tools, it will make it easier for us to study patterns of flakiness, test timings etc, across matrix builds within a run, and across runs. Producing the files now will allow us to test tools for doing this.

@dandavison dandavison force-pushed the junit-xml branch 2 times, most recently from 64e0a30 to 861daa9 Compare September 12, 2024 10:08
@dandavison dandavison marked this pull request as ready for review September 12, 2024 13:49
@dandavison dandavison requested a review from a team as a code owner September 12, 2024 13:49
@dandavison dandavison marked this pull request as draft September 12, 2024 14:02
dandavison
dandavison commented Sep 12, 2024
View reviewed changes
.github/workflows/ci.yml Outdated Show resolved Hide resolved
dandavison
dandavison commented Sep 12, 2024
View reviewed changes
internal/cmd/build/go.sum
@@ -3,18 +3,27 @@ github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03
github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c h1:pxW6RcqyfI9/kWtOwnv/G+AzdKuy2ZrqINhenH4HyNs=
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All changes to go.mod and go.sum are the result of running go get gotest.tools/gotestsum (looks like it was already a transitive dependency IIUC).

@dandavison dandavison marked this pull request as ready for review September 12, 2024 14:51
yuandrew
yuandrew approved these changes Sep 12, 2024
View reviewed changes
.github/workflows/ci.yml Outdated Show resolved Hide resolved
@dandavison
Copy link
Contributor Author

@Quinn-With-Two-Ns To output junit XML I had to switch to using gotestsum. One question here is are you OK with the new test output format from gotestsum? From a quick look it appears that one big difference is that output from successful tests is hidden by default. I was optimistic that we'd be OK with this switch since https://github.com/temporalio/temporal uses it, and it looks like it's quite configurable, and fairly widely used by high profile Go projects: https://github.com/gotestyourself/gotestsum

@dandavison dandavison mentioned this pull request Sep 14, 2024
Merged
@Quinn-With-Two-Ns
Copy link
Contributor

No concern about the test format. Question about the general PR though, from the issue it says

The motivation for this is that, given appropriate tools, it will make it easier for us to study patterns of flakiness

Does that appropriate tool exist yet today?

@dandavison
Copy link
Contributor Author

Does that appropriate tool exist yet today?

Good question. I couldn't find something that would work well, e.g. correlating tests across our matrix builds, so I've started creating our own from scratch. I'm not quite ready to have others test it (but absolutely welcome any help developing it).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yuandrew yuandrew yuandrew approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Modify test suite to dump junit XML
3 participants
@dandavison @Quinn-With-Two-Ns @yuandrew