Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve sharness tests failure reports #8991

Closed
Jorropo opened this issue May 24, 2022 · 6 comments · Fixed by #9355
Closed

Improve sharness tests failure reports #8991

Jorropo opened this issue May 24, 2022 · 6 comments · Fixed by #9355
Assignees
Labels
need/triage Needs initial labeling and prioritization team/ipdx Notify IP Developer Experience team topic/ci Continuous integration

Comments

@Jorropo
Copy link
Contributor

Jorropo commented May 24, 2022

It's very unclear which test fail.
The error message is not in the failing test (which is just a grep failed).
You need to deep dive through logs, CTRL + F failed (which will show other unrelated failures often).

@Jorropo Jorropo added kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization topic/ci Continuous integration labels May 24, 2022
@Jorropo Jorropo changed the title sharness tests failure sucks sharness tests failure reports sucks May 24, 2022
@Stebalien
Copy link
Member

There are two different issues here:

  1. Sharness tests are a pain to write, maintain, etc. Especially if you don't know bash.
  2. Our current CI setup doesn't parse sharness tests.

That second point is much easier to fix. IIRC, we had a way to export junit result files, but we're not currently using that.

@Stebalien Stebalien removed the kind/bug A bug in existing code (including security flaws) label Jun 1, 2022
@galargh galargh added the team/ipdx Notify IP Developer Experience team label Sep 26, 2022
@galargh galargh self-assigned this Sep 26, 2022
@galargh
Copy link
Contributor

galargh commented Sep 26, 2022

It seems that the JUnit report we generate now are not valid but I might be able to do something about it.

@galargh
Copy link
Contributor

galargh commented Sep 30, 2022

Quick update:
I've been able to upgrade sharness, update our implementation of JUnit XML reports generation and add XML to HTML conversion. I've tested in on GitHub Actions and here are example reports that it spits out https://github.com/galorgh/kubo/actions/runs/3153345461 (note: errors = known breakages, failures = unknown breakages + fixed known breakages).

TODO:

  • test valid JUnit XML generation on CircleCI
  • update strings in html report to reflect how WE use JUnit (e.g. let's explicitly change Errors to Known breakages rather than leaving it up to the reader to know) edit: treating it as out of scope for this task; we can iterate on the reports later
  • make the GitHub Actions sharness workflow include results summary - https://github.com/marketplace/actions/test-reporter
  • figure out how to publish the HTML one-pager so that one doesn't have to download and unzip it
  • maybe (?) add failure type next to the not ok string edit: treating it as out of scope for this task

@BigLep BigLep changed the title sharness tests failure reports sucks Improve sharness tests failure reports Sep 30, 2022
@BigLep
Copy link
Contributor

BigLep commented Sep 30, 2022

test JUnit XML generation on CircleCI

@galargh : do we need to do this? If we are moved over to GitHub Actions this is no longer relevant right? Basically, if this is more than 30minutes I wouldn't spend time on it from what I understand.

@galargh
Copy link
Contributor

galargh commented Sep 30, 2022

This should take definitely less than 30 minutes. I think it only entails ensuring that xmllint is installed on the machine where we run sharness.

@galargh
Copy link
Contributor

galargh commented Oct 20, 2022

The new reporting proposed in #9355 will look like this:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
need/triage Needs initial labeling and prioritization team/ipdx Notify IP Developer Experience team topic/ci Continuous integration
Projects
No open projects
Archived in project
Development

Successfully merging a pull request may close this issue.

4 participants