Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce new ability to include flaky tests in summary #1115

Merged
merged 7 commits into from
Jun 21, 2024
Merged

Introduce new ability to include flaky tests in summary #1115

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b01f5b6
- add ability to include flaky tests in summary in its own table
mikepenz Jun 21, 2024
18021b5
- add flaky scenario to python test
mikepenz Jun 21, 2024
0a68922
- enable job summary for pytest report
mikepenz Jun 21, 2024
67874d4
- fix config
mikepenz Jun 21, 2024
d8e88a9
- update testcases for new meta data
mikepenz Jun 21, 2024
b2496c6
- add new test build for flaky handling
mikepenz Jun 21, 2024
744eaf9
- don't require annotate_notice anymore for the parser to parse passe…
mikepenz Jun 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,23 @@ jobs:
uses: ./
with:
check_name: Example Pytest Report
report_paths: test_results/python/report.xml
report_paths: test_results/python/report_flaky.xml
include_passed: true
detailed_summary: true
annotate_only: ${{ github.event_name == 'workflow_dispatch' }}

- name: Test JUnit flaky test import
uses: ./
with:
check_name: Example JUnit Flaky Report
report_paths: test_results/junit-web-test/expectedRetries.xml
include_passed: true
check_retries: true
job_summary: true
detailed_summary: true
flaky_summary: true
annotate_only: ${{ github.event_name == 'workflow_dispatch' }}

- name: Test Multi test import
uses: ./
with:
Expand Down
93 changes: 50 additions & 43 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,8 @@
- Lighweight
- Rich build log output

This action processes JUnit XML test reports on pull requests and shows the result as a PR check with summary and annotations.
This action processes JUnit XML test reports on pull requests and shows the result as a PR check with summary and
annotations.

Based on action for [Surefire Reports by ScaCap](https://github.com/ScaCap/action-surefire-report)

Expand Down Expand Up @@ -71,40 +72,41 @@ jobs:

### Inputs

| **Input** | **Description** |
|----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `report_paths` | Optional. [Glob](https://github.com/actions/toolkit/tree/master/packages/glob) expression to junit report paths. Defaults to: `**/junit-reports/TEST-*.xml`. |
| `token` | Optional. GitHub token for creating a check run. Set to `${{ github.token }}` by default. |
| `test_files_prefix` | Optional. Prepends the provided prefix to test file paths within the report when annotating on GitHub. |
| `exclude_sources` | Optional. Provide `,` seperated array of folders to ignore for source lookup. Defaults to: `/build/,/__pycache__/` |
| `check_name` | Optional. Check name to use when creating a check run. The default is `JUnit Test Report`. |
| `suite_regex` | Optional. Regular expression for the named test suites. E.g. `Test*` |
| `commit` | Optional. The commit SHA to update the status. This is useful when you run it with `workflow_run`. |
| `fail_on_failure` | Optional. Fail the build in case of a test failure. |
| `require_tests` | Optional. Fail if no test are found. |
| `require_passed_tests` | Optional. Fail if no passed test are found. (This is stricter than `require_tests`, which accepts skipped tests). |
| `include_passed` | Optional. By default the action will skip passed items for the annotations. Enable this flag to include them. |
| `check_retries` | Optional. If a testcase is retried, ignore the original failure. |
| `check_title_template` | Optional. Template to configure the title format. Placeholders: {{FILE_NAME}}, {{SUITE_NAME}}, {{TEST_NAME}}, {{CLASS_NAME}}. |
| `summary` | Optional. Additional text to summary output |
| `check_annotations` | Optional. Defines if the checks will include annotations. If disabled skips all annotations for the check. (This does not affect `annotate_only`, which uses no checks). |
| `update_check` | Optional. Uses an alternative API to update checks, use for cases with more than 50 annotations. Default: `false`. |
| `annotate_only` | Optional. Will only annotate the results on the files, won't create a check run. Defaults to `false`. |
| `transformers` | Optional. Array of `Transformer`s offering the ability to adjust the fileName. Defaults to: `[{"searchValue":"::","replaceValue":"/"}]` |
| `job_summary` | Optional. Enables the publishing of the job summary for the results. Defaults to `true`. May be required to disable [Enterprise Server](https://github.com/mikepenz/action-junit-report/issues/637) |
| `detailed_summary` | Optional. Include table with all test results in the summary. Defaults to `false`. |
| `annotate_notice` | Optional. Annotate passed test results along with warning/failed ones. Defaults to `false`. (Changed in v3.5.0) |
| `follow_symlink` | Optional. Enables to follow symlinks when searching test files via the globber. Defaults to `false`. |
| `job_name` | Optional. Specify the name of a check to update |
| `annotations_limit` | Optional. Specify the limit for annotations. This will also interrupt parsing all test-suites if the limit is reached. Defaults to: `No Limit`. |
| **Input** | **Description** |
|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `report_paths` | Optional. [Glob](https://github.com/actions/toolkit/tree/master/packages/glob) expression to junit report paths. Defaults to: `**/junit-reports/TEST-*.xml`. |
| `token` | Optional. GitHub token for creating a check run. Set to `${{ github.token }}` by default. |
| `test_files_prefix` | Optional. Prepends the provided prefix to test file paths within the report when annotating on GitHub. |
| `exclude_sources` | Optional. Provide `,` seperated array of folders to ignore for source lookup. Defaults to: `/build/,/__pycache__/` |
| `check_name` | Optional. Check name to use when creating a check run. The default is `JUnit Test Report`. |
| `suite_regex` | Optional. Regular expression for the named test suites. E.g. `Test*` |
| `commit` | Optional. The commit SHA to update the status. This is useful when you run it with `workflow_run`. |
| `fail_on_failure` | Optional. Fail the build in case of a test failure. |
| `require_tests` | Optional. Fail if no test are found. |
| `require_passed_tests` | Optional. Fail if no passed test are found. (This is stricter than `require_tests`, which accepts skipped tests). |
| `include_passed` | Optional. By default the action will skip passed items for the annotations. Enable this flag to include them. |
| `check_retries` | Optional. If a testcase is retried, ignore the original failure. |
| `check_title_template` | Optional. Template to configure the title format. Placeholders: {{FILE_NAME}}, {{SUITE_NAME}}, {{TEST_NAME}}, {{CLASS_NAME}}. |
| `summary` | Optional. Additional text to summary output |
| `check_annotations` | Optional. Defines if the checks will include annotations. If disabled skips all annotations for the check. (This does not affect `annotate_only`, which uses no checks). |
| `update_check` | Optional. Uses an alternative API to update checks, use for cases with more than 50 annotations. Default: `false`. |
| `annotate_only` | Optional. Will only annotate the results on the files, won't create a check run. Defaults to `false`. |
| `transformers` | Optional. Array of `Transformer`s offering the ability to adjust the fileName. Defaults to: `[{"searchValue":"::","replaceValue":"/"}]` |
| `job_summary` | Optional. Enables the publishing of the job summary for the results. Defaults to `true`. May be required to disable [Enterprise Server](https://github.com/mikepenz/action-junit-report/issues/637) |
| `detailed_summary` | Optional. Include table with all test results in the summary. Defaults to `false`. |
| `flaky_summary` | Optional. Include table with all falky results in the summary. Defaults to `false`. |
| `annotate_notice` | Optional. Annotate passed test results along with warning/failed ones. Defaults to `false`. (Changed in v3.5.0) |
| `follow_symlink` | Optional. Enables to follow symlinks when searching test files via the globber. Defaults to `false`. |
| `job_name` | Optional. Specify the name of a check to update |
| `annotations_limit` | Optional. Specify the limit for annotations. This will also interrupt parsing all test-suites if the limit is reached. Defaults to: `No Limit`. |

<details><summary><b>Common report_paths</b></summary>
<p>

- Surefire:
`**/target/surefire-reports/TEST-*.xml`
- Surefire:
`**/target/surefire-reports/TEST-*.xml`
- sbt:
`**/target/test-reports/*.xml`
`**/target/test-reports/*.xml`

</p>
</details>
Expand Down Expand Up @@ -143,16 +145,20 @@ After action execution it will return the test counts as output.

A full set list of possible output values for this action.

| **Output** | **Description** |
|-----------------------|----------------------------------------------------------------------------------------|
| `outputs.total` | The total number of test cases covered by this test-step. |
| `outputs.passed` | The number of passed test cases. |
| `outputs.skipped` | The number of skipped test cases. |
| `outputs.failed` | The number of failed test cases. |
| **Output** | **Description** |
|----------------------------|---------------------------------------------------------------------------------------------------------------------|
| `outputs.total` | The total number of test cases covered by this test-step. |
| `outputs.passed` | The number of passed test cases. |
| `outputs.skipped` | The number of skipped test cases. |
| `outputs.failed` | The number of failed test cases. |
| `outputs.summary` | The short summary of the junit report. In html format (as also constructed by GitHub for the summary). |
| `outputs.detailed_summary` | The full table with all test results in a summary. In html format (as also constructed by GitHub for the summary). |
| `outputs.flaky_summary` | The full table with all flaky results in a summary. In html format (as also constructed by GitHub for the summary). |

### PR run permissions

The action requires `write` permission on the checks. If the GA token is `read-only` (this is a repository configuration) please enable `write` permission via:
The action requires `write` permission on the checks. If the GA token is `read-only` (this is a repository
configuration) please enable `write` permission via:

```yml
permissions:
Expand All @@ -161,9 +167,10 @@ permissions:

Additionally for [security reasons], the github token used for `pull_request` workflows is [marked as read-only].
If you want to post checks to a PR from an external repository, you will need to use a separate workflow
which has a read/write token, or use a PAT with elevated permissions.
which has a read/write token, or use a PAT with elevated permissions.

[security reasons]: https://securitylab.github.com/research/github-actions-preventing-pwn-requests/

[marked as read-only]: https://docs.github.com/en/actions/security-guides/automatic-token-authentication#permissions-for-the-github_token

<details><summary><b>Example</b></summary>
Expand Down Expand Up @@ -195,9 +202,9 @@ jobs:
name: report
on:
workflow_run:
workflows: [build]
types: [completed]
workflows: [ build ]
types: [ completed ]

permissions:
checks: write

Expand All @@ -218,7 +225,7 @@ jobs:
report_paths: '**/build/test-results/test/TEST-*.xml'
```

This will securely post the check results from the privileged workflow onto the PR's checks report.
This will securely post the check results from the privileged workflow onto the PR's checks report.

</p>
</details>
Expand Down Expand Up @@ -261,7 +268,7 @@ Original idea and GitHub Actions by: https://github.com/ScaCap/action-surefire-r

## License

Copyright (C) 2023 Mike Penz
Copyright (C) 2024 Mike Penz

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand Down
Loading
Loading