Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Codecov can generate inconsistent coverage reports #2972

Closed
GCHQDeveloper314 opened this issue Jun 8, 2023 · 0 comments · Fixed by #2974
Closed

Codecov can generate inconsistent coverage reports #2972

GCHQDeveloper314 opened this issue Jun 8, 2023 · 0 comments · Fixed by #2974
Assignees
@GCHQDeveloper314
Labels
bug Confirmed or suspected bug
Milestone
v2.1.0

Comments

@GCHQDeveloper314
Copy link
Member

Describe the bug
There are currently two situations where the coverage reports codecov generates are incorrect:

  • When the CI runs coverage reports are generated in 7 different matrix jobs. These do not complete at the same time which results in an initial coverage report that's incorrect and is then updated as each report is uploaded, eventually becoming correct only if all jobs pass. In the interim the report is wrong and this is misleading.
  • When there is a problem with the codecov upload for one of the 7 matrix steps this can cause coverage to become wrong if the problem doesn't effect all of the steps. The coverages misses out part of the code and reports wildly higher/lower coverage. Currently there's no warning or error that the coverage didn't upload.

If coverage did not upload then the tests must all be rerun as the coverage upload is directly coupled to this job.

Desired behavior
The report should only be generated if all coverage information is available, and if the codecov upload fails due to an intermittent error, we should be able to retry just the upload without a need to rerun any tests.
@GCHQDeveloper314 GCHQDeveloper314 added the bug Confirmed or suspected bug label Jun 8, 2023
@GCHQDeveloper314 GCHQDeveloper314 added this to the v2.1.0 milestone Jun 8, 2023
@GCHQDeveloper314 GCHQDeveloper314 self-assigned this Jun 8, 2023
@GCHQDeveloper314 GCHQDeveloper314 linked a pull request Jun 9, 2023 that will close this issue
Gh-2972: Codecov can generate inconsistent coverage reports #2973
Closed
@GCHQDeveloper314 GCHQDeveloper314 removed a link to a pull request Jun 9, 2023
Gh-2972: Codecov can generate inconsistent coverage reports #2973
Closed
@GCHQDeveloper314 GCHQDeveloper314 linked a pull request Jun 9, 2023 that will close this issue
Gh-2972: Codecov can generate inconsistent coverage reports #2973
Closed
@GCHQDeveloper314 GCHQDeveloper314 removed a link to a pull request Jun 9, 2023
Gh-2972: Codecov can generate inconsistent coverage reports #2973
Closed
GCHQDeveloper314 added a commit that referenced this issue Jun 9, 2023
@GCHQDeveloper314
Merge branch 'develop' into gh-2972-codecov-inconsistent-report-fix
912b10b
@github-actions github-actions bot mentioned this issue Jun 9, 2023
Merged
GCHQDeveloper314 added a commit that referenced this issue Jun 9, 2023
@GCHQDeveloper314
Gh-2972: Codecov can generate inconsistent coverage reports (#2974)
e2e254f 
* Store reports in an artifact, upload once in a separate job
Compared to before this avoids making multiple uploads to codecov and allows the codecov upload to be reattempted with rerunning any tests.

* Fix Codecov 'unusable report' caused by missing source code
The source files are required to be present when uploading coverage - https://docs.codecov.com/docs/error-reference#unusable-reports

* Include reports for example
JaCoCo was disabled for these causing all reports to be missing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@GCHQDeveloper314 GCHQDeveloper314

Labels
bug Confirmed or suspected bug
Projects
None yet
Milestone
v2.1.0
Development

Successfully merging a pull request may close this issue.

Gh-2972: Codecov can generate inconsistent coverage reports gchq/Gaffer
1 participant
@GCHQDeveloper314