Include test suite in code coverage #870
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Rationale: - Tests are code. - Identify tests that aren’t running. - Ensure that test helpers are used and tested. Caveats: - Coverage may require extra configuration if tests are deliberately excluded. - Tests may clutter the coverage report; use `--skip-covered` for this. - The coverage metric generally goes up. https://nedbatchelder.com/blog/202008/you_should_include_your_tests_in_coverage.html
By default, coverage data contains absolute paths for the test suite. During CI, coverage data from various runners is combined in an environment where these paths do not exist. This results in errors like the following: No source for code: '/Users/runner/work/cookiecutter-hypermodern-python-instance/cookiecutter-hypermodern-python-instance/tests/__init__.py'. Aborting report output, consider using -i. The `coverage.paths` setting does not help with the test suite: Unlike the code under `src`, the test suite is not installed to `site-packages`, so we'd need to either hardcode paths from individual runners, or use overly generic wildcards. Instead, `coverage.run.relative_files` can be used to store the paths relative to the repository. We still need `coverage.paths` for `src` though, because these file paths still include the path into the various Nox environments.
cjolowicz
force-pushed
the
retrocookie-pr/coverage-for-tests
branch
from
affed1b
to
eb31214
Compare
|
Converted to draft, because for generated projects this fails locally and in CI, see cjolowicz/cookiecutter-hypermodern-python-instance#545 . Looks like this needs some more time to figure out--a minimal repro and digging into Coverage.py's handling of Job log: |
|
This appears to have been fixed in Coverage 6.1.2, nedbat/coveragepy#1147. |
|
The coverage job still fails: EDIT: Locally the coverage session succeeds when running |
|
This is apparently triggered when coverage is reported on POSIX for coverage data generated on Windows, due to platform-dependent path separators. The following was run on macOS using the coverage-data artifact from GA: |
|
The coverage session succeeds if the coverage data from Windows is removed prior to |
|
Superceded by 5c5e256 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #792
Include the test suite in code coverage, via the coverage.run.source setting.
Rationale:
Caveats:
--skip-coveredfor this.https://nedbatchelder.com/blog/202008/you_should_include_your_tests_in_coverage.html
This PR also enables the coverage.run.relative_files setting, to store paths relative to the repository in the coverage data files. By default, coverage data contains absolute paths for the test suite. During CI, coverage data from various runners is combined in an environment where these paths do not exist. This results in errors like the following:
The coverage.paths setting does not help with the test suite: Unlike the code under
src, the test suite is not installed tosite-packages, so we'd need to either hardcode paths from individual runners, or use overly generic wildcards. The setting is still needed forsrcthough, because even the relative paths include the location of the various Nox environments.