Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable code coverage properly in Github Actions #977

Closed
Wauplin opened this issue Aug 9, 2022 · 2 comments
Closed

Enable code coverage properly in Github Actions #977

Wauplin opened this issue Aug 9, 2022 · 2 comments
Assignees

Comments

@Wauplin
Copy link
Contributor

Wauplin commented Aug 9, 2022

Following up on PR #976 that disables the coverage in tests.

It would be good to re-enable and fix it properly. Potential fixes have been mentioned here (internal url) and here (internal url).

Originally from @osanseviero :

The coverage tool we use is currently not working since it's actually looking at the coverage of the tests (i.e. if the tests are tested). The initial solution was simple (change --cov to --cov=src), but the way we setup our tests is that we don't do relative imports of the code under huggingface_hub, but rather we install huggingface_hub and then run the tests. That means that we're really interested in the coverage of the locally installed huggingface_hub,
Making it work is somewhat possible but starts to be hacky (we need to set up .coveragerc pointing to huggingface_hub), but then pytest will look at the coverage of all the files even if we're only interested in a group of files (e.g. we have independent test coverage runs for repository, hub mixin, keras, fastai), so the approach above would not work very nice. I also face the issue that it seems codecov might not share the report for source_package). Locally you can try

pytest --cov=.github --cov-config=.coveragerc  tests/test_utils_sha.py

and it works ok, but again it gives the coverage for all the files.

and from @adrinjalali :

In this repo: https://github.com/skops-dev/skops/blob/main/.github/workflows/build-test.yml
I install the library, then test, and in tests imports are not relative, and the coverage looks quite okay, example: https://app.codecov.io/gh/skops-dev/skops/compare/18/diff
I'm not sure why on huggingface_hub we have these issues.
I've also set the codecov config to wait for all the 12 CI runs before reporting anything, we could do the same for huggingface_hub.

Finally, an example of misleading codecov intervention: https://github.com/huggingface/huggingface_hub/pull/904/files

@Wauplin
Copy link
Contributor Author

Wauplin commented Aug 11, 2022

Mentioning this attempt from osanseviero #925 just in case (closed without merging).

@Wauplin
Copy link
Contributor Author

Wauplin commented Aug 16, 2022

Done in #992.

@Wauplin Wauplin closed this as completed Aug 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant