Seeking Guidance on Using Codecov to Transformers Repository

[yunbolyu]

Feature request

Hello Transformers maintainers and community,

In an effort to use the Codecov module and obtain code coverage, I attempted to base my changes on the previous removal process. However, I've encountered challenges since the tests appear to have been divided into multiple segments PR #7970. I also came across PR #8093, which suggests merging two slow tests (TF and PT) for coverage. While this seems to be a lead, I'm uncertain about the actual implementation.

Motivation

I am currently conducting an empirical study focusing on code coverage in deep-learning open-source projects. While trying to use Codecov for assessing the code coverage of Transformers, I observed that Codecov was previously utilized by the project. However, as per PR #7970, due to the workflow refactoring and the diminishing utility of Codecov reports, the Codecov module was removed.

Your contribution

Could anyone provide insights or suggestions on how best to use Codecov and achieve correct code coverage for Transformers?

Thank you in advance for your assistance and guidance.

[ArthurZucker]

Pinging @LysandreJik as he was here during the good old times when Codecov was used!

[yunbolyu]

Thanks for pinging out. I've seen him a lot in previous discussions and appreciate his ongoing contributions to the open-source community and your timely response!

[LysandreJik]

Hey @yunbolyu! Setting up codecov right now for transformers would likely be a significant engineering endeavor given the different flavors of tests that we support. We rarely run the entire test suite at once on all supported hardware, so making sense of the different workflows would likely be tough for codecov.

@ydshieh, who's managing our testing infra can probably share a bit more than I can.

[ydshieh]

Hi @yunbolyu As Lysandre said, the CI only runs a subset of tests (and would be different subsets in each run, except for the nightly CI run). Furthermore, I never used Codecov before and don't have useful insights to help you.

A quick look tell me we need the following (but I think you know this already) in each CircleCI job

run: pip install codecov pytest-cov
run: pytest ... `--cov`
run: codecov  

But I don't know what problems you encountered here. If you mean multiple reports presented, it's probably better to treat them as different CI and report them (well, this probably doesn't make sense...), or you will have to manually to merge those reports.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.