Disable codecov github annotations in the org-level yaml

[jakkdl]

I've been bothered by codecov's github annotations for a long while, where it fairly regularly kept complaining about previously-uncovered code still being uncovered after later commits fixing it. The latest instance of this happened over in pytest but I know it's confused me and others at various points in different trio repositories. (happened in #2812)

When I started digging I found out this functionality is deprecated and has a big red box in the documentation https://docs.codecov.com/docs/github-checks, and I'm fairly certain that more time has been lost collectively from false alarms (and code being near-unreadable in the review window when it's very low on coverage) than from the marginal gain of seeing it in the code review window rather than elsewhere. We'll still have the CI check and everything, and you can easily click through to codecov.io in several places.

I've opened an issue upstream codecov/codecov-action#1710, but in the meantime I think we should edit the YAML at https://app.codecov.io/account/github/python-trio/yaml/

It's a one-line fix, that doesn't even need a PR, but thought I should open it to discussion and make people aware of the change so they don't wonder where it went. And if you personally want the functionality there's a browser extension: https://docs.codecov.com/docs/the-codecov-browser-extension

[webknjaz]

@jakkdl in my experience, one thing that causes Codecov to post statuses prematurely (not just annotations but Checks, comments and their web UI) is allowing it to guess the moment when it's okay to start processing the report files and combining them into one.
I think this is because when there's a matrix of jobs not covering the same lines each, there would be portions of lines for which coverage might end up being reported much later than for many other jobs. And if Codecov decides to process the reports earlier than that, it usually reports lower coverage than it really is. It then updates these statuses later, when that slow job coverage arrives, though. The problem is that the annotations are immutable IIRC and so they can't update them, while the comment and the checks would be.

To fight this, I like setting the config as follows, setting the number of expected uploads explicitly: https://github.com/aio-libs/yarl/blob/f304dd7/.codecov.yml#L3-L8.

[jakkdl]

To fight this, I like setting the config as follows, setting the number of expected uploads explicitly: https://github.com/aio-libs/yarl/blob/f304dd7/.codecov.yml#L3-L8.

ooh, that'd resolve #2689 that has been bugging me forever. Thanks!

But I don't think it'd resolve the problems in the pytest PR where the uncovered code was introduced in one commit, and the tests adding coverage added in a later commit.

The problem is that the annotations are immutable IIRC and so they can't update them, while the comment and the checks would be.

yeah the codecov docs say

The GitHub API does not provide a way to clear annotations once they are sent. Even if subsequent reports processed by Codecov include updated coverage data, previous annotations remain.

[webknjaz]

Yep, that assessment is correct.

[jakkdl]

I've now edited the org-level yaml, could leave this issue open for a bit and wait for impact.

Opened #3156 for the other repo-level improvements