Only show comment when coverage/bundle is negatively changed (configurable)

[codecovdesign]

Problem to solve

Today, you can can configure the pr comment to only show when any changes occur (https://docs.codecov.com/docs/common-recipe-list#only-comment-on-coverage-changes), both negative or positive. However, there are some customer requests for the ability to see Codecov reports only when there's a decrease in coverage or negative changes. This contrasts to another common preferences to confirm / see the ✅ - consideration here is if there was a config to remove only for negative reporting, that it's opt-in and showing is default.

As related to bundle:

It would also be nice to set a threshold change to not trigger a comment. We use Vite for our builds and it seems there is often a 10-35kB variation between builds even without any substantive changes to the code or dependencies. Being able to only have the comment appear when the change matters would help to avoid people from learning to ignore it when it shows up every time.

Proposed Solutions

Areas to explore:

• YAML configuration update: introduce YAML options to display reports exclusively on coverage reduction.
  • not a default setting, but opt-in configuration
• Leverage existing configurations: explore combining current YAML settings like threshold with a hypothetical require_changes: true to meet this need.
• How this could work too on bundle analysis
• Common issue is around email updating, are there any ways we can make the update less noisy?
• Considerations around re-run scenarios
  • Do you still show a PR comment if coverage/bundle size had dropped on one commit and increased in a subsequent commit on the same PR
  • Opposite scenario : do you show a PR comment if coverage/bundle size had not dropped on one commit and dropped in a subsequent commit on the same PR
• For bundle: only show is X threshold is changed

Tasks

Preview Give feedback

1. Extend YAML configuration to handle "only notify on negative change" #1966
   +0
   
2. Extend YAML configuration with bundle changes and threshold #2018
   enhancement +1
   
3. Adjust coverage comment notification to new configuration #2003
   enhancement +1
   
4. Adjust Bundle Analysis comment notification to new coniguration #2021
   enhancement +1
   
5. Don't delete comment anymore if it exists but required_changes are not met #2023
   +0
   
6. Adjust documentation to address new options

[dabrahams]

I don't mind seeing an increase in the coverage in the PR itself; in fact I think it's a nice affirmation. But I really don't want to get an email unless something has changed for the worse.

[rohan-at-sentry]

philosophically related - #1485

[rohan-at-sentry]

Related - codecov/feedback#340

[giovanni-guidini]

UPDATE

I'm still working on this, but wanted to give a small update here.

• We just need to be careful creating the GitHub comment. Updates don't trigger email notifications (at least for me they don't…);
• Posting statuses or checks also doesn't seem to trigger email notifications 🥳
• There's no way to “silently” post a comment in GitHub (or GitLab);
• Improving the YAML configuration is the easiest way to improve this functionality. We should do that;

There are other details (and more avenues of exploration we might consider) in this investigation: https://l.codecov.dev/1DKPDc (codecov-only)

The final thing I'd like to say is that customers can very easily create a filter in the email client that will silence the emails from Codecov. This might still be useful in the future if you find the emails annoying but doesn't have the "power" in your team to disable them completely.

I did a filter like in the image below. It auto-deletes emails from the Codecov comment if the patch is 100% covered. Otherwise I still see the email.

[giovanni-guidini]

New configuration options have been released for both coverage and bundle analysis [beta] comments.

Check the updated documentation here:

• coverage - https://docs.codecov.com/docs/pull-request-comments#requiring-changes
• bundle analysis [beta] - https://docs.codecov.com/docs/javascript-bundle-analysis#main-features (scroll to "Bundle information via PR comment")