Config option to show/hide a "bundle analysis is working message" when no bundle changes occur.

[Adal3n3]

Problem to solve

Description

According to @nicholas-codecov this message is working as intended, so it's not a bug. We will receive this message if they run their CI and run a build step. The problem is that developers will be annoyed by receiving this unnecessarily/non-actionable message "Bundle size has no change ✅" on their PR comment, especially for those who push 15 times PRs in a day.

Slack 🧵: https://sentry.slack.com/archives/C060Y7J8420/p1711645978145129
Example: getsentry/sentry#67874 (comment)

Possible solution

See comment at the bottom of this issue

[Adal3n3]

@trent-codecov @rohan-at-sentry Can you prioritize this?

[eliatcodecov]

I've been reviewing this issue, and while I agree with the idea that not everyone wants to see a PR comment when "nothing changes", I really think this should be a configuration level change. I'll lay out some justification for why I feel this way:

1. Codecov's PR comment has this same behavior. If you make a PR that doesn't impact coverage, we still tell you that coverage isn't impacted. So I think "no changes, everything's fine" PR comment has some precedence from our other features
2. More importantly, though, as an engineer if I'm interacting with a service (in this case by sending a bundle) and I receive no response from the service, my first expectation is that it is flaky or down.
3. Depending on context, it's not necessarily intuitive to me that the bundle size didn't change:
   • In some cases, the fact that the bundle didn't change is really easy to infer. In Sentry's case for example, if I have a PR that only changes the backend, any PR comment about bundle size is probably useless to me. There's no way I could've impacted the bundle to begin with.
   • What about cases where it isn't so easy to infer that my bundle shouldn't have changed? For example, I've made a net zero change to my bundle size but I've still made changes. In this case it would definitely help me to receive the PR comment in question.

So I think this PR comment is conditionally helpful, and when something is conditionally helpful it's typically worth leveraging configuration to let users decide if they want it or not. I'm not even super picky about what the default is, but I do think this needs to be a configuration setting.

Can we instead tackle this problem from a configuration approach?

[aj-codecov]

Agreed with Eli in this case, I think we can use the same yaml config that we have used for Coverage to only show when there's a change:

comment:
require_changes: true

Additionally I think we can alter the language of the no change message that we're currently showing to specify why it's there - ex. "Your most recent bundler run showed no change in bundle size ✅ "
Doesn't have to be that, but something similar would be positive to better contextualize the message.

[rohan-at-sentry]

^ I think that's a good approach to consider. Throwing my 🎩 into this ring, there's feedback on only showing PR comments when "a thing" has materially changed - materially being config based.

Here's a link to that for coverage - #1350

There's an opportunity to take a first principles thinking approach into how we do comments overall

I think (in the order of weeks) I'd like for us to solve this specific BA issue - since this is PR related, we can tackle that on the platform team..

We can

• leverage a yaml config and default to false, AND
• update copy to show a comment if there were no bundle size changes (in order to mitigate the "is this working" concern)

and revisit more fundamental approach to all this later with @Adal3n3 and @codecovdesign

[rohan-at-sentry]

Summarizing above convo and setting out path forward

• Add a new config to the yaml
  • require_bundle_changes : true / false (false is default)
• Should be the under the comment config (which is used to modify the behavior of the coverage PR comment). Therefore, the following should be a valid config

comment:
require_changes:true
require_bundle_changes:true

IF require_bundle_changes is false, on commits/PRs where bundle size has not changed, then users see

This is the behavior today, and is going to be the default moving forward.

IF require_bundle_changes is true, on commits/PRs where bundle size has not changed, then users don't see the bundle PR comment from Codecov

Add documentation as necessary under https://docs.codecov.com/docs/javascript-bundle-analysis