Encourage users to install the Codecov Github App on PR comment

[rohan-at-sentry]

Problem to solve

Description

Today, Codecov depends on using the Codecov Github App to post PR comments. The Github App uses an existing user account (or a new user account) created by the repo/org admin to interact with Github's API (using the user account) that enables uploads, PR comments etc. The app accomplishes this using the user's OAuth Token

If the Github App is not set up, Codecov tries its best to fulfil the workflow, but falls back to using Codecov's own OAuth Token. This is something that is shared across all Codecov users (i.e. 1 set of tokens for all users of Codecov) and therefore we run into situations where uploads, PR comments etc fail because they hit rate limits.

Giving proactive messaging to a user or admin that they need to install the Codecov Github App will help reduce frustration for users when they encounter issues (which are made worse by the fact that we don't clearly indicate when users run into rate limit issues )

Steps to Reproduce / Current UX

• Go to any PR on a repo that doesn't have the Codecov App set (you should see a comment from codecov-commenter ). For example

  • Upload test run results to codecov maplibre/maplibre-gl-js#4014

• Compare that to the comment posted by Codecov (which is the Codecov App)

  • Bundle Analysis: Add to timeseries MeasurementName enum worker#471 (comment)

Possible solution

• All future PR comments from codecov-commenter needs to have a message that encourages setting up of the Codecov App on the repo.

  Something like

  ⚠️ Please set up the [Codecov App
  ](https://github.com/apps/codecov) to ensure uploads and comments are reliably processed by Codecov. 
  

• Change the icon of the codecov-commenter bot to something that obviously indicates that the user needs to take action

Action Checklist for Issue Creation

• Add the design project (to the right under "projects" > settings icon > select "design team")
• Share in relevant Slack channel (#discuss-codecov-product-design) for visiblity
• Add targeted launch date/delivery date as/if needed

Additional Information

Figma: link

[Adal3n3]

@rohan-at-sentry

1. Are we able to push a Codecov PR comment before users even installed their Codecov Github App?
2. How do we know where to push so the same user will see the message?

We know in average developers push a PR can be from 7 times in a day to within 2 weeks. Maybe this is ok for now.

[rohan-at-sentry]

Re 1. Are we able to push a Codecov PR comment before users even installed their Codecov Github App?

Yeah this is exactly what is described in

If the Github App is not set up, Codecov tries its best to fulfil the workflow, but falls back to using Codecov's own OAuth Token. This is something that is shared across all Codecov users (i.e. 1 set of tokens for all users of Codecov) and therefore we run into situations where uploads, PR comments etc fail because they hit rate limits.

See the linked PR (maplibre/maplibre-gl-js#4014) - comment without app being set up.

Note : It's likely that this is legacy behavior that has carried over (meaning new orgs will likely not work at all unless an app is set up)

2. How do we know wehre to push so the same user will see the message.

Especially relevant for open source, you want it to be on the PR so that reviewers and maintainers can see.

[Adal3n3]

If the Github App is not set up, Codecov tries its best to fulfill the workflow, but falls back to using Codecov's own OAuth Token.

@rohan-at-sentry I don't follow this description, can you be more specific and elaborate the user flow? like if the Github App is not set up, where would customer find this PR comment that encourage them to install the Codecov App?

[Adal3n3]

@rohan-at-sentry @aj-codecov @codecovdesign @thomasrockhu-codecov Please review and here's the design on figma: link

You can find the markdown here and export the img files from Figma or just ping me.

[Adal3n3]

@thomasrockhu-codecov can you assign this issue? Terima Kasih 🙏

[thomasrockhu-codecov]

@michelletran-codecov also should be a pretty quick change, not 100% sure if we will need infra to host the image ourselves, or to keep it as an asset on GitHub.

[Adal3n3]

@thomasrockhu-codecov @michelletran-codecov Thanks! I will follow your decision - whatever works :)

When customers receive GH email from Codecov comment, can we change the asset name "Group 776. svg" to "Codecov App"?

[michelletran-codecov]

I think the GitHub assets link includes a bunch of processing on the image (for security and tracking purposes) and causes the email issue as seen by @Adal3n3 . So, the only way to get it to work in email would be to host it ourselves. @thomasrockhu-codecov Can we also get some support with hosting this image from Infra? (I'm not sure if we want to add it to GCS via Terraform, or I've found that even our wordpress images work: https://about.codecov.io/wp-content/themes/codecov/assets/brand/logos/codecov.svg was sent fine through email).

[thomasrockhu-codecov]

@michelletran-codecov yup, will look into it today/tomorrow

[thomasrockhu-codecov]

@Adal3n3 can you confirm specifically which image here is the one to use?

[Adal3n3]

@thomasrockhu-codecov, @michelletran-codecov and I chatted on slack and we are not going to host the image anymore because it adds a lot more work. To solve the image file name problem on GH email, I just need to re-upload the file with the corrected name and I did that already, so we are going to use this img without hosting it.

[michelletran-codecov]

Also, just adding that the image won't show up, but we'll have text that looks like:

[rohan-at-sentry]

That text looks weird... Can we simply just have a text on the comment (and therefore the email) to sidestep this? Or am I misunderstanding the behavior

[michelletran-codecov]

That text looks weird... Can we simply just have a text on the comment (and therefore the email) to sidestep this? Or am I misunderstanding the behavior

I agree that it looks weird. As I understand it, @Adal3n3 's suggestion to add the image is to make it stand out a bit more. However, the GitHub images will not show up on email (because of the way that GitHub's image processing work). So, the alternative would be to host the images ourselves. The options that I see are as follows:

1. don't have an image. The downside is that it might not stand out enough.
2. use the image hosted via GitHub assets. The downside is that it won't show up properly in email (will be fine in the comment)
3. use the image hosted ourselves so that the images show up properly in email. The downside is that we'll likely need to also set up a CDN to front the asset (so more infra work).

FWIW, my preference is for the first option (and maybe supplement with emojis to make it stand out). But I will defer to @Adal3n3 for what experience we would like!

[Adal3n3]

@rohan-at-sentry @michelletran-codecov The concern I have with using text to make the call to action is it will be simple gets ignore, people usually don't read, they just scan on what they care then move on, and the warning emoji is a bit annoying as nothing urgent. If the text on the email is weird, I can fix the file name. Maybe use "Codecov_app.svg", wdyt?

Here is a piece of study that show how image can grab attention.

[Adal3n3]

@michelletran-codecov @rohan-at-sentry This issue is closed but the codecov-commenter profile pic hasn't updated yet so I DM-ed @eliatcodecov to upload the new profile pic but there's an access issue. Will need to check with @eliatcodecov later on. No need to reopen this issue.