Feedback for Codecov AI reviewer in beta

[Adal3n3]

We’re excited to introduce the Codecov AI Reviewer Assistant! It’s designed to help you review code changes and suggest improvements.

We’d love your feedback on: 🦻

• The ease of interacting with the AI assistant
• The usefulness of the AI's responses

This issue is intended to gather feedback on the beta feature.

[andihalberkamp]

Hi team, 👋

I recently integrated the Codecov AI Reviewer into this repository and requested an AI review on this PR using the command @codecov-ai-reviewer review. However, I haven't seen any response or feedback from the AI yet.

Could you please help me troubleshoot this? Here's what I’ve already checked:

• The Codecov AI app is installed and permissions seem correct.
• The command format was used as specified.
• Waited some time to ensure there wasn’t a delay.

If there's any configuration or additional setup I might have missed, please let me know. Thanks in advance for your support! 😊

Best regards,
Andi

[Adal3n3]

Hi @andihalberkamp Thanks for reaching out. It looks like you have set up the AI bot correctly. I’ve reached the team to investigate this issue further. Thanks for your patience.

[Adal3n3]

Hi @andihalberkamp The bot is probably working, but it may not response if it doesn't have any feedback on your code or nothing to review. Can you try making some meaningful changes and use the command @codecov-ai-reviewer review in your PR comment again. Thank you.

[reactive-firewall]

Note

Please don't panic, this feedback is on the beta (see timestamp), in hope it helps. 🙈

Feedback on my 14 days of daily use:

👍🏻 The GOOD

• Easy to setup; no complaints. ( 👍🏻 for limiting the needed permissions)
  

• Free access to an AI for open-source projects ( 👍 for giving back to the community )

• Documentation is easy to find, (and appropriately leads user here during the BETA)

TL;DR - Suggestion Expand the FAQ

This GH issue provides a feeling of support and communicates interest in Codecov's community right from the get-go. (again, I'm sure this is nothing new to your team)

Regarding the FAQ, a quick-win could be expanding the FAQ by answering these questions:

• What @ prompts does @codecov-ai-reviewer recognize?

  • e.g. review - trigger a PR review
  • e.g. help - list this usage guide as a reply
  • etc.

• What are the rate/usage/API limits of the bot?

  • e.g. set expectations for the beta (especially for open-source projects 🙉)
  • e.g. justify any throttling that is enforced (OpenAI backend probably is not cost-free to leverage 🙊)
  • e.g. C.M.A. statement 🤷 (e.g. we reserve desecration to stop evil)?

• Can review instructions be specified to the bot?

  • e.g. at this time the bot uses these instructions passed to openAI (rational: along with the mentioned data from the user already mentioned)

• e.g. "yes" and this is how ... (🌈 optimistic example)

• what about security?

• e.g. get ahead of this historically tricky question by setting expectations (the AI does have write access to the PR after all 🙈)
• e.g. see terms and policy links 🤷🏻 (might get users to pay more attention)

Hopefully I've left the questions open enough that they apply to more than just me.

Thanks for reading this far. Hopefully this early feedback continues to help 🙇🏻

🙅🏻 Where the Beta still shows

The ease of interacting with the AI assistant

• Currently this is where the BETA really shows; (as already mentioned by @andihalberkamp), we can't get the bot's attention. To clarify, the @ reference does not appear to be correct:
  

TL;DR - Is it me? or the bot?

@Adal3n3 (and team), I'm not sure I'm properly setup. Here's what I know:

• I've installed the app:
  

• I can see the feature both on GitHub's side and CodeCov's side:
  
  

• I can get other bots to respond to @ reference / prompts (e.g. @dependabot, etc.)
  

• Copy-Paste and Type-it-out for "@codecov-ai-reviewer review" yields the same no-result behavior (i.e. just a plain PR comment, with no bot response)

• I've tried, a draft PR, a different "normal/ready-for-review" PR, a different PR (in each state, both before and after major changes (e.g. 60+ LoC of changes)), and even on a dependabot opened PR (just for completeness)

• All attempts have been on Public repositories (with working CodeCov Coverage PR comments)

Again Thanks for reading this far. Hopefully this feedback continues to help 🤷

Please Advise. 🙇

• bot Lacks feedback for in-progress.

TL;DR - Plea for `Ack` feature

Feedback for in-progress is a must have (something that might help Codecov's infrastructure throttle traffic at scale too). Consider GitHubs's @dependabot will thumbs-up respond to @ prompts from the user. This simple act is not only helpful for the user (e.g. reduces anxiety, allows user to move to another task, rather than watch the bot, etc.) but critical for responding to the scale GitHub's @dependabot must respond to. 🌱

Slow is not a big issue, live people reviewing are typically slow too. However combined with no feedback about in-progress, and I see impatient users spamming the bot's API, (I'm sure this is not a new thought for your team, but I hope it is outside confirmation for decision makers 🙊)

Hopefully this early feedback helps 🤷🏻

Summary - 🚫 Not ready for daily use, yet?

Unfortunately, without feedback, I'm unable to provide a better analysis from the user perspective. 🤷 I'm going to uninstall the bot for now. I'm still interested in the ai-reviewer, so members of the CodeCov AI team should feel free to mention me on this thread if/when there is an update on this, 🙇 assuming they find this kind of feedback helpful, of-course.

[mikebronner]

This is also not working for me. I have installed the GitHub app, but there is no response or indication that anything is actually happening after submitting a comment with @codecov-ai-reviewer review