CodeRabbit for AI code reviews

[Piedone]

Attention, contributors: See #15439 (comment).

Is your feature request related to a problem? Please describe.

PRs usually sit open for too long (related: #15029), partially due to nobody coming to review earlier.

Describe the solution you'd like

We could try CodeRabbit, which is an AI code review bot: it uses ChatGPT and perhaps other similar LLM tools to provide code review comments. Additionally:

• You can have a discussion with it.
• Tt's capable of learning the habits of a project.
• When there's an issue linked, it can verify whether the PR addresses the issue.
• It can summarize the changes of a PR.
• Reviews are done either automatically or on-demand (requested with a comment).
• It's free for open-source projects.

For a while now we at Lombiq tested it on a couple of repositories. It's a mixed bag, with perhaps 50-50 useful and useless comments, but the experience varies greatly PR by PR. Some examples:

• Useful comment.
• Mixed bag for the whole PR.
• Mostly useful comments
• No actionable comments despite enough code changes

So, I'm not entirely convinced, but we could try if it's useful for us and helps with reviews, or it's more like noise.

• Set it up first just from the CodeRabbit UI so it can be tested immediately.
• Once we figured out how to configure it, move the config to a file. See https://github.com/Lombiq/Open-Source-Orchard-Core-Extensions/pull/662/files for a good starting point.
• Document it for contributors that you'll get AI reviews, and take them the same as human reviews. See Get our pull request game in order #15029 for contribution docs too.

This will throw off our community metrics a bit, since the time to first reply for PRs will be down to a few minutes, so we won't be able to track the time to human reviews (and the useful remaining metric will be time to close). Also, if as a reviewer you scan for PRs to review by looking at the number of comments (as an indicator that somebody else reviewed already) then that won't apply anymore. Using the "No reviews" filter can be appropriate, but a "review" here is a changes requested/approved flag, what some people don't use during review (but they just comment).

Describe alternatives you've considered

Perhaps there are other such tools that we can check.

[MikeAlhayek]

It would be nice to give it a try. Are you going to implement it to see how it does?

[Piedone]

Sure, it's just a few minutes.

[Piedone]

I've set it up, as an experiment for now: all non-draft PRs will get CodeRabbit review when opened, and you can ask it to review existing PRs too with https://github.com/coderabbitai review. If you think that CodeRabbit is annoying, or otherwise the feedback it provides under pull requests is not worth it, then please add a comment below!

Core contributors: You should be able to log in under https://app.coderabbit.ai/ with your GitHub account and see the OC repo settings. But this shouldn't be necessary going forward, since CodeRabbit can also be configured from code, what I did under #15813.

I'll wait a few weeks to see how it goes, gather feedback, fine-tune the config, and we should only adopt it long-term if we're all happy with it.

[Piedone]

The bot comments don't skew the PR metrics, since they're excluded, only human replies count for "Time to first response": #15929

[Piedone]

Please vote: #15935.

[Piedone]

I've disabled automatic reviews for now, but am waiting a bit more on #15935.

[hishamco]

CodeRabbit is good and bad, sometimes I find it annoying rather than it helps someone :)

[Piedone]

Yeah, same.

[MikeAlhayek]

I am okay with keeping it if we trigger it manually.

[hishamco]

Sometimes the suggestion is excellent, thanks for having this experiment, we can leave before releasing 2.0 until we get more feedback then we can decide

[Piedone]

I've finalized the config and docs under #15813, then, please review. It's off by default but available.