Offering help: Improving test coverage

[erlapso]

Hi everyone! I noticed a lot of Pull Requests, particularly in relation to Core, API and Audio.

I'd like to help improve the test coverage using CodeBeaver, an AI agent that I built (it's free for open source!). To demonstrate what's possible, I've created a PR:
github.com/CodeBeaverAI/Red-DiscordBot/pull/1
that adds comprehensive tests to the last Pull Request that was opened here: Add --cog-path CLI argument .

What the PR demonstrates:

• Increases coverage by 72% for streamtypes.py
• By 94% for cli.py
• 53% for i18.py

On top of that: If you check the PR you will see that CodeBeaver does not only go for coverage - it also adds all the edge cases it can think of (we make it iterate 3 times on that alone).

How it works:

• CodeBeaver analyzes your code changes in PRs
• Automatically generates and updates tests
• Opens PRs with new/updated tests
• Helps catch bugs before they reach production

Next steps

If this interests the maintainer team, I'm happy to:

• Walk through how the tests were generated
• Help set up automated test generation for future PRs (takes ~5 minutes)
• Focus on specific areas you'd like to improve coverage for

Let me know what you think! Happy to adapt this approach based on the project's needs.

Vittorio

[Jackenmen]

Hi Vittorio,

We are not interested in using AI-powered tooling with the project at the moment. On a personal note, I think you should clearly disclose your involvement with the project, I could only spot one part of the message that this can be deduced from ("we make it iterate 3 times on that alone") and if not for that, it wouldn't necessarily be obvious.

To not be completely ignorant, I did look through the PR briefly and it is not directly related to the PR that you mentioned - it just adds test coverage to a few parts of the codebase. One could argue that the CLI flags part at least relates to the same area of the codebase but --cog-path is not mentioned/tested explicitly anywhere. I'll admit that, in the parts of the PR that I actually looked at, there are some useful test scenarios. Whether that's how I'd approach testing those scenarios, I'm not so sure about but at least there's some value in AI coming up with useful scenarios/edge cases for testing.

[erlapso]

Thank you for your feedback! I did not mean to hide my involvement in any way (it's front and center in my profile), but I see what you mean. I edited my comment to indicate that I built CodeBeaver in the first sentence.

About the direct relation between the original PR and the Unit Test: You are right. I should have mentioned that CodeBeaver goes through the files that changed the Pull Request (so cli.py in this case), but it considers the file as a whole, not the specific diffs added (the --cog-path in this case). It does not do that yet at least :) Let's say we built that feature, would you be open to trying CodeBeaver then?

Thank you!

Vittorio

[Jackenmen]

Let's say we built that feature, would you be open to trying CodeBeaver then?

As I mentioned, currently, we are not interested in the use of AI tooling for this project, so no, we wouldn't. Maybe that changes in the future, but if so, you've already put CodeBeaver on our radar, and we'd just be able to try it on our own 😄 On a personal note, I definitely think it would be more useful in this way for PRs since it would be more directly relevant to whatever PR CodeBeaver went through and would also aid the reviewer by revealing potential issues through the proposed test cases. The current behaviour seems more useful for running through the codebase as a whole, if one had a goal of widely covering the project with tests. It is a way larger goal than just covering newly introduced code though as it requires reviewer time for the huge amount of new tests that would have to be added.