Fix issue where Gemini CLI creates tests in a new file

[gundermanc]

Summary

Fixes an issue where Gemini CLI either does not add new tests or creates them in a new file instead of placing them next to existing tests.

Adds an eval test: https://github.com/google-gemini/gemini-cli/actions/runs/21807626011

Related Issues

Fixes #18361

How to Validate

Pre-Merge Checklist

• Updated relevant documentation and README (if needed)
• Added/updated tests (if needed)
• Noted breaking changes (if any)
• Validated on required platforms/methods:
  • MacOS
    • npm run
    • npx
    • Docker
    • Podman
    • Seatbelt
  • Windows
    • npm run
    • npx
    • Docker
  • Linux
    • npm run
    • npx
    • Docker

[gemini-code-assist]

Summary of Changes

Hello @gundermanc, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refines the AI model's behavior regarding test file management. It introduces a new evaluation test to validate that the system correctly updates existing test files instead of generating redundant ones. Concurrently, it modifies the core mandates to explicitly guide the AI to prioritize updating related tests and creating new tests within existing, relevant test files, thereby preventing the duplication of test code and ensuring more efficient test suite maintenance.

Highlights

• New Evaluation Test for Test File Management: A new evaluation test (evals/edit-locations-eval.eval.ts) has been added. This test specifically verifies that the system updates existing test files (e.g., src/math.test.ts) when a bug is fixed in a corresponding source file (e.g., src/math.ts), ensuring that new, duplicate test files are not created.
• Updated Core Mandates for Test Handling: The core prompts (packages/core/src/prompts/snippets.ts) have been modified to include a new, explicit 'Testing' mandate. This mandate instructs the model to always search for and update related tests after making code changes, and to add new tests within existing, related test files or create new ones if no related file exists.
• Refined 'Proactiveness' Mandate: The existing 'Proactiveness' mandate was adjusted to remove the specific instruction about adding tests, as the new 'Testing' mandate now covers this aspect more precisely and comprehensively.

Changelog

• evals/edit-locations-eval.eval.ts
  • Added a new evaluation test Edits location eval to verify that the system updates existing test files (e.g., src/math.test.ts) when fixing a bug in a source file (e.g., src/math.ts), rather than creating a new test file.
• packages/core/src/prompts/snippets.ts
  • Modified the 'Proactiveness' mandate to focus on considering all created files as permanent artifacts, removing the explicit mention of adding tests for quality.
  • Introduced a new 'Testing' mandate, instructing the model to always search for and update related tests after code changes, and to add new tests in related test files or create new ones if a related file doesn't exist.

Ignored Files

• Ignored by pattern: .gemini/** (1)
  • .gemini/commands/fix-behavioral-eval.toml

Activity

• No specific human activity (comments, reviews, progress updates) has been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new evaluation test and updates the core system prompt to prevent the agent from creating duplicate test files when existing ones are present. The changes are well-aligned with the goal. I've suggested a minor wording tweak to the new prompt instruction to make it even clearer for the language model, further reducing the chance of incorrect behavior.

[gemini-code-assist]

The phrasing Add new tests in a related test file, if one exists, or create a new test file could be ambiguous to the language model. A more direct conditional statement would make the instruction clearer and more robust, ensuring the model prioritizes updating existing test files.

Suggested change

	- **Testing**: ALWAYS search for and update related tests after making a code change. Add new tests in a related test file, if one exists, or create a new test file. e.g.: fix a bug in set.ts, add a test in set.test.ts.
	- **Testing**: ALWAYS search for and update related tests after making a code change. If a related test file exists, add new tests to it; otherwise, create a new test file. e.g.: fix a bug in set.ts, add a test in set.test.ts.

[github-actions]

Size Change: +205 B (0%)

Total Size: 23.9 MB

ℹ️ View Unchanged

Filename	Size	Change
./bundle/gemini.js	23.9 MB	+205 B (0%)
./bundle/sandbox-macos-permissive-closed.sb	1.03 kB	0 B
./bundle/sandbox-macos-permissive-open.sb	890 B	0 B
./bundle/sandbox-macos-permissive-proxied.sb	1.31 kB	0 B
./bundle/sandbox-macos-restrictive-closed.sb	3.29 kB	0 B
./bundle/sandbox-macos-restrictive-open.sb	3.36 kB	0 B
./bundle/sandbox-macos-restrictive-proxied.sb	3.56 kB	0 B

_{compressed-size-action}

[alisa-alisa]

/gemini review

[gemini-code-assist]

Code Review

This pull request aims to fix an issue where the Gemini CLI creates a new test file instead of updating an existing one. This is addressed by updating the core system prompt to explicitly instruct the agent to search for and update existing test files. A new evaluation test is added to verify this behavior. The changes to the prompt and the addition of the test are well-aligned with the goal. I've found one issue in the new test file where an assertion could be made more specific to better reflect the test's intent.

[gemini-code-assist]

The assertion message 'Expected only two files changed' suggests an exact check, but greaterThanOrEqual(2) allows for more than two files to be changed. To make the test stricter and align with the stated expectation, you should check for an exact size of 2.

      expect(new Set(targetFiles).size, 'Expected only two files changed').toBe(2);