fix(evals): prevent false positive in hierarchical memory test

[Abhijit-2592]

Isolates stdout from stderr in the test assertion to prevent 'Apple' matches in system logs (e.g., 'apple-darwin' paths in ripgrep cache messages) from failing the fruit preference check.

Summary

Details

Related Issues

Fixes #18779

How to Validate

Pre-Merge Checklist

• Updated relevant documentation and README (if needed)
• Added/updated tests (if needed)
• Noted breaking changes (if any)
• [ x] Validated on required platforms/methods:
  • [ x] MacOS
    • [ x] npm run
    • npx
    • Docker
    • Podman
    • Seatbelt
  • Windows
    • npm run
    • npx
    • Docker
  • Linux
    • npm run
    • npx
    • Docker

[gemini-cli]

Hi @Abhijit-2592, thank you so much for your contribution to Gemini CLI! We really appreciate the time and effort you've put into this.

We're making some updates to our contribution process to improve how we track and review changes. Please take a moment to review our recent discussion post: Improving Our Contribution Process & Introducing New Guidelines.

Key Update: Starting January 26, 2026, the Gemini CLI project will require all pull requests to be associated with an existing issue. Any pull requests not linked to an issue by that date will be automatically closed.

Thank you for your understanding and for being a part of our community!

[gemini-code-assist]

Summary of Changes

Hello @Abhijit-2592, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses an issue in the hierarchical memory test where system-generated output could interfere with test assertions, leading to intermittent failures. By explicitly separating standard output from standard error, the test now accurately evaluates the model's response without being affected by irrelevant log messages, improving test reliability.

Highlights

• Test Fix: Modified the hierarchical memory test to prevent false positives by isolating stdout from stderr before performing assertions. This ensures that system log messages containing words like 'Apple' (e.g., 'apple-darwin' paths) do not incorrectly fail the fruit preference check.

Changelog

• evals/hierarchical_memory.eval.ts
  • Modified the assertion logic to split the result string into stdout and stderr components.
  • Updated the assertModelHasOutput and expect calls to only evaluate the stdout portion, preventing stderr content from causing false negatives.

Activity

• The author has validated the changes on MacOS using npm run.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request correctly isolates stdout from stderr in the conflictResolutionTest to prevent false positives in test assertions. This is a good change to improve test stability. However, the extensionVsGlobalTest in the same file also contains a negative assertion and is vulnerable to the same issue. To ensure all tests are robust, I recommend applying this same logic to extensionVsGlobalTest to prevent future flakiness.

[github-actions]

Size Change: -2 B (0%)

Total Size: 24.3 MB

ℹ️ View Unchanged

Filename	Size	Change
./bundle/gemini.js	24.3 MB	-2 B (0%)
./bundle/sandbox-macos-permissive-closed.sb	1.03 kB	0 B
./bundle/sandbox-macos-permissive-open.sb	890 B	0 B
./bundle/sandbox-macos-permissive-proxied.sb	1.31 kB	0 B
./bundle/sandbox-macos-restrictive-closed.sb	3.29 kB	0 B
./bundle/sandbox-macos-restrictive-open.sb	3.36 kB	0 B
./bundle/sandbox-macos-restrictive-proxied.sb	3.56 kB	0 B

_{compressed-size-action}