feat(plan): create metrics for usage of AskUser tool

[Adib234]

Summary

This PR implements comprehensive telemetry metrics for the AskUser tool and enhances the telemetry system to track the active approval mode across all events. These changes enable detailed analysis of user interaction patterns, question type usage, and tool effectiveness, specifically allowing for data dissection by modes like Plan mode.

Details

• AskUser Tool Metrics: Updated AskUserInvocation to collect and report the following metrics via the ToolResult.data field:
  • ask_user_question_types: An array of the types of questions asked (text, choice, yes/no).
  • ask_user_dismissed: Whether the user escaped or cancelled the dialog.
  • ask_user_empty_submission: Whether the user submitted without providing any answers.
  • ask_user_answer_count: The number of questions actually answered.
• Telemetry Infrastructure:
  • Enhanced the ToolCallEvent constructor to automatically merge custom data from tool results into the event metadata.
  • Updated ClearcutLogger to include active_approval_mode (fetched from the policy engine) as a common attribute in every event, fulfilling the requirement to dissect data by approval mode.
• Documentation: Updated docs/cli/telemetry.md to include descriptions of the new AskUser metadata and the inclusion of active_approval_mode in common attributes.

Related Issues

Fixes #18455

How to Validate

Searched for logs here https://cloudlogging.app.goo.gl/mr3SD483iVC3PqYGA, verified that it matches with what happened when we called AskUser tool

Pre-Merge Checklist

• Updated relevant documentation and README (if needed)
• Added/updated tests (if needed)
• Noted breaking changes (if any)
• Validated on required platforms/methods:
  • MacOS
    • npm run
    • npx
    • Docker
    • Podman
    • Seatbelt
  • Windows
    • npm run
    • npx
    • Docker
  • Linux
    • npm run
    • npx
    • Docker

[gemini-code-assist]

Summary of Changes

Hello @Adib234, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the application's telemetry capabilities by introducing detailed metrics for the AskUser tool and integrating the active approval mode into all logged events. These changes provide a deeper understanding of user interactions with the AskUser tool and enable more granular analysis of system usage based on different approval contexts, ultimately improving data-driven decision-making for tool effectiveness and user experience.

Highlights

• AskUser Tool Telemetry: Implemented comprehensive telemetry metrics for the AskUser tool, capturing question types, dismissal status, empty submission status, and the count of answered questions.
• Active Approval Mode Tracking: Enhanced the telemetry system to track the active_approval_mode across all events, allowing for data dissection by different approval modes like 'Plan mode'.
• Telemetry Infrastructure Update: Updated the ToolCallEvent constructor to automatically merge custom data from tool results into the event metadata, streamlining data collection.
• Documentation Update: Updated the telemetry documentation to reflect the new AskUser metadata and the inclusion of active_approval_mode in common attributes.

Changelog

• docs/cli/telemetry.md
  • Added active_approval_mode to the list of common attributes included in all logs and metrics.
  • Documented new metadata fields for the AskUser tool, including ask_user_question_types, ask_user_dismissed, ask_user_empty_submission, and ask_user_answer_count.
• packages/core/src/telemetry/clearcut-logger/clearcut-logger.test.ts
  • Added a test case to verify that active_approval_mode is correctly logged as a default metadata attribute.
  • Introduced a new test to ensure AskUser tool-specific metadata (question types, dismissed, empty submission, answer count) is properly logged.
• packages/core/src/telemetry/clearcut-logger/clearcut-logger.ts
  • Mapped new AskUser related metadata keys (ask_user_question_types, ask_user_dismissed, ask_user_empty_submission, ask_user_answer_count) for telemetry logging.
  • Included active_approval_mode in the default log metadata, retrieving its value from the policy engine configuration.
• packages/core/src/telemetry/clearcut-logger/event-metadata-key.ts
  • Updated the Next ID for EventMetadataKey enum to 156.
  • Added new EventMetadataKey enums for GEMINI_CLI_ASK_USER_QUESTION_TYPES, GEMINI_CLI_ASK_USER_DISMISSED, GEMINI_CLI_ASK_USER_EMPTY_SUBMISSION, and GEMINI_CLI_ASK_USER_ANSWER_COUNT.
• packages/core/src/telemetry/loggers.test.ts
  • Imported AnyDeclarativeTool type.
  • Added a test case to confirm that data from a tool's response is correctly merged into the event metadata for logging.
• packages/core/src/tools/ask-user.test.ts
  • Added assertions to verify that AskUser tool results include data containing ask_user_question_types, ask_user_dismissed, ask_user_empty_submission, and ask_user_answer_count for successful submissions.
  • Extended assertions to check result.data for empty submissions, confirming ask_user_empty_submission is true and ask_user_answer_count is zero.
  • Included assertions for result.data when the dialog is dismissed, ensuring ask_user_dismissed is true.
• packages/core/src/tools/ask-user.ts
  • Introduced logic to collect ask_user_question_types based on the questions presented.
  • Modified the execute method to include ask_user_dismissed in the ToolResult.data when the dialog is cancelled.
  • Added ask_user_dismissed, ask_user_empty_submission, and ask_user_answer_count to the metrics object before returning the ToolResult.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request effectively adds valuable telemetry for the AskUser tool and enhances the overall telemetry system by including the active approval mode in all events. The changes are well-implemented across the board, with corresponding updates to documentation and comprehensive tests. I've identified one high-severity maintainability concern regarding how metadata is constructed in ToolCallEvent, which could lead to issues in the future. Otherwise, the implementation is solid.

[github-actions]

Size Change: +1.83 kB (+0.01%)

Total Size: 24.3 MB

Filename	Size	Change
./bundle/gemini.js	24.3 MB	+1.83 kB (+0.01%)

ℹ️ View Unchanged

Filename	Size
./bundle/sandbox-macos-permissive-closed.sb	1.03 kB
./bundle/sandbox-macos-permissive-open.sb	890 B
./bundle/sandbox-macos-permissive-proxied.sb	1.31 kB
./bundle/sandbox-macos-restrictive-closed.sb	3.29 kB
./bundle/sandbox-macos-restrictive-open.sb	3.36 kB
./bundle/sandbox-macos-restrictive-proxied.sb	3.56 kB

_{compressed-size-action}

[jerop]

is it defaulting to choice? the type should be a required field

[Adib234]

looks like question type can be undefined here, is there a better default value for this?

gemini-cli/packages/core/src/confirmation-bus/types.ts

Line 151 in 099aa96

type?: QuestionType;

[jerop]

ah i see, and it defaults to choice there as well - all good

cc @jackwotherspoon wonder if we should make type required

[jackwotherspoon]

probably

[jerop]

Filed #18930