feat: add role-specific statistics to telemetry and UI#15234
feat: add role-specific statistics to telemetry and UI#15234
Conversation
Summary of ChangesHello @jacob314, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the system's ability to monitor and understand LLM usage by introducing role-specific categorization for all LLM calls. This allows for a detailed breakdown of how different functional components, such as the main agent, subagents, and various internal utilities (e.g., summarizers, routers, edit correctors), consume LLM resources. The user interface has been polished to display these new granular statistics, providing users and developers with a clearer, more actionable view of model performance, token consumption, and potential cost drivers. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
|
Size Change: +9.2 kB (+0.04%) Total Size: 22 MB
ℹ️ View Unchanged
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces role-specific statistics for LLM calls, a valuable addition for telemetry and debugging. The changes are extensive, touching core client logic, telemetry types, and UI components to track and display these new metrics. The implementation is generally solid, with new tests to validate the UI changes.
I've identified one potential issue in the ModelStatsDisplay component that could lead to a UI crash if a model's metrics are missing the roles property. My review includes a specific comment with a suggested fix for this.
Overall, this is a well-executed feature that enhances the observability of the system.
jacob314
force-pushed
the
stats_breakdown
branch
from
8fd23bd to
5fc5efd
Compare
chore: updated call sites to use role parameter UI Polish UI: Update StatsDisplay role breakdown layout to be indented and below model row feat: add specific LlmRole for codebase_investigator subagent feat(cli): breakdown tokens by input/output/cache in model stats roles
jacob314
force-pushed
the
stats_breakdown
branch
from
5fc5efd to
31236f8
Compare
jacob314
force-pushed
the
stats_breakdown
branch
from
31236f8 to
2aeddc1
Compare
abhipatel12
left a comment
abhipatel12
left a comment
There was a problem hiding this comment.
Thanks for making this change to track model usage by role!
Couple questions, but also not 100% sure for the primary /stats including this additional info. Could feel cluttered in a TUI
WDYT?
| options: _CommonGenerateOptions, | ||
| shouldRetryOnContent: (response: GenerateContentResponse) => boolean, | ||
| errorContext: 'generateJson' | 'generateContent', | ||
| role: LlmRole = 'utility_tool' as LlmRole, |
There was a problem hiding this comment.
Should we have defaults? I could see avoiding defaults and forcing a role to callers to avoid missing attribution to a caller.
| if (!logRecord.attributes) logRecord.attributes = {}; | ||
| logRecord.attributes['role'] = event.role; | ||
| } |
There was a problem hiding this comment.
Isn't this logic also set in the line above at:
const logRecord: LogRecord = event.toLogRecord(config);The code looks like:
toLogRecord(config: Config): LogRecord {
const attributes: LogAttributes = {
...getCommonAttributes(config),
'event.name': EVENT_API_REQUEST,
'event.timestamp': this['event.timestamp'],
model: this.model,
prompt_id: this.prompt.prompt_id,
request_text: this.request_text,
};
if (this.role) {
attributes['role'] = this.role;
}
return { body: `API request to ${this.model}.`, attributes };
}| export enum LlmRole { | ||
| MAIN = 'main', | ||
| SUBAGENT = 'subagent', | ||
| SUBAGENT_CODEBASE_INVESTIGATOR = 'subagent_codebase_investigator', |
There was a problem hiding this comment.
should we use use subagent roles like this?
What will happen when users start adding subagents of their own? Will we group all custom subagents under a single category?
There was a problem hiding this comment.
I'm not sure about the UI of the primary /stats
For example, looking at the screenshot below, it feels crowded and slightly difficult to find the "main" stats that I'm interested in.
Utility calls could be kept within /stats model since that is way more detailed. I see an argument for "agents" b/c that will have a material impact on token usage while utility calls are typically lightweight.
gemini-cli
bot
added
priority/p2
| const role = | ||
| this.definition.name === 'codebase_investigator' | ||
| ? LlmRole.SUBAGENT_CODEBASE_INVESTIGATOR | ||
| : LlmRole.SUBAGENT; |
There was a problem hiding this comment.
we have another built-in subagent and users could have a lot more of other subagents they are going to build. Either we should categorized all of them into one role maybe something like (built-in-agent, local-agent, remote-agent), or use their actual names.
|
wondering if we should provide a separate view for role metrics as the number of tools/agents could grow and may not fit in the screen. |
|
Automatically closing PR as it is older than 7 days and authored by a team member. |
… telemetry and UI (cont. google-gemini#15234)
4 participants
Summary
Add role-specific statistics to
/statsand/stats modelDemos:
Colors and styling is quickly tossed together please suggest feedback for how to make this polished.
Some topics for discussion:
should we dash separate things like subagent-codebase-investigator or write it as "Subagent Codebase Investigator"?
What colors should we use for the roles?
Should we tweak the table rendering to leverage things like utilities typically using a fixed model like flash-lite so there is no need to have so many rows for their columns.
Other roles like main will tend to have multiple columns with auto model routing.
We could surface errors on a per role basis as we are tracking it for completeness but it doesn't matter that much.
Details
LlmRoleenum to classify LLM interactions (e.g., main agent, subagents, various utility functions like summarization, routing, autocomplete, etc.).LlmRole.ModelStatsDisplayandStatsDisplaycomponents to visually present these role-specific LLM usage breakdowns, offering a more insightful view of model activity.LlmRoleparameter has been consistently integrated across numerous core components, utility functions, and client calls that interact with LLMs, ensuring comprehensive data collection.UI changes are minimal and just intended as a starting point for UX discussions. Core work is the migration of all the code in packages/core to use the new tracking.
Related Issues
Fixes: #14538
How to Validate
Exercise various models and trigger subagents, autocomplete, and other ways Gemini CLI uses models. Make sure they are attributed correctly.
When reviewing the code in package/core make sure that we are never setting a default value for LlmRole. My instructions for the plan for Gemini CLI was to require all code to pass in a valid LlmRole value so that static analysis could catch any issues where it wasn't passed in but there was one case I had to catch manually where Gemini CLI added a default value to make it easier for tests to pass that introduced a bug where tokens that weren't from the main event loop were tagged as main event loop.