Skip to content

Comments

chore: refine system prompt and small update to eval#1940

Merged
MrgSub merged 1 commit intostagingfrom
prompt-evalite
Aug 7, 2025
Merged

chore: refine system prompt and small update to eval#1940
MrgSub merged 1 commit intostagingfrom
prompt-evalite

Conversation

@retrogtx
Copy link
Contributor

@retrogtx retrogtx commented Aug 6, 2025

Summary by cubic

Refined the system prompt for the email assistant to clarify tool usage, safety protocols, and response guidelines. Updated eval test case builders for more realistic coverage and improved test data generation.

  • Prompt Improvements

    • Expanded instructions on when and how to use tools, safety checks, and bulk actions.
    • Added detailed workflow examples, safety protocols, and clearer self-check steps.
    • Updated common use cases and removed manual instruction responses.
  • Eval Updates

    • Replaced and improved test case builders for Gmail search and email composition.
    • Made test prompts and expected outputs more realistic and varied.

Summary by CodeRabbit

  • New Features

    • Enhanced AI assistant guidance with more detailed instructions for tool usage, safety protocols, and workflow examples.
    • Added comprehensive safety protocols for bulk and destructive email operations, including confirmation steps and undo guidance.
    • Expanded support for contextual assistance and smart organization workflows.
  • Refactor

    • Improved and modularized test case generation for AI email search and composition, with stricter validation and clearer prompts.
  • Style

    • Updated prompt language to prioritize relevance in email retrieval instead of a fixed number of recent emails.

@jazzberry-ai
Copy link

jazzberry-ai bot commented Aug 6, 2025

Bug Report

Name Severity Example test case Description
Insufficient Confirmation for Bulk Deletions Medium Delete these 5 emails. The AI might not provide a clear preview of the emails to be deleted or explicitly ask for confirmation before proceeding with the deletion when dealing with a number of emails within the 3-5 thread range.
Failure to Clarify Ambiguous Queries Medium Find emails about the meeting The AI might not ask for clarification when the user's query is ambiguous, leading to incorrect or irrelevant search results.

Comments? Email us.

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Aug 6, 2025

Walkthrough

This update overhauls the AI assistant’s system prompts for email workflows, expanding guidance on tool usage, safety protocols, and workflow examples in both client and server prompt definitions. It also modularizes and enhances test case builders for AI chat evaluations, shifting from fixed numeric limits to relevance-based approaches and enforcing stricter validation.

Changes

Cohort / File(s) Change Summary
Mail Prompt Update
apps/mail/lib/prompts.ts
Modified the AiChatPrompt system prompt, updating the "all_emails" use case to recommend limiting to the most recent relevant emails instead of a fixed number, and retaining the suggestion to use search filters. No changes to exported/public entity signatures.
Server Prompt Overhaul
apps/server/src/lib/prompts.ts
Extensively expanded and restructured the AiChatPrompt system prompt. Added detailed tool usage rules, safety protocols, step-by-step workflow examples, and stricter instructions emphasizing tool invocation over manual or descriptive responses. Revised self-checks, organizational guidance, and bulk operation safety. No changes to exported/public entity signatures.
Test Case Builder Refactor
apps/server/evals/ai-chat-basic.eval.ts
Removed the old buildGmailSearchTestCases function and introduced modular test case builders (makeGmailSearchTestCaseBuilder, makeEmailCompositionTestCaseBuilder) with enhanced prompts, stricter validation, and more detailed instructions. Updated all references to use new builders and standardized prompt usage. Improved the generic test case builder with more explicit system prompt guidance.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant AI Assistant
    participant Tooling Layer

    User->>AI Assistant: User request (e.g., "Show all emails from Alice")
    AI Assistant->>Tooling Layer: Invoke appropriate tool (e.g., inboxRag, getThread)
    Tooling Layer-->>AI Assistant: Return results
    AI Assistant->>User: Respond with results (no manual instructions)
    User->>AI Assistant: Bulk action request (e.g., "Delete all spam")
    AI Assistant->>Tooling Layer: Prepare preview, enforce safety protocols
    Tooling Layer-->>AI Assistant: Return preview/sample
    AI Assistant->>User: Ask for explicit confirmation, show consequences
    User->>AI Assistant: Confirm action
    AI Assistant->>Tooling Layer: Execute bulk operation
    Tooling Layer-->>AI Assistant: Operation complete
    AI Assistant->>User: Confirm completion, provide undo guidance
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~40 minutes

Possibly related PRs

Suggested labels

content, high priority

Suggested reviewers

  • MrgSub

Poem

🚀
Prompts revised, with rules so clear,
Safety, tools, and context near.
From fixed to relevant, emails flow,
Test cases modular, ready to go.
Bulk actions guarded, no manual fuss—
AI gets smarter, just like us!
✉️🤖

Note

⚡️ Unit Test Generation is now available in beta!

Learn more here, or try it out under "Finishing Touches" below.


📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 17639d4 and e2e246f.

📒 Files selected for processing (3)
  • apps/mail/lib/prompts.ts (1 hunks)
  • apps/server/evals/ai-chat-basic.eval.ts (12 hunks)
  • apps/server/src/lib/prompts.ts (5 hunks)
🧰 Additional context used
📓 Path-based instructions (4)
**/*.{js,jsx,ts,tsx}

📄 CodeRabbit Inference Engine (AGENT.md)

**/*.{js,jsx,ts,tsx}: Use 2-space indentation
Use single quotes for strings
Limit lines to 100 characters in length
Semicolons are required at the end of statements

Files:

  • apps/mail/lib/prompts.ts
  • apps/server/src/lib/prompts.ts
  • apps/server/evals/ai-chat-basic.eval.ts
**/*.{js,jsx,ts,tsx,css,scss}

📄 CodeRabbit Inference Engine (AGENT.md)

Use Prettier with sort-imports and Tailwind plugins for code formatting

Files:

  • apps/mail/lib/prompts.ts
  • apps/server/src/lib/prompts.ts
  • apps/server/evals/ai-chat-basic.eval.ts
**/*.{ts,tsx}

📄 CodeRabbit Inference Engine (AGENT.md)

Enable TypeScript strict mode

Files:

  • apps/mail/lib/prompts.ts
  • apps/server/src/lib/prompts.ts
  • apps/server/evals/ai-chat-basic.eval.ts
**/*.{css,js,ts,jsx,tsx,mdx}

📄 CodeRabbit Inference Engine (.cursor/rules/tailwind-css-v4.mdc)

**/*.{css,js,ts,jsx,tsx,mdx}: Chain variants together for composable variants (e.g., group-has-data-potato:opacity-100).
Use new variants such as starting, not-*, inert, nth-*, in-*, open (for :popover-open), and ** for all descendants.
Do not use deprecated utilities like bg-opacity-*, text-opacity-*, border-opacity-*, and divide-opacity-*; use the new syntax (e.g., bg-black/50).
Use renamed utilities: shadow-sm is now shadow-xs, shadow is now shadow-sm, drop-shadow-sm is now drop-shadow-xs, drop-shadow is now drop-shadow-sm, blur-sm is now blur-xs, blur is now blur-sm, rounded-sm is now rounded-xs, rounded is now rounded-sm, outline-none is now outline-hidden.
Use bg-(--brand-color) syntax for CSS variables in arbitrary values instead of bg-[--brand-color].
Stacked variants now apply left-to-right instead of right-to-left.

Files:

  • apps/mail/lib/prompts.ts
  • apps/server/src/lib/prompts.ts
  • apps/server/evals/ai-chat-basic.eval.ts
🧠 Learnings (5)
📓 Common learnings
Learnt from: retrogtx
PR: Mail-0/Zero#1354
File: apps/mail/components/ui/prompts-dialog.tsx:148-159
Timestamp: 2025-06-20T05:34:41.297Z
Learning: In the prompt management system, users should be allowed to save empty prompts as this gives them the choice to disable certain AI functionality or start fresh. Validation should not prevent empty prompts from being saved.
Learnt from: retrogtx
PR: Mail-0/Zero#1573
File: apps/mail/components/create/template-button.tsx:197-216
Timestamp: 2025-07-28T05:37:50.566Z
Learning: retrogtx prefers less nitpicky code review suggestions and may dismiss detailed accessibility/best practice recommendations with casual responses like "bro please".
Learnt from: retrogtx
PR: Mail-0/Zero#1468
File: apps/server/src/main.ts:854-858
Timestamp: 2025-08-04T18:33:59.515Z
Learning: retrogtx gets frustrated with contradictory review suggestions and prefers consistent guidance. When previous suggestions conflict with new ones, acknowledge the contradiction and defer to the existing implementation rather than pushing alternative approaches.
📚 Learning: in apps/server/src/lib/driver/google.ts, the normalization of "draft" to "drafts" in the count() met...
Learnt from: retrogtx
PR: Mail-0/Zero#1734
File: apps/server/src/lib/driver/google.ts:211-221
Timestamp: 2025-07-15T06:46:33.349Z
Learning: In apps/server/src/lib/driver/google.ts, the normalization of "draft" to "drafts" in the count() method is necessary because the navigation item in apps/mail/config/navigation.ts has id: 'drafts' (plural) while the Google API returns "draft" (singular). The nav-main.tsx component matches stats by comparing stat.label with item.id, so the backend must return "drafts" for the draft counter badge to appear in the sidebar.

Applied to files:

  • apps/mail/lib/prompts.ts
📚 Learning: in the threadlabels prompt system, existing labels should not be automatically preserved. the ai age...
Learnt from: MrgSub
PR: Mail-0/Zero#1837
File: apps/server/src/lib/brain.fallback.prompts.ts:211-217
Timestamp: 2025-07-26T20:39:06.670Z
Learning: In the ThreadLabels prompt system, existing labels should not be automatically preserved. The AI agent should re-evaluate all labels (both existing and new) against the current thread summary and only return labels that currently apply to the thread content, even if it means dropping previously applied labels that are no longer relevant.

Applied to files:

  • apps/mail/lib/prompts.ts
  • apps/server/src/lib/prompts.ts
📚 Learning: in apps/mail/lib/hotkeys/mail-list-hotkeys.tsx, the switchcategorybyindex function using hardcoded i...
Learnt from: retrogtx
PR: Mail-0/Zero#1328
File: apps/mail/lib/hotkeys/mail-list-hotkeys.tsx:202-209
Timestamp: 2025-06-18T17:26:50.918Z
Learning: In apps/mail/lib/hotkeys/mail-list-hotkeys.tsx, the switchCategoryByIndex function using hardcoded indices for category hotkeys does not break when users reorder categories, contrary to the theoretical index-shifting issue. The actual implementation has constraints or mechanisms that prevent hotkey targeting issues.

Applied to files:

  • apps/mail/lib/prompts.ts
📚 Learning: in the prompt management system, users should be allowed to save empty prompts as this gives them th...
Learnt from: retrogtx
PR: Mail-0/Zero#1354
File: apps/mail/components/ui/prompts-dialog.tsx:148-159
Timestamp: 2025-06-20T05:34:41.297Z
Learning: In the prompt management system, users should be allowed to save empty prompts as this gives them the choice to disable certain AI functionality or start fresh. Validation should not prevent empty prompts from being saved.

Applied to files:

  • apps/server/src/lib/prompts.ts
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
  • GitHub Check: cubic · AI code reviewer
  • GitHub Check: Analyze (javascript-typescript)
🔇 Additional comments (12)
apps/server/src/lib/prompts.ts (7)

337-367: First principles thinking on tool usage rules - looks solid

The explicit delineation between when to use tools vs responding directly is exactly what we need. This prevents the AI from just explaining what it could do instead of actually doing it. Ship it.


377-383: Thinking process expansion from 4 to 6 steps - this is the way

Adding ambiguity resolution and context maintenance steps makes the assistant more robust. The safety protocols check at step 4 is particularly important for preventing unintended mass operations.


459-519: Workflow examples are comprehensive and practical

These examples cover the key use cases well. The explicit action sequences help ensure consistent behavior across similar requests.


521-545: Safety protocols implementation - absolutely critical

The staged approach to bulk operations with different thresholds (1-2, 3-5, 6-20, 21+) is smart. The validation patterns ensure users know exactly what's happening before destructive actions. This prevents "oops, deleted everything" moments.


547-571: Smart organization and contextual assistance sections are well-structured

The sequence-based approach and explicit handling of relative references ("this email", "those emails") will make the assistant much more intuitive to use.


589-599: Common use cases section properly emphasizes immediate tool usage

The repeated emphasis on "ALWAYS use inboxRag tool immediately" prevents the assistant from falling back to manual instructions. Good reinforcement of the core principle.


601-608: Self-check list is comprehensive and actionable

Point 6 "Did I take action immediately rather than explaining what I could do?" is the key differentiator. This checklist will help maintain consistent behavior.

apps/mail/lib/prompts.ts (1)

423-423: Relevance > arbitrary limits

Moving from "10 most recent" to "most recent that are relevant" is the right call. Fixed limits are arbitrary - what matters is relevance to the user's query. This aligns with the first principles approach in the main prompt updates.

apps/server/evals/ai-chat-basic.eval.ts (4)

51-74: Test case builder improvements - more specific and actionable

The updated system prompt for the test case generator now explicitly lists the available tools and provides clearer guidelines. The schema validation with minimum string lengths adds robustness.


76-101: Gmail search test case builder properly scoped

Separating Gmail search test generation into its own builder makes sense. The focus on key operators rather than full queries is smart - tests the essential functionality without getting bogged down in syntax details.


108-109: Removing thread ID argument - correct simplification

All the AiChatPrompt() calls now have no arguments, which is consistent with the function signature. Clean.


235-260: Email composition test builder is well-designed

Having a dedicated builder for email composition tests with specific validation for key phrases makes the tests more meaningful. The focus on common email types (thank you, follow-up, etc.) covers the primary use cases.

✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch prompt-evalite

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Explain this complex logic.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai generate unit tests to generate unit tests for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@coderabbitai coderabbitai bot requested a review from MrgSub August 6, 2025 05:56
@coderabbitai coderabbitai bot added content High Priority High Priority Work labels Aug 6, 2025
Copy link
Contributor

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cubic analysis

No issues found across 3 files. Review in cubic

Copy link
Collaborator

MrgSub commented Aug 7, 2025

Merge activity

  • Aug 7, 4:00 AM UTC: A user started a stack merge that includes this pull request via Graphite.
  • Aug 7, 4:00 AM UTC: @MrgSub merged this pull request with Graphite.

@MrgSub MrgSub merged commit 69e0676 into staging Aug 7, 2025
8 checks passed
@MrgSub MrgSub deleted the prompt-evalite branch August 7, 2025 04:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

content High Priority High Priority Work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants