perf: switch from open ai to groq api#594
Conversation
|
@Vicentesan is attempting to deploy a commit to the Zero Team on Vercel. A member of the Team first needs to authorize it. |
WalkthroughThis pull request updates the email processing and AI response logic by transitioning API calls from OpenAI to a new Groq-based implementation. It revises the token truncation strategy in email threads, adds an email thread summarization helper, and adjusts error handling in the reply composer. A new module is introduced for Groq API interactions, including schema validation and utility functions for generating completions and embeddings. Additionally, package dependencies have been updated accordingly. Changes
Sequence Diagram(s)sequenceDiagram
participant U as User
participant RC as Reply Composer
participant AR as generateAIResponse (ai-reply.ts)
participant ES as extractEmailSummary
participant GC as generateCompletions (groq.ts)
participant GA as Groq API
U->>RC: Clicks AI Reply button
RC->>AR: Initiates generateAIResponse
AR->>ES: Calls extractEmailSummary to summarize thread
ES-->>AR: Returns email summary
AR->>GC: Sends summary and prompt details
GC->>GA: Makes API call to Groq
GA-->>GC: Returns generated completion
GC-->>AR: Passes back completion
AR-->>RC: Returns finalized AI response
RC->>U: Displays response
Possibly related PRs
Suggested reviewers
Poem
✨ Finishing Touches
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (7)
apps/mail/actions/ai-reply.ts (4)
8-57: Consider unifying safety margins and clarifying truncation strategy.
The truncation logic is well-structured and accounts for large emails. However, there is a slight inconsistency in using an 80% margin (line 30) for the final email truncation and a 90% margin (line 42) when adding older emails. Introducing clear constants or a unified approach may improve maintainability and clarity.- const safeCharLimit = Math.floor(maxTokens * 4 * 0.8); + // Example: unify to a single margin constant (e.g., 0.85) + const SAFETY_MARGIN = 0.85; + const safeCharLimit = Math.floor(maxTokens * 4 * SAFETY_MARGIN);
59-107: Handle possible multiline 'Subject' and 'From' fields or multiple older emails as needed.
Currently, theextractEmailSummaryfunction splits the thread and extracts single-line subject/sender fields. If you ever encounter multiline or irregular headers, consider more robust parsing. Additionally, the logic only adds one previous email; you might allow adding more older emails if the token budget allows.
109-131: Watch for overly broad regex that might remove valid content.
The cleanup function uses broad patterns (e.g., lines starting with "Here is" or "Subject:") that could potentially remove normal text. While this is likely intentional, confirm that you will not remove legitimate content in edge cases.
133-196: Double-check console logging for potential data leakage.
In the event of an error, consider sanitizing any sensitive user data before logging. Currently, the entire error object is logged. This can be acceptable in dev environments but might be risky if logs are persisted in production.apps/mail/lib/groq.ts (3)
102-119: Partial success handling for multiple embeddings.
Skipping problematic entries (rather than failing entirely) is a valid design choice. If you’d prefer strict enforcement, consider rethrowing errors at the first failure.
141-269: Be cautious about logging request details.
While debugging is important, ensure that sensitive user data isn’t overexposed in logs (line 210). You may want to redact or minimize content in production logs.
292-326: Duplicate cleanup logic may be unified or shared.
You have near-identical cleanup logic inai-reply.ts, so factoring it out into a single helper ensures consistency and DRY code.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
bun.lockis excluded by!**/*.lock
📒 Files selected for processing (7)
apps/mail/actions/ai-reply.ts(2 hunks)apps/mail/actions/ai-search.ts(3 hunks)apps/mail/components/mail/reply-composer.tsx(1 hunks)apps/mail/lib/ai.ts(4 hunks)apps/mail/lib/groq.ts(1 hunks)apps/mail/package.json(1 hunks)package.json(0 hunks)
💤 Files with no reviewable changes (1)
- package.json
🧰 Additional context used
🧬 Code Definitions (2)
apps/mail/lib/ai.ts (1)
apps/mail/lib/groq.ts (1)
generateCompletions(141-269)
apps/mail/actions/ai-search.ts (1)
apps/mail/lib/groq.ts (1)
generateCompletions(141-269)
🔇 Additional comments (20)
apps/mail/package.json (1)
17-17: Dependency addition for @better-fetch/fetch looks good.The addition of
@better-fetch/fetchdependency aligns with the PR objective of transitioning to the Groq API. This likely provides improved fetch functionality used in the Groq API implementation.apps/mail/components/mail/reply-composer.tsx (1)
501-501: Error handling updated to check for Groq API errors.The error message check has been properly updated from checking for "OpenAI API" to "Groq API" errors, which aligns with the PR objective of transitioning from OpenAI to Groq API.
apps/mail/actions/ai-search.ts (4)
5-5: Import updated for Groq integration.The import statement has been correctly updated to use the
generateCompletionsfunction from the Groq library instead of OpenAI.
27-31: Environment variable check updated for Groq API.The check for the API key has been updated from OpenAI to Groq, which is consistent with the API change. The error message has been correctly updated to reflect this change.
54-59: API call replaced with Groq implementation.The OpenAI API call has been properly replaced with a call to the
generateCompletionsfunction, maintaining the same functionality while taking advantage of the Groq API.
61-61: Response handling updated for Groq API.The response handling has been updated to work with the structure provided by the
generateCompletionsfunction, correctly extracting the completion from the response.apps/mail/lib/ai.ts (7)
1-2: Import updated for Groq integration.The OpenAI import has been correctly replaced with the
generateCompletionsfunction from the Groq library.
30-32: Environment variable check updated for Groq API.The error message and check have been correctly updated to verify the Groq API key instead of the OpenAI API key.
56-61: System prompt construction refactored for Groq API.The system prompt construction has been refactored to work with the Groq API, extracting system messages from the conversation history to build the prompt.
65-66: Context enrichment updated for Groq API format.The code now correctly appends the current email draft and recipient information to the system prompt in the format expected by the Groq API implementation.
Also applies to: 70-71
73-78: User prompt construction refactored for Groq API.The user prompt construction has been refactored to build a conversation history string from user and assistant messages, which is consistent with the Groq API expectations.
80-86: API call replaced with Groq implementation.The OpenAI API call has been properly replaced with a call to the
generateCompletionsfunction with appropriate parameters:
- Uses 'gpt-4o-mini' model
- Passes system and user prompts
- Sets appropriate temperature and token limits based on whether it's a question
This implementation aligns with the PR objective to transition to the Groq API.
88-88: Response handling updated for Groq API.The response handling has been updated to extract the completion from the response object returned by the
generateCompletionsfunction, correctly adapting to the new API structure.apps/mail/actions/ai-reply.ts (1)
5-5: No concerns with the new import.
This change ensures that the file now uses the Groq-basedgenerateCompletionsfunction instead of OpenAI.apps/mail/lib/groq.ts (6)
1-33: Validation schema for chat completions looks consistent.
ThegroqChatCompletionSchemacomprehensively matches the expected fields in the response.
35-49: Embedding schema looks accurate.
ThegroqEmbeddingSchemaappears correct for the Groq embeddings response format.
51-60: Model constants are clear and concise.
Defining an explicit mapping for Groq models helps maintain clarity.
62-68: Effective model name mapping.
Providing a fallback to the providedmodelstring if it’s unrecognized is a good approach.
70-101: Embedding creation flow is robust.
The usage ofbetterFetchwith schema validation and error handling covers critical scenarios.
121-139: Completions parameters and request body definitions look good.
Allowing flexible properties inGroqRequestBodyhelps manage additional fields.
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
apps/mail/actions/ai-reply.ts(2 hunks)
🧰 Additional context used
🧬 Code Definitions (1)
apps/mail/actions/ai-reply.ts (1)
apps/mail/lib/groq.ts (3)
truncateThreadContent(274-290)cleanupEmailContent(293-325)generateCompletions(141-269)
🔇 Additional comments (8)
apps/mail/actions/ai-reply.ts (8)
5-5: Switch from OpenAI to Groq importsThis import change aligns with the PR objective of transitioning from OpenAI to Groq API. It now imports the required utility functions from the Groq library.
7-55: Good enhancement to summarization logicThe new
extractEmailSummaryfunction is a significant improvement over simple truncation. It intelligently:
- Handles edge cases (single emails, already small content)
- Extracts metadata from all emails (subject, sender)
- Creates a structured summary
- Prioritizes the most recent email's full content
- Conditionally includes previous email content based on remaining token budget
This approach should result in more context-aware AI responses while staying within token limits.
93-95: Updated error handling for Groq APIGood update to the error message to correctly reflect the Groq API key requirement.
97-99: Reduced token budgetThe token limit has been reduced from the previous version (implied by the PR summary), which aligns with Groq API recommendations. Using
extractEmailSummarywith a 3000 token limit should help optimize API usage while maintaining meaningful context.
100-116: Improved system promptThe updated system prompt provides clearer instructions for the AI to generate better email replies. It addresses common issues with AI-generated content like:
- Avoiding placeholders and templates
- Removing meta-text and explanations
- Ensuring the response is ready to send
- Maintaining appropriate formatting
This should result in higher quality email replies.
118-125: Concise user promptThe user prompt has been streamlined and clarified to work better with the Groq API. The 200-word limit instruction helps ensure responses remain concise.
128-135: Updated API call to GroqThe implementation correctly uses the new
generateCompletionsfunction with appropriate parameters for the Groq API:
- Using 'llama3-8b-8192' model specific to Groq
- Configured temperature and max_tokens parameters
- Passing the updated system prompt and user prompt
This aligns with the PR objective of switching to Groq.
140-143: Simplified error handlingThe error handling has been simplified while maintaining appropriate error logging, which is a good practice.
apps/mail/lib/groq.ts
Outdated
| @@ -0,0 +1,326 @@ | |||
| import { betterFetch } from "@better-fetch/fetch"; | |||
There was a problem hiding this comment.
we can use the base javascript fetch if u wanna
There was a problem hiding this comment.
lets do it. replace betterFetch with a base JS or axios
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
apps/mail/actions/ai-reply.ts (1)
7-55: New email summarization implementation looks solid but has potential edge casesThe new
extractEmailSummaryfunction implements a smart approach to handle email thread summarization by:
- Extracting metadata from all emails
- Including the full content of the most recent email
- Conditionally adding parts of the previous email if token budget allows
A few observations:
- The character-to-token estimation ratio of 4:1 is a reasonable approximation but may vary with special characters or languages
- The function handles single emails and short threads appropriately
- The approach prioritizes recent context which is ideal for reply generation
Consider adding handling for empty
threadContentinput with a defensive check at the beginning of the function to avoid potential issues with empty strings.function extractEmailSummary(threadContent: string, maxTokens: number = 4000): string { + if (!threadContent || threadContent.trim() === '') { + return ''; + } // Split the thread into individual emails const emails = threadContent.split('\n---\n'); // ...rest of the function
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
apps/mail/actions/ai-reply.ts(2 hunks)
🔇 Additional comments (8)
apps/mail/actions/ai-reply.ts (8)
5-5: Updated imports to use Groq APIThe imports have been updated to use
generateCompletionsandtruncateThreadContentfrom the Groq library, aligning with the PR objective to transition from OpenAI to Groq API.
70-71: Updated error message for API key configurationError message correctly updated to reference Groq instead of OpenAI.
75-75: Lower token limit for content processingThe token limit has been reduced from 12000 to 3000, which aligns with Groq's model capabilities and helps improve performance.
77-93: Improved system prompt for email generationThe revised system prompt is more structured and provides clearer instructions for generating email replies. It focuses on:
- Creating ready-to-send emails without placeholders
- Eliminating meta-text and formatting instructions
- Maintaining professional tone and conciseness
These changes should result in higher quality email responses that require less cleanup.
96-102: Simplified user promptThe user prompt has been streamlined to provide the necessary context while being more concise. The explicit 200-word limit helps keep responses focused.
114-114: Simplified response handlingThe response handling has been simplified to directly return the completion from the Groq API.
116-117: Error logging and propagationError handling is appropriate with logging to console before propagating the error to the caller.
106-113:Details
❓ Verification inconclusive
Integration with Groq API
The implementation now uses the
generateCompletionsfunction from the Groq library with appropriate parameters:
- Using the llama3-8b-8192 model
- Setting reasonable temperature (0.7) for creativity
- Limiting max tokens to 500 for concise replies
Verify that the
llama3-8b-8192model is the optimal choice for this use case. Other Groq models likemixtral-8x7bmight provide better quality for email generation tasks.
🏁 Script executed:
#!/bin/bash # Check available models in the Groq implementation grep -r "model:" --include="*.ts" --include="*.js" . | grep -v "ai-reply.ts"Length of output: 731
Groq Model Selection Verification
After checking the codebase:
- The default for email generation in
ai-reply.tsis set to"llama3-8b-8192", consistent with the GROQ library defaults (see the use ofGROQ_MODELS.LLAMA_8B).- No alternative Groq models (like
"mixtral-8x7b") appear anywhere in the repository.- While the current implementation intentionally uses
"llama3-8b-8192", its optimality for email replies remains subject to performance and quality testing. If output quality becomes a concern, consider running benchmarks or integration tests against other available models.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
apps/mail/actions/ai-reply.ts (3)
7-55: Good email summarization approach, but token estimation could be improvedThe new
extractEmailSummaryfunction provides a more sophisticated approach to handling email threads than simple truncation, with smart prioritization of recent content.However, the token estimation using
length / 4(lines 41 and 45) is a rough approximation that may not be accurate for all content types, especially non-English text or content with many special characters.Consider using a more accurate token counting method or at least adding a comment explaining this approximation:
- const estimatedSummaryTokens = summary.length / 4; + // Approximate token count (chars/4) - a common heuristic that works for English text + const estimatedSummaryTokens = summary.length / 4;
21-28: Consider adding error handling for malformed email contentWhile the code handles missing subject and sender fields gracefully, there's no logging when these patterns don't match, which could help with debugging issues in production.
const emailMetadata = emails.map((email, index) => { const subjectMatch = email.match(/Subject: (.*?)(\n|$)/i); const fromMatch = email.match(/From: (.*?)(\n|$)/i); + + // Log if we couldn't parse important email fields + if (!subjectMatch || !fromMatch) { + console.warn(`Email parsing incomplete at position ${index}. Missing: ${!subjectMatch ? 'subject' : ''}${!subjectMatch && !fromMatch ? ', ' : ''}${!fromMatch ? 'sender' : ''}`); + } + return { subject: subjectMatch ? subjectMatch[1] : 'No subject', from: fromMatch ? fromMatch[1] : 'Unknown sender' }; });
115-118: Consider more detailed error handlingWhile the current error handling passes through the error from the Groq API, it might be helpful to provide more specific error messages for different failure modes to assist with debugging.
} catch (error: any) { console.error('Error generating AI response:', error); - throw error; + // Provide more context for the error + if (error.message?.includes('rate limit')) { + throw new Error('Rate limit exceeded when generating email reply. Please try again later.'); + } else if (error.message?.includes('context length')) { + throw new Error('Email thread too long for AI processing. Please try with a shorter thread.'); + } else { + throw new Error(`Failed to generate email reply: ${error.message || 'Unknown error'}`); + } }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
apps/mail/actions/ai-reply.ts(2 hunks)
🔇 Additional comments (7)
apps/mail/actions/ai-reply.ts (7)
5-5: Import update aligns with Groq API transitionThe import now correctly fetches
generateCompletionsandtruncateThreadContentfrom the Groq library instead of using OpenAI, which aligns with the PR objective.
13-14: Short-circuit logic looks goodThe condition to bypass complex summarization for short content is a good optimization that helps avoid unnecessary processing.
42-51: Smart handling of previous email contentThe conditional inclusion of previous email content based on remaining token budget is a good approach to maximizing context while respecting limits.
70-72: API key check updated correctlyThe error message now correctly references the Groq API key configuration.
78-93: Improved prompt structure for generating email repliesThe system prompt is now more focused and includes specific instructions that should result in better email replies, particularly by avoiding placeholders and meta-text.
75-75:Details
✅ Verification successful
Verify if the reduced token limit is sufficient
The maximum token limit has been reduced from 12000 (based on PR summary) to 3000 tokens, which is a significant reduction. Make sure this is sufficient for typical email threads in your application.
🏁 Script executed:
#!/bin/bash # Check average email thread length in the codebase or documentation rg -i "token|email.{0,20}length" --glob="*.{ts,js,md}" apps/mail/Length of output: 9169
Based on the verification results, there is no clear usage or documentation in the codebase indicating that typical email threads exceed 3000 tokens. The
extractEmailSummaryfunction defaults to 4000 tokens, and the helper functiontruncateThreadContent(inapps/mail/lib/groq.ts) uses 12000 tokens by default, but most invocations (like in AI reply) now explicitly use 3000 tokens. This suggests that the change is intentional for performance or relevance reasons, and no counter-evidence was found regarding typical thread lengths.Below is the rewritten review comment:
Token Limit Reduction Verification
The call in
apps/mail/actions/ai-reply.tsnow uses a 3000-token limit (previously 12000 as referenced elsewhere). Based on a search of the codebase, we don’t see any evidence that typical email threads exceed 3000 tokens. However, please double-check against production data to ensure this limit remains sufficient in practice.
106-112:Details
✅ Verification successful
Verify the selected Groq model
The code now uses the 'llama3-8b-8192' model. Ensure this is the appropriate model for your use case considering factors like response quality, latency, and cost.
🌐 Web query:
What is llama3-8b-8192 model in Groq API and what are its token limits and performance characteristics?💡 Result:
The Llama3-8B-8192 model on Groq API is a high-speed, cost-efficient implementation of Meta's 8-billion-parameter Llama 3 model, optimized for real-time inference. Here are its key specifications:
Token Limits and Architecture
- 8,192-token context window for processing input sequences, matching its training configuration [1][2].
- Utilizes Grouped-Query Attention (GQA) to maintain inference efficiency comparable to smaller models while handling larger contexts [1][2].
- Despite having 1B more parameters than Llama 2 7B, achieves parity in inference efficiency through improved tokenizer design (15% fewer tokens than Llama 2) [2].
Performance Characteristics
- Industry-leading speed: Generates 1,250 tokens/second, making it one of the fastest implementations for its model size [1].
- Cost structure:
- Input tokens: $0.05 per million
- Output tokens: $0.08 per million [1].
- Optimized for high-throughput applications requiring rapid responses, such as real-time chatbots and API-based services [5].
The model balances performance with Meta's architectural improvements in reasoning and code generation, while Groq's LPU hardware enables deterministic low-latency responses critical for production deployments [2][5].
Citations:
- 1: https://console.groq.com/docs/model/llama3-8b-8192
- 2: https://ai.meta.com/blog/meta-llama-3/
- 3: https://console.groq.com/docs/model/llama3-70b-8192
- 4: https://zilliz.com/tutorials/rag/langchain-and-milvus-and-groq-llama3-70b-8192-and-voyage-code-3
- 5: https://docs.llamaindex.ai/en/stable/examples/llm/groq/
- 6: https://learn.microsoft.com/en-sg/answers/questions/2180782/azure-ai-foundry-completion-token-limit
- 7: https://console.groq.com/docs/model/llama-3.1-8b-instant
- 8: https://docs.llamaindex.ai/en/stable/examples/cookbooks/llama3_cookbook_groq/
- 9: https://huggingface.co/Groq/Llama-3-Groq-8B-Tool-Use
- 10: https://deepinfra.com/docs/advanced/max_tokens_limit
Action: Confirm Groq Model Suitability for AI Reply
Based on the latest Groq API documentation, the
llama3-8b-8192model is a high-speed, cost-efficient implementation of Meta's 8-billion-parameter Llama 3 model. It supports an 8,192-token context window and delivers rapid inference (approximately 1,250 tokens/second) with competitive pricing (input tokens at $0.05/million and output tokens at $0.08/million). These characteristics make it well-suited for real-time, high-throughput applications like the AI reply functionality used in this module.Please ensure that the overall response quality, latency, and cost align with your application's specific requirements. The implementation in
apps/mail/actions/ai-reply.ts(lines 106-112) uses the model assignment correctly.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
apps/mail/lib/groq.ts (2)
69-79: Ensure text length handling for embeddings.
While short texts return quickly, very long texts could be expensive or lead to runtime errors. Consider adding an upper limit for text length to prevent excessive payloads.
326-359: Regex-based cleanup might remove user-intended text.
The regex approach is concise but can remove user content if it matches the patterns. Consider testing edge cases, like emails containing words that match your removal rules.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
apps/mail/lib/groq.ts(1 hunks)
🔇 Additional comments (5)
apps/mail/lib/groq.ts (5)
3-33: Well-structured schema for chat completion.
This Zod schema offers a clear contract for Groq chat completion responses, ensuring robust validation and reducing runtime errors.
34-48: Comprehensive embedding schema definition.
The well-defined fields make embedding validation straightforward. Good job using Zod to enforce expected data shapes.
143-156: Graceful error handling for multiple embeddings.
You are silently catching all embedding errors and continuing. This can be desirable, but be aware that partial failures might complicate downstream usage if one key’s embedding quietly fails.Is this approach intentional to skip problematic texts while processing others?
178-303: Robust error handling and validation for chat completions.
Good use of both HTTP error checks and schema parsing to ensure consistent Groq responses. Any changes in Groq’s response format will be quickly caught.
308-324: Duplicate truncation logic
This function is nearly identical to the one mentioned previously in ai-reply.ts. Please consider centralizing to avoid duplication and reduce maintenance overhead.
3 participants
What Changed
This PR transitions our AI email reply and search functionalities from OpenAI to the Groq API, including:
apps/mail/actions/ai-reply.tsapps/mail/actions/ai-search.tsapps/mail/lib/ai.tsextractEmailSummaryfor better content summarizationtruncateThreadContentfor more efficient token handlingcleanupEmailContentfor better text preparationgenerateCompletionsimport from@/lib/groqWhy This Change
This transition to the Groq API provides several benefits:
Notes for Reviewers
Type of Change
Summary by CodeRabbit
New Features
Refactor
Chores