Display token counting and Chain-of-Thought steps in UI

[manavgup]

Feature Request: Token Counting and Chain-of-Thought UI Display

Description

The backend already provides token counting and Chain-of-Thought (CoT) reasoning steps, but these are not prominently displayed in the UI. Users should be able to see token usage and reasoning steps to understand how the AI arrived at its answers.

Acceptance Criteria

• Display token usage prominently in the chat interface
• Show Chain-of-Thought reasoning steps in an expandable section
• Add token usage warnings when approaching limits
• Display token breakdown (input tokens, output tokens, total)
• Show CoT steps with clear numbering and formatting
• Allow users to expand/collapse CoT sections
• Add visual indicators for token usage levels (low, medium, high)
• Show reasoning confidence scores if available
• Add tooltips explaining token usage and CoT concepts

Technical Considerations

• Backend already provides token_warning and cot_output in search responses
• Frontend components for token display and CoT visualization
• UI/UX design for complex reasoning information
• Consider mobile responsiveness for CoT display
• Integration with existing chat interface

Priority

Medium - Enhances transparency and user understanding

Labels

enhancement, frontend, ui, transparency

[manavgup]

@claude read this issue and the related issues. What’s missing?

[claude]

Claude finished @manavgup's task —— View job

---

Analysis of Issue #253: Token Counting and CoT UI Display

Todo List

• Read the current issue and understand requirements
• Search for related issues (token counting, CoT, UI)
• Examine backend implementation for token counting
• Examine backend implementation for CoT output
• Review search response schemas
• Identify gaps in the feature request
• Provide comprehensive analysis

---

📋 Summary

I've analyzed issue #253 and the related codebase. The backend infrastructure is fully implemented and provides comprehensive token tracking and Chain-of-Thought data. However, the feature request is missing several critical technical details needed for implementation.

---

✅ What's Already Implemented (Backend)

Token Tracking System

The backend has a complete token tracking system:

1. TokenWarning Schema (backend/rag_solution/schemas/llm_usage_schema.py:70-98):

   • warning_type: Enum with APPROACHING_LIMIT, CONTEXT_TRUNCATED, AT_LIMIT, CONVERSATION_TOO_LONG
   • current_tokens: Current token count
   • limit_tokens: Token limit for the model
   • percentage_used: Percentage (0-100)
   • message: Human-readable warning message
   • severity: One of info, warning, critical
   • suggested_action: Optional user guidance

2. TokenTrackingService (backend/rag_solution/services/token_tracking_service.py):

   • Automatic threshold detection (70%, 85%, 95%)
   • Per-model context window tracking
   • Per-user and per-session usage accumulation
   • Comprehensive TokenUsageStats with breakdowns by service and model

3. SearchOutput Integration (backend/rag_solution/schemas/search_schema.py:32-59):

   • token_warning: TokenWarning | None - Already included in search responses
   • Populated by SearchService at lines 788-858, 973-995

Chain-of-Thought System

The backend has comprehensive CoT reasoning:

1. cot_output in SearchOutput (backend/rag_solution/schemas/search_schema.py:55):

   • Returns structured reasoning steps when show_cot_steps: true
   • Includes:
     • original_question
     • reasoning_steps[] with:
       • step_number
       • step_question
       • source_attributions[] (document IDs, titles, relevance scores, excerpts)
       • intermediate_answer
       • confidence_score (0-1)
       • reasoning_trace
       • execution_time
     • final_answer
     • total_confidence
     • reasoning_strategy

2. Conversation Message Metadata (backend/rag_solution/schemas/conversation_schema.py:48-62):

   • MessageMetadata includes:
     • cot_used: bool
     • token_count: int
     • execution_time: float
     • confidence_score: float
     • model_used: str
     • conversation_aware: bool

---

❌ What's Missing from the Feature Request

The issue acceptance criteria are too vague for implementation. Here's what needs to be specified:

1. Frontend Technical Stack Not Specified

• Question: What UI framework is the frontend using?
  • The docs mention React 18 + Carbon Design System (CLAUDE.md:11, docs/features/frontend-interface.md)
  • But which specific Carbon components should be used?
  • Are there existing UI component patterns to follow?

2. Token Display Requirements Underspecified

The issue says "display token usage prominently" but doesn't specify:

Missing Details:

• Location: Where exactly should token usage be displayed?

  • Per-message in chat history?
  • In a persistent sidebar/header?
  • In a modal/dialog?
  • All of the above?

• Breakdown Format: The backend provides:

  • current_tokens and limit_tokens (from TokenWarning)
  • total_prompt_tokens, total_completion_tokens, total_tokens (from TokenUsageStats)
  • Per-service and per-model breakdowns

  Which of these should be displayed? Should there be:

  • A simple view (just current/limit)?
  • An expanded view with full breakdown?
  • Different views for different user roles?

• Visual Design:

  • What specific Carbon Design components? (Progress bars? Inline notifications? Badges?)
  • Color scheme for severity levels (the backend provides info/warning/critical)?
  • Should match existing UI patterns - where are those documented?

3. CoT Display Requirements Underspecified

Missing Details:

• Initial State: Should CoT steps be:

  • Collapsed by default?
  • Expanded by default?
  • User preference with persistence?

• Step Display Format:

  • Should each step show ALL fields or a subset?
  • The backend provides 8+ fields per step - which ones matter to users?
  • Should confidence scores be shown as percentages? Progress bars? Color codes?

• Source Attribution Display:

  • Each step has source_attributions[] with document IDs, titles, relevance scores, excerpts
  • Should these be:
    • Shown inline with each step?
    • In a separate expandable section per step?
    • Aggregated at the end?

• Interaction Model:

  • Can users expand individual steps or only all/none?
  • Should there be a "reasoning timeline" visualization?
  • Should steps be collapsible independently?

4. Token Warning Thresholds Missing

The issue mentions "token usage warnings when approaching limits" but:

Missing Details:

• The backend already implements thresholds (70%, 85%, 95%)
• Should the UI use these same thresholds or different ones?
• Should thresholds be:
  • Hard-coded?
  • User-configurable?
  • Admin-configurable?

5. Mobile Responsiveness Not Defined

The issue mentions "Consider mobile responsiveness for CoT display" but:

Missing Details:

• Should CoT be hidden on mobile by default?
• Should there be a simplified mobile view?
• What's the breakpoint for mobile vs. desktop?
• Should token display be different on mobile?

6. User Roles and Permissions Not Addressed

Questions:

• Should all users see token counts or only admins?
• Should there be different displays for different user tiers (free vs. paid)?
• Are there privacy concerns with showing token usage?

7. Real-time vs. Post-response Display

Questions:

• Should token usage update in real-time as the response streams?
• Or only show final counts after response completion?
• The backend supports WebSocket streaming - should token counts stream too?

8. Error Handling Not Specified

Questions:

• What if token_warning is null in the response?
• What if cot_output is null when expected?
• How should the UI handle partial CoT data (e.g., missing confidence scores)?

9. Accessibility Requirements Missing

Questions:

• Should token warnings be announced to screen readers?
• Should CoT steps be keyboard-navigable?
• Are there WCAG requirements to meet?

10. Testing Criteria Not Defined

Questions:

• What constitutes "done" for this feature?
• Should there be unit tests? Integration tests? E2E tests?
• What browsers/devices need to be tested?

11. API Endpoint Documentation

The issue states "Backend already provides token_warning and cot_output" but doesn't specify:

Missing:

• Which API endpoints return this data? (It's the /api/search endpoint, but this should be documented in the issue)
• Request format to enable CoT: config_metadata: {show_cot_steps: true}
• Are there other endpoints that should also display this data?

12. Existing Frontend Code Location

Questions:

• Where is the current search UI code located?
• According to docs/features/frontend-interface.md, there are already accordion components for sources - should token/CoT use the same pattern?
• What TypeScript interfaces currently exist for the search response?

---

🔍 Related Issues/Documentation

Based on my search, there's extensive documentation that should be referenced:

1. CoT Documentation (docs/features/chain-of-thought/):

   • schemas.md - Complete schema reference
   • api-reference.md - API usage
   • examples.md - Implementation examples
   • These should be linked from the issue

2. Frontend Interface Documentation (docs/features/frontend-interface.md):

   • Already mentions "Token Usage Tracking" section (lines 50-64)
   • Already mentions "Chain of Thought Reasoning" section (lines 65-74)
   • But these are aspirational - the doc describes features that may not exist yet

3. Search API Documentation (docs/api/search_api.md, docs/api/search_schemas.md):

   • Should be referenced for complete response schema

---

📝 Recommendations

To make this issue actionable, I recommend:

1. Add Technical Specification Section

Include:

• Target UI framework and specific components
• Mockups or wireframes showing exact placement
• Data mapping from backend schemas to UI display
• Component hierarchy diagram

2. Break Into Smaller Issues

This is really 2-3 separate features:

• Issue A: Token Usage Display Component
• Issue B: Chain-of-Thought Reasoning Display Component
• Issue C: Integration with existing search interface

3. Add Acceptance Criteria Details

For each criterion, specify:

• Given: Initial state
• When: User action
• Then: Expected outcome
• Example: Actual API response and expected UI rendering

4. Reference Existing Patterns

• Link to existing UI components that should be reused
• Reference existing Carbon Design implementations in the codebase
• Specify which existing API response handling code to modify

5. Add Design Assets

• Screenshots or mockups of proposed UI
• Color scheme for severity levels
• Layout diagrams for different screen sizes

6. Define API Contract

Even though backend is implemented, clarify:

• Request format to enable features
• Example request/response pairs
• Error scenarios and handling

---

Would you like me to:

1. Create a more detailed technical specification document?
2. Search for and analyze the existing frontend code to determine current patterns?
3. Create separate, more focused issues for each sub-feature?
4. Draft mockups or component hierarchy diagrams?

---

Branch: claude/issue-253-20251005-1118 | Job Run