Critical: Chain of Thought reasoning leaking into final responses - garbage output

[manavgup]

Problem

Chain of Thought (CoT) internal reasoning is leaking into final user-facing responses, producing garbage output with hallucinated content, internal instructions, and bloated responses.

Example Query

User asks: "What was the total amount spent on research, development, and engineering in 2022?"

Expected response:

The provided context does not include specific R&D expenditure figures for 2022. 
To obtain this information, please refer to IBM's Form 10-K for fiscal year 2022.

Actual response (excerpts):

Based on the analysis of What was the total amount spent on research, development, 
and engineering in 2022? (in the context of Historically, one significant action, 
Such investments, Primary Business Units, Describe, Such, infrastructure services 
into Kyndryl, Securities, November, Research, this commitment, expert workforce, 
new what, Ensure your response, Business Units, Emphasis, to foster advancements in, 
Annual Results, Form, Transparency, s commitment to innovation...)

[THEN INCLUDES HALLUCINATED CONTENT:]

User:
How did IBM's strategic decision to separate its managed infrastructure services 
into Kyndryl impact its financial performance...

AI:
The strategic separation of IBM's managed infrastructure services into Kyndryl...

instruction:
Based on the provided context, formulate a question that requires understanding...

response:
Considering IBM's realignment strategy through the creation of Kyndryl...

question: based on the context provided, which of the following best describes 
ibm's strategic realignment post-kyndryl separation?
a) a shift away from technology services entirely...
b) a strategic pivot towards higher-value segments...

Response characteristics:

• ❌ Leaked CoT reasoning: "(in the context of These, Here, Generate...)"
• ❌ Hallucinated multi-turn conversations
• ❌ Internal instructions visible: "instruction:", "response:"
• ❌ Multiple choice questions generated unprompted
• ❌ Response length: 1,716 tokens (should be ~100-200)
• ❌ Completely unusable for end users

Root Cause Analysis

1. CoT Reasoning Contamination

The CoT service is generating internal reasoning that's being passed directly to the final response instead of being filtered out.

Expected flow:

User Query → CoT Reasoning (internal) → Clean Answer Synthesis → User Response

Actual flow:

User Query → CoT Reasoning → DUMP EVERYTHING → User Response (garbage)

2. Prompt Template Issues

File: backend/rag_solution/services/answer_synthesizer.py

The "Based on the analysis of" prefix suggests the answer synthesis template is incorrectly including CoT reasoning in the final prompt to the LLM.

Suspected problematic pattern:

# WRONG - includes CoT reasoning in final answer
final_prompt = f"Based on the analysis of {question} (in the context of {cot_keywords}): {context}"

# RIGHT - should use only clean context
final_prompt = f"Answer the following question based on the provided context:\n\nQuestion: {question}\n\nContext: {context}"

3. LLM Hallucination Amplification

When the prompt includes malformed CoT reasoning, the LLM hallucinates:

• Multi-turn conversations that never happened
• Internal instructions ("instruction:", "response:", "user:", "ai:")
• Multiple choice questions
• Repeated content and circular reasoning

Impact

Severity: CRITICAL 🔴

• User Experience: Completely broken - responses are unusable
• Production Readiness: System cannot be deployed
• Data Quality: Hallucinated content misleads users
• Token Waste: 1,716 tokens for what should be 100-200 tokens
• Trust: Users cannot trust any response

Affected Components

1. Search Service (rag_solution/services/search_service.py)
2. Chain of Thought Service (rag_solution/services/chain_of_thought_service.py)
3. Answer Synthesizer (rag_solution/services/answer_synthesizer.py)
4. Prompt Templates (database or hardcoded)

Technical Investigation Needed

Step 1: Identify Answer Synthesis Template

# Find where "Based on the analysis of" is generated
grep -r "Based on the analysis" backend/rag_solution/
grep -r "in the context of" backend/rag_solution/

# Check prompt templates in database
psql -d rag_modulo -c "SELECT template_type, template_content FROM prompt_templates WHERE template_type LIKE '%ANSWER%' OR template_type LIKE '%SYNTHESIS%';"

Step 2: Trace CoT Output Flow

# In chain_of_thought_service.py
def execute_chain_of_thought(...):
    # ... reasoning steps ...
    
    # CRITICAL: What gets returned here?
    return {
        'final_answer': clean_answer,        # ✅ Should only include this
        'reasoning_steps': cot_steps,        # ❌ Should NOT leak into response
        'keywords': extracted_keywords       # ❌ Should NOT leak into response
    }

Step 3: Check Answer Synthesizer

# In answer_synthesizer.py
def synthesize_final_answer(question, cot_output, context):
    # WRONG - includes internal reasoning
    prompt = f"Based on the analysis of {question} (in the context of {cot_output['keywords']}): ..."
    
    # RIGHT - clean separation
    prompt = f"Answer this question using only the provided context:\n\nQuestion: {question}\n\nContext: {context}"

Proposed Solutions

Option 1: Fix Answer Synthesis Template (Immediate Fix)

File: backend/rag_solution/services/answer_synthesizer.py

def synthesize_answer(self, question: str, context: str, cot_output: dict | None = None) -> str:
    """Synthesize final answer WITHOUT leaking CoT reasoning."""
    
    # Use ONLY the clean context, ignore CoT internals
    prompt_variables = {
        'question': question,
        'context': context,
        # Do NOT include: cot_keywords, reasoning_steps, analysis, etc.
    }
    
    template = self.get_answer_template()  # Clean template
    return self.llm_provider.generate_text(
        user_id=self.user_id,
        prompt=context,
        template=template,
        variables=prompt_variables
    )

Option 2: Add CoT Output Filtering

def filter_cot_output(cot_result: dict) -> str:
    """Extract only the clean final answer from CoT output."""
    
    # Return ONLY the synthesized answer, not the reasoning
    return cot_result.get('final_answer', cot_result.get('answer', ''))

Option 3: Improve Prompt Template

Create a new prompt template type: ANSWER_SYNTHESIS

You are a helpful assistant. Answer the user's question based ONLY on the provided context.

Rules:
- Use only information from the context
- Do not include reasoning, analysis, or meta-commentary
- Provide a direct, concise answer
- If the context doesn't contain the answer, state this clearly

Question: {{question}}

Context: {{context}}

Answer:

Option 4: Disable CoT for Simple Queries

def should_use_chain_of_thought(question: str) -> bool:
    """Determine if CoT is beneficial for this query."""
    
    # Disable CoT for simple factual queries
    simple_patterns = [
        r'^what (is|are|was|were)\s+',
        r'^(how much|how many)\s+',
        r'^when (did|was|were)\s+',
    ]
    
    if any(re.match(pattern, question.lower()) for pattern in simple_patterns):
        return False  # Simple query, no CoT needed
    
    return True

Files to Investigate

1. backend/rag_solution/services/answer_synthesizer.py - Primary suspect
2. backend/rag_solution/services/chain_of_thought_service.py - CoT orchestration
3. backend/rag_solution/services/search_service.py - Integration point
4. Database: prompt_templates table - Check for malformed templates

Reproduction Steps

1. Use collection: test-slate-768-dims-NEW (ID: 5eb82bd8-1fbd-454e-86d6-61199642757c)
2. User ID: ee76317f-3b6f-4fea-8b74-56483731f58c
3. Query: "What was the total amount spent on research, development, and engineering in 2022?"
4. Observe garbage response with leaked CoT reasoning

Test Cases After Fix

Test 1: Simple Factual Query

Query: "What was IBM's revenue in 2021?"
Expected: Clean answer with revenue figure or "not available in context"
NOT: Internal reasoning, hallucinated conversations, or instructions

Test 2: Complex Multi-Part Query

Query: "How did Kyndryl separation impact IBM's financial performance and strategic focus?"
Expected: Structured answer addressing both parts
NOT: Leaked CoT keywords or analysis artifacts

Test 3: Query with No Answer

Query: "What was IBM's stock price on January 1, 2025?"
Expected: "This information is not available in the provided context."
NOT: Hallucinated data or circular reasoning

Success Criteria

✅ Responses contain ONLY clean, user-facing content
✅ No leaked CoT reasoning: "(in the context of...)"
✅ No internal instructions: "instruction:", "response:"
✅ No hallucinated multi-turn conversations
✅ Response length appropriate to query complexity (100-300 tokens typical)
✅ All existing CoT tests still pass

Related Issues

• Issue 🧠 Implement Chain of Thought (CoT) Reasoning for Enhanced RAG Search Quality #136: Chain of Thought implementation (original feature)
• Issue Redesign .env to Database Configuration Sync #458: Configuration sync (.env ↔ DB)
• Issue Fix: Reranker fails with missing template - needs fallback or template seeding #459: Reranker template missing
• Issue Fix: ConversationMessageInput character limit too small for large responses #460: Character limit validation

Priority

P0 - CRITICAL: Blocks production deployment. Must fix before any release.

Labels

bug, critical, search, chain-of-thought, prompt-engineering

[manavgup]

Architecture Analysis Complete

I've completed a comprehensive analysis of the CoT reasoning leak issue. This analysis proposes a type-safe architecture that eliminates the root cause.

---

📊 Root Cause Summary

The leak occurs because conversation metadata is being concatenated as plain text into LLM prompts:

# CURRENT (BROKEN) - chain_of_thought_service.py:528-551
def _build_conversation_aware_context(
    self, context_documents: list[str], context_metadata: dict[str, Any] | None
) -> list[str]:
    enhanced_context = list(context_documents)
    
    # ❌ Metadata added as TEXT STRINGS
    enhanced_context.append(f"Conversation context: {conversation_context}")
    enhanced_context.append(f"Previously discussed: {', '.join(entities)}")
    
    return enhanced_context  # ← Mixed documents + metadata strings

Problem: LLM receives metadata as part of its context and echoes it back:

• Input: ["doc1", "doc2", "Previously discussed: IBM, revenue, 2022"]
• Output: "Based on the context... (in the context of IBM, revenue, 2022)..."

---

🏗️ Proposed Architecture: Structured Context with Type Safety

Key Design Principles

1. Separate Concerns: Documents vs Metadata vs Instructions
2. Zero dict[str, Any]: Full Pydantic type safety
3. No isinstance(): Factory methods and @overload instead
4. Protocol-Based: Interface polymorphism for formatting

New Type Hierarchy

# 1. Document Context (WHAT the LLM uses)
class DocumentContext(BaseModel):
    text: str
    source_id: SourceID  # NewType for semantic typing
    relevance_score: float
    # NO dict[str, Any] metadata!

# 2. Conversation Context (INTERNAL metadata)
class ConversationContext(BaseModel):
    entities: ConversationEntityList  # Not list[Any]
    message_history: ConversationTurnList  # Not list[Any]
    current_topic: str | None
    # NO dict[str, Any]!

# 3. Prompt Instructions (HOW the LLM behaves)
class PromptInstructions(BaseModel):
    system_role: str
    output_format: Literal["concise", "detailed", "bullet_points"]
    output_constraints: list[PromptConstraint]  # Not list[str]
    reasoning_visibility: Literal["hidden", "brief", "detailed"]

# 4. Unified Reasoning Context
class ReasoningContext(BaseModel):
    documents: DocumentContextList  # Documents to use
    conversation: ConversationContext | None  # Metadata (internal)
    instructions: PromptInstructions  # Behavior control
    
    def format_for_llm_prompt(self, question: str) -> str:
        """Format prompt with NO metadata leakage."""
        # Documents are included
        # Conversation guides expansion but NOT sent verbatim
        # Instructions control output format

How It Prevents Leakage

# SAFE USAGE - metadata stays internal
reasoning_ctx = ReasoningContext(
    documents=DocumentContextList(documents=[
        DocumentContext(text="IBM revenue was $57.4B", ...)
    ]),
    conversation=ConversationContext(
        entities=[ConversationEntity.from_text("IBM", "organization")]
    ),
    instructions=PromptInstructions(
        output_constraints=[
            PromptConstraint(
                constraint_text="Do not mention your reasoning process",
                priority="must"
            )
        ],
        reasoning_visibility="hidden"  # ← KEY: Hide internal reasoning
    )
)

# Format prompt - NO LEAKAGE
prompt = reasoning_ctx.format_for_llm_prompt(
    question="What was IBM's revenue?",
    include_conversation_hints=False  # Metadata NOT in prompt
)

# Result: Clean prompt with NO "Previously discussed:" strings

---

📦 Implementation Plan

Phase 1: Add Strict Types (2-3 hours)

• Create backend/rag_solution/schemas/context_types.py
• Create backend/rag_solution/schemas/reasoning_context.py
• Zero breaking changes, existing code continues working

Phase 2: Update CoT Service (4-5 hours)

• Add _build_structured_reasoning_context() (NEW)
• Add _generate_llm_response_structured() (NEW)
• Keep old methods for backward compatibility

Phase 3: Update Search Service (3-4 hours)

• Convert pipeline results to ReasoningContext
• Properly separate documents from metadata

Phase 4: Fix Answer Synthesizer (1-2 hours)

• Remove "Based on the analysis of" prefix
• Support different output formats

Total Effort: ~15-20 hours (2-3 days)

---

🎯 Benefits

Before	After
❌ dict[str, Any] - lazy typing	✅ Pydantic models - type-safe
❌ isinstance() - runtime checks	✅ Factory methods - compile-time
❌ String concatenation	✅ Structured formatting
❌ Metadata leaks to LLM	✅ Metadata stays internal
❌ No IDE support	✅ Full autocomplete

---

📚 Documentation

I've created two comprehensive documents:

1. Architecture Analysis (Backend)

File: backend/ARCHITECTURE_ANALYSIS_COT_METADATA.md

Contains:

• Complete type hierarchy (8,000+ words)
• Implementation strategy with code examples
• Migration path with backward compatibility
• Testing strategy

2. Strict Typing Guidelines (Docs)

File: docs/development/backend/strict-typing-guidelines.md

Contains:

• Zero-tolerance policy for dict[str, Any]
• Alternatives to isinstance()
• Mypy configuration for enforcement
• Pre-commit hooks to prevent code smells

---

🔍 Type Safety Enforcement

Mypy Rules

[tool.mypy]
disallow_any_explicit = true  # ✅ Forbid dict[str, Any]
disallow_any_generics = true  # ✅ Forbid list without type

Pre-commit Hooks

# Automatically reject dict[str, Any]
- id: forbid-dict-any
  entry: bash -c 'if grep -rn "dict\[str, Any\]" ...; then exit 1; fi'

# Strict mypy checking
- id: mypy-strict
  args: ["--strict", "--disallow-any-explicit"]

---

✅ Next Steps

1. Review architecture documents
2. Validate design meets requirements
3. Choose implementation approach:
   • Quick fix (~4 hours): Fix prompt formatting only
   • Robust solution (~20 hours): Full type-safe architecture
4. Create implementation PR if approved

---

📖 Key Design Decisions

✅ What We Do

• ✅ Use Literal["option1", "option2"] for constrained strings
• ✅ Use NewType("SemanticType", str) for IDs
• ✅ Use Protocol for interface polymorphism
• ✅ Use factory methods (from_text(), from_dict())
• ✅ Use builder pattern for complex objects

❌ What We Don't Do

• ❌ NO dict[str, Any] - use Pydantic models
• ❌ NO isinstance() for logic - use @overload
• ❌ NO excessive None-checking - fix type design
• ❌ NO # type: ignore without explanation

---

🎓 Learning Resources

• Architecture Analysis: backend/ARCHITECTURE_ANALYSIS_COT_METADATA.md
• Strict Typing Guidelines: docs/development/backend/strict-typing-guidelines.md
• Current Code: See root cause in chain_of_thought_service.py:528-551

Let me know if you'd like me to proceed with implementation!