docs: Add agentic RAG architecture documentation

[manavgup]

Summary

Add comprehensive architecture documentation for the Agentic RAG Platform. These documents
establish the design foundation for transforming RAG Modulo into a fully agentic system.

Documents Added (6 files, ~3,700 lines)

Document	Description
agentic-ui-architecture.md	React component hierarchy, state management, API integration
backend-architecture-diagram.md	Backend architecture with Mermaid diagrams
mcp-integration-architecture.md	MCP client/server strategy, PR comparison
rag-modulo-mcp-server-architecture.md	RAG as MCP server (tools, resources, auth)
search-agent-hooks-architecture.md	3-stage agent pipeline architecture
system-architecture.md	Complete system architecture overview

Architecture Highlights

3-Stage Agent Pipeline

User Query → Pre-Search Agents → RAG Search → Post-Search Agents → Generation → Response Agents

• Pre-search: Query expansion, translation, intent classification
• Post-search: Re-ranking, deduplication, enrichment
• Response: Artifact generation (PowerPoint, PDF, charts) in parallel

MCP Integration

• Client: Consume external tools via Context Forge
• Server: Expose rag_search, rag_ingest, etc. to Claude Desktop

Implementation Roadmap

These documents guide:

• PR feat: SPIFFE/SPIRE Integration Architecture for Agent Identity #695 (SPIFFE/SPIRE agent identity)
• PR feat(mcp): Add MCP Gateway integration for tool invocation and enrichment #671 (MCP Gateway client)
• Issue feat: Implement SearchService 3-stage agent execution hooks #697 (Agent execution hooks)
• Issue feat: Expose RAG Modulo as MCP Server #698 (MCP Server)
• Issue feat: Agentic UI components for agent configuration and artifacts #699 (Agentic UI)

Test Plan

• All markdown files lint-clean
• Only documentation files in this PR (no code changes)
• Team review for architectural decisions

Closes #696

🤖 Generated with Claude Code

[github-actions]

🚀 Development Environment Options

This repository supports Dev Containers for a consistent development environment.

Option 1: GitHub Codespaces (Recommended)

Create a cloud-based development environment:

1. Click the green Code button above
2. Select the Codespaces tab
3. Click Create codespace on docs/agentic-architecture-696-clean
4. Wait 2-3 minutes for environment setup
5. Start coding with all tools pre-configured!

Option 2: VS Code Dev Containers (Local)

Use Dev Containers on your local machine:

1. Install Docker Desktop
2. Install VS Code
3. Install the Dev Containers extension
4. Clone this PR branch locally
5. Open in VS Code and click "Reopen in Container" when prompted

Option 3: Traditional Local Setup

Set up the development environment manually:

# Clone the repository
git clone https://github.com/manavgup/rag_modulo.git
cd rag_modulo
git checkout docs/agentic-architecture-696-clean

# Initialize development environment
make dev-init
make dev-build
make dev-up
make dev-validate

Available Commands

Once in your development environment:

make help           # Show all available commands
make dev-validate   # Validate environment setup
make test-atomic    # Run atomic tests
make test-unit      # Run unit tests
make lint          # Run linting

Services Available

When running make dev-up:

• Backend API: http://localhost:8000
• Frontend: http://localhost:3000
• MLflow: http://localhost:5001

---

This automated message helps reviewers quickly set up the development environment.

[github-actions]

Code Review: Agentic RAG Architecture Documentation

I've completed a comprehensive review of PR #701, which adds ~3,700 lines of architecture documentation across 6 new files. This is a documentation-only PR with no code changes.

Summary

Comprehensive architecture documentation for transforming RAG Modulo into a fully agentic system.

Files Added:

• agentic-ui-architecture.md (1,470 lines)
• backend-architecture-diagram.md (517 lines)
• mcp-integration-architecture.md (200 lines)
• rag-modulo-mcp-server-architecture.md (689 lines)
• search-agent-hooks-architecture.md (416 lines)
• system-architecture.md (425 lines)

Strengths

1. Excellent Documentation Quality - Clear diagrams, comprehensive code examples, well-organized
2. Thorough 3-Stage Agent Pipeline - Pre-search, Post-search, Response agents with clear separation
3. Complete Component Specifications - TypeScript/Python interfaces, YAML specs, database schemas
4. Security & Identity Integration - SPIFFE/SPIRE, CBAC, authentication flows documented
5. Implementation Guidance - Links to related PRs, phased roadmap

Areas for Improvement

1. Frontend Component Complexity

• Proposes 22 new React components - recommend MVP subset (5-8 components)
• Risk: Frontend could balloon to 5,000+ lines
• Suggest phased approach: Core display → Config UI → Advanced features

2. MCP Integration Complexity

• Dual MCP roles (client + server) adds complexity
• Recommend: Prioritize MCP server first (~40% scope reduction)
• Server enables Claude Desktop integration immediately

3. Database Schema Evolution

• New tables (agent_configs, collection_agents) lack migration strategy
• Missing: Alembic migration script, rollback plan, backward compatibility

4. Error Handling

• High-level strategy mentioned but not fully specified
• Need: Retry policies, circuit breaker thresholds, user notification strategy

5. Performance Concerns

• 3-stage pipeline adds latency without quantification
• Need: Latency budgets, SLOs, performance testing plan
• Suggest: Pre-search <500ms, Post-search <1s, Response <5s

6. Missing Sections

• Testing strategy (E2E, mocks, integration tests)
• Observability details (span names, metrics, dashboards)
• Agent marketplace (discovery, versioning, quality review)
• Deployment strategy (rollout plan, feature flags)

Potential Issues

1. Agent Context Overhead - Passing full conversation history + documents could consume significant memory
2. Agent Chaining - Sequential execution means one failure blocks chain, compounds latency
3. Identity Complexity - Two identity systems (SPIFFE + Bearer tokens) complex to implement
4. Frontend State - Real-time updates, WebSocket management not specified

Code Quality Assessment

Category	Rating	Notes
Documentation	5/5	Extremely thorough
Architecture	4/5	Minor complexity concerns
Implementation	4/5	Missing testing/deployment
Security	4/5	SPIFFE solid, auth complexity
Performance	3/5	Needs SLOs
Error Handling	3/5	Lacks concrete policies

Overall: 4/5 - Excellent foundation, needs refinement

Recommendations

Priority 1: Simplify MVP scope (remove ~40% of components)
Priority 2: Add implementation checklists
Priority 3: Specify performance budgets and SLOs
Priority 4: Document rollout strategy

Final Verdict

APPROVE with recommendations for refinement before implementation.

Excellent architectural foundation with thorough documentation. Before starting implementation:

1. Simplify MVP scope
2. Add performance budgets and SLOs
3. Specify error handling policies
4. Include database migration strategy
5. Add testing and deployment checklists

Estimated effort: MVP 4-6 weeks, Full system 10-12 weeks

Related PRs: #695, #671, #697-699
Closes: #696

Great work!