docs: Add agentic RAG architecture documentation

[manavgup]

Summary

Add comprehensive architecture documentation for the Agentic RAG Platform. These documents
establish the design foundation for transforming RAG Modulo into a fully agentic system.

Documents Added

Document	Lines	Description
agentic-ui-architecture.md	~1,470	React component hierarchy, state management, API integration
backend-architecture-diagram.md	~510	Backend architecture with Mermaid diagrams
mcp-integration-architecture.md	~200	MCP client/server strategy, PR comparison
rag-modulo-mcp-server-architecture.md	~450	RAG as MCP server (tools, resources, auth)
search-agent-hooks-architecture.md	~410	3-stage agent pipeline architecture
system-architecture.md	~410	Complete system architecture overview

Total: ~3,450 lines of documentation

Architecture Highlights

3-Stage Agent Pipeline (search-agent-hooks-architecture.md)

User Query → Pre-Search Agents → RAG Search → Post-Search Agents → Generation → Response Agents → Final Response

• Pre-search: Query expansion, translation, intent classification
• Post-search: Re-ranking, deduplication, enrichment
• Response: Artifact generation (PowerPoint, PDF, charts) in parallel

MCP Integration (mcp-integration-architecture.md)

• RAG Modulo as MCP Client: Consume external tools via Context Forge
• RAG Modulo as MCP Server: Expose rag_search, rag_ingest, etc. to Claude Desktop

Agentic UI (agentic-ui-architecture.md)

• Agent configuration per collection
• Artifact display in search results
• Real-time pipeline status
• Agent marketplace and dashboard

Implementation Roadmap

These documents guide:

• PR feat: SPIFFE/SPIRE Integration Architecture for Agent Identity #695 (SPIFFE/SPIRE agent identity)
• PR feat(mcp): Add MCP Gateway integration for tool invocation and enrichment #671 (MCP Gateway client)
• Issue feat: Implement SearchService 3-stage agent execution hooks #697 (Agent execution hooks)
• Issue feat: Expose RAG Modulo as MCP Server #698 (MCP Server)
• Issue feat: Agentic UI components for agent configuration and artifacts #699 (Agentic UI)

Test Plan

• All markdown files lint-clean (markdownlint passed)
• Cross-references between documents are valid
• Mermaid diagrams render correctly in GitHub
• Team review for architectural decisions

Closes #696

🤖 Generated with Claude Code

[github-actions]

🚀 Development Environment Options

This repository supports Dev Containers for a consistent development environment.

Option 1: GitHub Codespaces (Recommended)

Create a cloud-based development environment:

1. Click the green Code button above
2. Select the Codespaces tab
3. Click Create codespace on docs/agentic-architecture-696
4. Wait 2-3 minutes for environment setup
5. Start coding with all tools pre-configured!

Option 2: VS Code Dev Containers (Local)

Use Dev Containers on your local machine:

1. Install Docker Desktop
2. Install VS Code
3. Install the Dev Containers extension
4. Clone this PR branch locally
5. Open in VS Code and click "Reopen in Container" when prompted

Option 3: Traditional Local Setup

Set up the development environment manually:

# Clone the repository
git clone https://github.com/manavgup/rag_modulo.git
cd rag_modulo
git checkout docs/agentic-architecture-696

# Initialize development environment
make dev-init
make dev-build
make dev-up
make dev-validate

Available Commands

Once in your development environment:

make help           # Show all available commands
make dev-validate   # Validate environment setup
make test-atomic    # Run atomic tests
make test-unit      # Run unit tests
make lint          # Run linting

Services Available

When running make dev-up:

• Backend API: http://localhost:8000
• Frontend: http://localhost:3000
• MLflow: http://localhost:5001

---

This automated message helps reviewers quickly set up the development environment.

[github-actions]

Pull Request Review: Agentic RAG Architecture Documentation & SPIFFE/SPIRE Integration

Overview

This PR introduces comprehensive architecture documentation and a complete SPIFFE/SPIRE identity integration for agent workload authentication. The implementation is ambitious, adding ~9,673 lines across 29 files including core infrastructure, database models, services, and extensive documentation.

---

✅ Strengths

1. Comprehensive Architecture Documentation

• Excellent documentation suite covering all aspects of the agentic RAG system
• Well-structured Mermaid diagrams for visual architecture representation
• Clear cross-references between documents
• Strong alignment with CLAUDE.md guidelines

2. Security-First Design

• Proper SPIFFE/SPIRE integration following CNCF standards
• Zero-trust architecture with cryptographic workload identities
• Capability-based access control system
• Multi-layer defense with JWT-SVID validation
• Security-conscious trust domain restrictions in agent_service.py:84-92

3. Production-Ready Code Quality

• Comprehensive test coverage (772 lines for spiffe_auth, 470 lines for agent_service)
• Proper error handling with custom exceptions
• Type hints throughout the codebase
• Database migrations with rollback support
• Well-documented code with docstrings

4. Database Design Excellence

• Proper composite indexes for query optimization (agent.py:70-74)
• GIN index for JSONB capabilities for efficient containment queries
• Automatic updated_at trigger
• Proper foreign key constraints with CASCADE/SET NULL

5. Clean Architecture

• Clear separation: Models → Repository → Service → Router
• Dependency injection pattern
• Repository pattern for data access
• Pydantic schemas for validation

---

🔍 Issues & Recommendations

CRITICAL Issues

1. Missing Dependency: py-spiffe

Location: backend/core/spiffe_auth.py:342, pyproject.toml

Issue: The code imports py-spiffe library but it's not added to pyproject.toml:

from spiffe import JwtSource, WorkloadApiClient  # type: ignore[import-not-found]

Impact: Runtime ImportError when SPIFFE is enabled

Fix Required:

poetry add py-spiffe
poetry lock

Verification: Check that poetry.lock was regenerated after adding dependency

---

2. Migration Script Missing psycopg2 Dependency Check

Location: migrations/apply_agents_migration.py:14

Issue: Script imports psycopg2 without try/except or dependency declaration

import psycopg2
from dotenv import load_dotenv

Impact: Migration will fail if psycopg2 not installed

Recommendation: Add error handling:

try:
    import psycopg2
except ImportError as e:
    print("ERROR: psycopg2 is required. Install with: pip install psycopg2-binary")
    sys.exit(1)

---

3. Security: Signature Validation Fallback

Location: backend/core/spiffe_auth.py:477-487

Issue: The fallback mode accepts tokens without signature validation in development:

if self.config.fallback_to_jwt:
    logger.warning(
        "SPIRE unavailable, accepting token without signature validation. "
        "This is ONLY safe in development environments."
    )

Concern: While the security note is present, this could be dangerous if accidentally enabled in production.

Recommendation: Add environment check:

if self.config.fallback_to_jwt:
    if os.getenv("ENVIRONMENT", "development") == "production":
        logger.error("SPIRE unavailable in production. Fallback disabled for security.")
        return None
    logger.warning("...")

---

HIGH Priority Issues

4. Authentication Middleware: Agent vs User Confusion

Location: backend/core/authentication_middleware.py:242-244

Issue: Agent authentication sets request.state.user for backward compatibility:

agent_data = {...}
request.state.user = agent_data  # For backward compatibility

Concern: This violates the principle of least surprise. Downstream code checking request.state.user might not expect an agent object.

Recommendation:

• Add request.state.principal for both users and agents
• Keep request.state.user only for actual users
• Update downstream code to check principal first, then user

---

5. Race Condition in Agent Registration

Location: backend/rag_solution/services/agent_service.py:80-81

Issue: Agent instance ID uses UUID prefix without checking uniqueness:

agent_instance_id = str(uuid.uuid4())[:8]

Concern: While collision probability is low, there's no database uniqueness check.

Recommendation: Either:

• Use full UUID for agent_instance_id
• Add a uniqueness retry loop (max 3 attempts)
• Add unique constraint in SPIFFE ID generation

---

6. Missing Index on last_seen_at

Location: migrations/add_agents_table.sql

Issue: last_seen_at is used for activity tracking but lacks an index.

Use Case: Queries like "find inactive agents" will do full table scans.

Recommendation: Add:

CREATE INDEX IF NOT EXISTS idx_agents_last_seen_at ON agents(last_seen_at DESC) 
WHERE last_seen_at IS NOT NULL;

---

7. SPIRE Docker Compose Not Integrated

Location: deployment/spire/docker-compose.spire.yml

Issue: This is a standalone compose file, not integrated with main docker-compose.yml.

Impact: Developers won't know how to run SPIRE with local dev

Recommendation:

• Add docker-compose.override.yml example
• Document integration in CLAUDE.md
• Add make local-dev-spire command

---

MEDIUM Priority Issues

8. Inconsistent Enum Definitions

Location: Multiple files

Issue: AgentType and AgentCapability are defined in 3 places:

• backend/core/spiffe_auth.py (core)
• backend/rag_solution/schemas/agent_schema.py (API layer)
• Tests import from both

Concern: Potential inconsistency and maintenance burden

Recommendation:

• Keep single source of truth in core/spiffe_auth.py
• Import from there in schemas
• OR create core/agent_types.py for shared types

---

9. Missing Error Handling for JWT Decode

Location: backend/core/spiffe_auth.py:433-437

Issue: JWT decode in is_spiffe_jwt_svid() has bare except Exception:

try:
    unverified = jwt.decode(token, options={"verify_signature": False})
    ...
except Exception:
    return False

Concern: Masks all errors, including programming errors

Recommendation: Be specific:

except (jwt.DecodeError, jwt.InvalidTokenError):
    return False
except Exception as e:
    logger.error(f"Unexpected error checking SPIFFE JWT-SVID: {e}")
    return False

---

10. Repository Error Handling - Lost Context

Location: backend/rag_solution/repository/agent_repository.py:82-85

Issue: Generic catch-all loses original exception context:

except Exception as e:
    self.db.rollback()
    logger.error(f"Error creating agent: {e!s}")
    raise RepositoryError(f"Failed to create agent: {e!s}") from e

Recommendation: Handle specific exceptions:

except (IntegrityError, SQLAlchemyError) as e:
    self.db.rollback()
    raise RepositoryError(f"Database error creating agent: {e!s}") from e

---

11. Missing API Documentation

Location: backend/rag_solution/router/agent_router.py

Issue: Endpoints lack OpenAPI examples in docstrings

Impact: API documentation will be less helpful

Recommendation: Add OpenAPI examples:

@router.post(
    "/register",
    response_model=AgentRegistrationResponse,
    responses={
        201: {"description": "Agent registered successfully"},
        400: {"description": "Invalid request", "model": ErrorResponse},
        409: {"description": "SPIFFE ID already exists"},
    }
)

---

12. Type Annotation: Self vs AgentRepository

Location: backend/rag_solution/repository/agent_repository.py:29

Issue: Using Any type hint for self:

def __init__(self: Any, db: Session) -> None:

Recommendation: Remove type hint (not needed) or use proper type:

def __init__(self, db: Session) -> None:

---

LOW Priority / Style Issues

13. Magic String for Trust Domain

Location: Multiple files

Issue: "rag-modulo.example.com" appears in multiple places

Recommendation: Use constant or config default

---

14. Logging: f-string vs Extra Context

Location: backend/rag_solution/services/agent_service.py:86

Issue: Uses f-string instead of structured logging:

logger.warning(f"Rejected trust domain '{request.trust_domain}'...")

Per CLAUDE.md Enhanced Logging guidelines:

logger.warning(
    "Rejected trust domain from user",
    extra={
        "requested_domain": request.trust_domain,
        "configured_domain": self._config.trust_domain,
        "user_id": str(owner_user_id)
    }
)

---

15. Test Fixture Naming

Location: tests/unit/services/test_agent_service.py:35-44

Issue: Mock fixtures lack docstrings

Recommendation: Add brief descriptions for clarity

---

📋 Testing Checklist

Required Before Merge:

• Add py-spiffe to pyproject.toml and regenerate poetry.lock
• Run full test suite (currently 947+ tests) with new changes
• Verify migration script works on clean database
• Test SPIFFE integration end-to-end with SPIRE containers
• Security review of fallback authentication logic
• Load testing for agent authentication at scale
• Documentation review by team (per PR description test plan)

Recommended:

• Add integration test for middleware agent authentication
• Test migration rollback scenario
• Verify all Mermaid diagrams render correctly on GitHub
• Run security scanners (Bandit, Safety) on new code

---

🎯 Summary Assessment

Overall Rating: 8.5/10

Verdict: Approve with Required Changes

This is an exceptionally well-designed and implemented feature. The architecture is sound, the code quality is high, and the documentation is outstanding. However, there are critical dependency issues that must be resolved before merge.

Must Fix Before Merge:

1. ✅ Add py-spiffe dependency to pyproject.toml
2. ✅ Add psycopg2 import error handling to migration script
3. ✅ Add production environment check to signature validation fallback
4. ⚠️ Consider the agent/user authentication backward compatibility approach

Recommended for Follow-Up PR:

• Refactor enum definitions to single source
• Improve structured logging throughout
• Add missing indexes (last_seen_at)
• Integrate SPIRE docker-compose with main setup

---

📚 Alignment with Repository Standards

✅ Follows CLAUDE.md guidelines:

• Production-ready emphasis ✅
• Comprehensive testing ✅
• Clean architecture ✅
• Type hints throughout ✅
• Documentation excellence ✅

⚠️ Minor deviations:

• Enhanced logging not consistently applied (Issue Build and tests fixes #14)
• Line length appears compliant (120 chars)

---

🚀 Impact Assessment

Positive Impact:

• Enables secure agent-to-agent authentication (critical for agentic RAG)
• Production-ready identity infrastructure
• Clear migration path for existing systems
• Excellent foundation for MCP integration (PR feat(mcp): Implement MCP Gateway integration for extensibility #684, feat: SPIFFE/SPIRE Integration Architecture for Agent Identity #695)

Risk Assessment:

• Medium Risk: Missing py-spiffe dependency will cause runtime failures
• Low Risk: Migration script is well-tested and has rollback
• Low Risk: Fallback authentication has clear security warnings

Migration Complexity: Medium

• Requires new database table
• Requires SPIRE infrastructure setup
• Backward compatible with existing JWT auth

---

🤝 Recommendations

For Immediate Action:

1. Add missing dependencies to pyproject.toml
2. Test with make test-all to ensure no regressions
3. Verify SPIRE Docker setup works with make local-dev-spire

For Documentation:

4. Add SPIRE setup guide to CLAUDE.md under "Common Development Commands"
5. Document agent registration workflow in getting-started docs

For Future Enhancement:

6. Consider implementing automatic SPIRE registration via SPIRE Server API
7. Add agent activity monitoring dashboard
8. Implement agent credential rotation automation

---

Great work on this comprehensive feature! The architecture is solid and the implementation quality is high. Address the critical dependency issues and this will be ready to merge.

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com

[manavgup]

Closing - branch included unrelated files from PR #695. Will recreate from clean main branch.