diff --git a/README.md b/README.md index 1dce8efa..94bfe40b 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ # 🤖 Devr.AI - AI-Powered Developer Relations Assistant - + [![License:MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) ![GitHub Org's stars](https://img.shields.io/github/stars/AOSSIE-Org/Devr.AI?style=social) [![Discord](https://img.shields.io/discord/1022871757289422898?color=7289da&logo=discord&logoColor=white)](https://discord.gg/BjaG8DJx2G) @@ -18,24 +18,28 @@ Devr.AI is revolutionizing open-source community management with advanced AI-pow ## 🚀 Features ### 🧠 LangGraph Agent-Based Intelligence -- **ReAct Reasoning Pattern** - Think → Act → Observe workflow for intelligent decision making + +- **ReAct Reasoning Pattern** - Think → Act → Observe workflow for intelligent decision-making - **Conversational Memory** - Persistent context across Discord sessions with automatic summarization - **Multi-Tool Orchestration** - Dynamic tool selection including web search, FAQ, and GitHub operations - **Self-Correcting Capabilities** - Iterative problem-solving with intelligent context awareness ### 💬 Discord Community Integration + - **Intelligent Message Processing** - Real-time classification and context-aware responses - **GitHub Account Verification** - OAuth-based account linking for enhanced personalization - **Command Interface** - Comprehensive bot commands for verification and management - **Thread Management** - Organized conversation flows with persistent memory ### 🔗 GitHub Integration + - **OAuth Authentication** - Secure GitHub account linking and verification - **User Profiling** - Automatic repository and contribution analysis - **Repository Operations** - Read access and basic GitHub toolkit functionality - **Cross-Platform Identity** - Unified profiles across Discord and GitHub ### 🏗️ Advanced Architecture + - **Asynchronous Processing** - RabbitMQ message queue with priority-based processing - **Multi-Database System** - Supabase (PostgreSQL) + Weaviate (Vector DB) integration - **Real-Time AI Responses** - Google Gemini LLM with Tavily web search capabilities @@ -44,28 +48,33 @@ Devr.AI is revolutionizing open-source community management with advanced AI-pow ## 💻 Technologies Used ### Backend Services + - **LangGraph** - Multi-agent orchestration and workflow management - **FastAPI** - High-performance async web framework - **RabbitMQ** - Message queuing and asynchronous processing - **Google Gemini** - Advanced LLM for reasoning and response generation ### AI & LLM Services + - **Gemini 2.5 Flash** - Primary reasoning and conversation model - **Tavily Search API** - Real-time web information retrieval - **Text Embeddings** - Semantic search and knowledge retrieval - **ReAct Pattern** - Reasoning and Acting workflow implementation ### Data Storage + - **Supabase** - PostgreSQL database with authentication - **Weaviate** - Vector database for semantic search - **Agent Memory** - Persistent conversation context and state management ### Platform Integrations + - **Discord.py (py-cord)** - Modern Discord bot framework - **PyGithub** - GitHub API integration and repository access - **OAuth Integration** - Secure account linking and verification ### Frontend Dashboard + - **React + Vite** - Modern web interface with TypeScript - **Tailwind CSS** - Responsive design system - **Framer Motion** - Interactive UI animations @@ -81,6 +90,7 @@ Devr.AI is revolutionizing open-source community management with advanced AI-pow Devr.AI utilizes a complex multi-service architecture with AI agents, message queues, and multiple databases. Setting up can be challenging, but we've streamlined the process. **Quick Start:** + 1. Clone the repository 2. Follow our comprehensive [Installation Guide](./docs/INSTALL_GUIDE.md) 3. Configure your environment variables (Discord bot, GitHub OAuth, API keys) @@ -106,7 +116,7 @@ For detailed setup instructions, troubleshooting, and deployment guides, please ## 📱 Screenshots
- + | Discord Integration | GitHub Verification | Agent Dashboard | | :----------------------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------- | | | | | @@ -124,6 +134,7 @@ For detailed setup instructions, troubleshooting, and deployment guides, please Thank you for considering contributing to Devr.AI! Contributions are highly appreciated and welcomed. To ensure a smooth collaboration, please refer to our [Contribution Guidelines](./CONTRIBUTING.md). ### Development Setup + 1. Fork the repository 2. Create a feature branch 3. Follow our coding standards and testing guidelines @@ -160,4 +171,52 @@ Thanks a lot for spending your time helping Devr.AI grow. Keep rocking 🥂
Built with ❤️ for the open-source developer community -
\ No newline at end of file +
+ +## 🐳 Docker Compose Setup + +Devr.AI includes a Docker setup for the GitHub MCP server to streamline local development. + +### Running the GitHub MCP Server + +1. **Configure Environment**: Ensure your `.env` file in the root directory has the required variables: + + ```bash + GITHUB_TOKEN=your_token + GITHUB_ORG=your_org + SUPABASE_URL=your_supabase_url + SUPABASE_KEY=your_supabase_key + REDIS_URL=redis://redis:6379/0 # For Docker + ``` + +2. **Start the Service**: + Run the following command from the **root** directory: + + ```bash + docker-compose up --build + ``` + + The service will be available at `http://localhost:5001`. + +3. **Verify Health**: + ```bash + curl http://localhost:5001/health + # Expected: {"status": "healthy", "service": "github-mcp"} + ``` + +### Running Backend Infrastructure + +The database logic and message queues (Weaviate, RabbitMQ, FalkorDB) are managed by a separate Docker Compose file in the `backend` directory. + +To start infrastructure services: + +```bash +cd backend +docker-compose up -d +``` + +### Troubleshooting + +- **Port Conflicts**: The MCP server maps internal port `8001` to external port `5001`. If `5001` is in use, modify `docker-compose.yml`. +- **Environment Variables**: If the container fails to start, check `docker-compose logs github-mcp` to see if tokens are missing. +- **Hot Reloading**: The `backend` directory is mounted to `/app` in the container, so code changes will reload the server automatically. diff --git a/backend/app/core/config/settings.py b/backend/app/core/config/settings.py index 1349a02f..ad91c48e 100644 --- a/backend/app/core/config/settings.py +++ b/backend/app/core/config/settings.py @@ -36,6 +36,10 @@ class Settings(BaseSettings): # RabbitMQ configuration rabbitmq_url: Optional[str] = None + # Redis configuration + # Default is for Docker network (redis:6379). Override with localhost for local dev. + redis_url: str = "redis://redis:6379/0" + # Backend URL backend_url: str = "" diff --git a/backend/app/core/redis.py b/backend/app/core/redis.py new file mode 100644 index 00000000..f957db7d --- /dev/null +++ b/backend/app/core/redis.py @@ -0,0 +1,48 @@ +import redis.asyncio as redis +import asyncio +from app.core.config import settings +import logging +from typing import Optional + +logger = logging.getLogger(__name__) + +class RedisClient: + _instance: Optional[redis.Redis] = None + _lock = asyncio.Lock() + + @classmethod + async def get_client(cls) -> redis.Redis: + if cls._instance is None: + async with cls._lock: + if cls._instance is None: + # Mask credentials for logging + log_url = settings.redis_url + if "@" in log_url: + try: + schema, rest = log_url.split("://", 1) + userinfo, sep, host = rest.rpartition("@") + if sep: + log_url = f"{schema}://****:****@{host}" + else: + log_url = f"{schema}://****:****@..." + except ValueError: + log_url = "redis://****:****@..." + + logger.info(f"Initializing Redis client connecting to {log_url}") + cls._instance = redis.from_url( + settings.redis_url, + encoding="utf-8", + decode_responses=True + ) + return cls._instance + + @classmethod + async def close(cls): + if cls._instance: + await cls._instance.close() + cls._instance = None + logger.info("Redis client closed.") + +async def get_redis_client() -> redis.Redis: + """Dependency to get the Redis client instance.""" + return await RedisClient.get_client() diff --git a/backend/app/services/auth/verification.py b/backend/app/services/auth/verification.py index cbfa156c..335ef847 100644 --- a/backend/app/services/auth/verification.py +++ b/backend/app/services/auth/verification.py @@ -1,48 +1,77 @@ import uuid +import json from datetime import datetime, timedelta -from typing import Optional, Dict, Tuple +from typing import Optional, Dict, Any from app.database.supabase.client import get_supabase_client from app.models.database.supabase import User +from app.core.redis import get_redis_client import logging logger = logging.getLogger(__name__) -# session_id -> (discord_id, expiry_time) -_verification_sessions: Dict[str, Tuple[str, datetime]] = {} +SESSION_EXPIRY_SECONDS = 300 # 5 minutes -SESSION_EXPIRY_MINUTES = 5 - -def _cleanup_expired_sessions(): +class VerificationSessionStore: """ - Remove expired verification sessions. + Redis-backed store for verification sessions. + Handles serialization of session data including datetimes. """ - current_time = datetime.now() - expired_sessions = [ - session_id for session_id, (discord_id, expiry_time) in _verification_sessions.items() - if current_time > expiry_time - ] + PREFIX = "devr:auth:verification:" + + def __init__(self, redis_client: Any): + self.redis = redis_client + + def _get_key(self, session_id: str) -> str: + return f"{self.PREFIX}{session_id}" + + async def save(self, session_id: str, discord_id: str, expiry_time: datetime): + """Save session to Redis with TTL.""" + key = self._get_key(session_id) + data = { + "discord_id": discord_id, + "expiry_time": expiry_time.isoformat() + } + # Redis expects seconds for ex + await self.redis.set( + key, + json.dumps(data), + ex=SESSION_EXPIRY_SECONDS + ) + + async def get(self, session_id: str) -> Optional[Dict[str, Any]]: + """Retrieve session data from Redis.""" + key = self._get_key(session_id) + data_str = await self.redis.get(key) + + if not data_str: + return None + + try: + return json.loads(data_str) + except json.JSONDecodeError: + logger.exception(f"Failed to decode session data for {session_id}") + await self.redis.delete(key) + return None - for session_id in expired_sessions: - discord_id, _ = _verification_sessions[session_id] - del _verification_sessions[session_id] - logger.info(f"Cleaned up expired verification session {session_id} for Discord user {discord_id}") + async def delete(self, session_id: str): + """Delete session from Redis.""" + key = self._get_key(session_id) + await self.redis.delete(key) - if expired_sessions: - logger.info(f"Cleaned up {len(expired_sessions)} expired verification sessions") async def create_verification_session(discord_id: str) -> Optional[str]: """ Create a verification session with expiry and return session ID. """ - supabase = get_supabase_client() - - _cleanup_expired_sessions() + try: + token = str(uuid.uuid4()) + session_id = str(uuid.uuid4()) + expiry_time = datetime.now() + timedelta(seconds=SESSION_EXPIRY_SECONDS) - token = str(uuid.uuid4()) - session_id = str(uuid.uuid4()) - expiry_time = datetime.now() + timedelta(minutes=SESSION_EXPIRY_MINUTES) + supabase = get_supabase_client() + redis_client = await get_redis_client() + store = VerificationSessionStore(redis_client) - try: update_res = await supabase.table("users").update({ "verification_token": token, "verification_token_expires_at": expiry_time.isoformat(), @@ -50,7 +79,7 @@ async def create_verification_session(discord_id: str) -> Optional[str]: }).eq("discord_id", discord_id).execute() if update_res.data: - _verification_sessions[session_id] = (discord_id, expiry_time) + await store.save(session_id, discord_id, expiry_time) logger.info( f"Created verification session {session_id} for Discord user {discord_id}, expires at {expiry_time}") return session_id @@ -60,6 +89,7 @@ async def create_verification_session(discord_id: str) -> Optional[str]: logger.error(f"Error creating verification session for Discord ID {discord_id}: {str(e)}") return None + async def find_user_by_session_and_verify( session_id: str, github_id: str, github_username: str, email: Optional[str] ) -> Optional[User]: @@ -67,17 +97,16 @@ async def find_user_by_session_and_verify( Find and verify user using session ID with expiry validation. Links GitHub account to Discord user. """ - supabase = get_supabase_client() - - _cleanup_expired_sessions() - try: - session_data = _verification_sessions.get(session_id) + supabase = get_supabase_client() + redis_client = await get_redis_client() + store = VerificationSessionStore(redis_client) + session_data = await store.get(session_id) if not session_data: logger.warning(f"No verification session found for session ID: {session_id}") return None - discord_id, expiry_time = session_data + discord_id = session_data["discord_id"] current_time = datetime.now().isoformat() user_res = await supabase.table("users").select("*").eq( @@ -90,17 +119,19 @@ async def find_user_by_session_and_verify( if not user_res.data: logger.warning(f"No valid pending verification found for Discord ID: {discord_id} (token may have expired)") - del _verification_sessions[session_id] + # Clean up Redis just in case + await store.delete(session_id) return None - # Delete the session after successful validation - del _verification_sessions[session_id] + # Atomic consumption: Removed early delete here to prevent race condition. + # We delete AFTER successful database update now. user_to_verify = user_res.data[0] existing_github_user = await supabase.table("users").select("*").eq( "github_id", github_id ).neq("id", user_to_verify['id']).limit(1).execute() + if existing_github_user.data: logger.warning(f"GitHub account {github_username} is already linked to another user") await supabase.table("users").update({ @@ -108,6 +139,7 @@ async def find_user_by_session_and_verify( "verification_token_expires_at": None, "updated_at": datetime.now().isoformat() }).eq("id", user_to_verify['id']).execute() + await store.delete(session_id) raise Exception(f"GitHub account {github_username} is already linked to another Discord user") update_data = { @@ -129,14 +161,19 @@ async def find_user_by_session_and_verify( raise Exception(f"Failed to fetch updated user with ID: {user_to_verify['id']}") logger.info(f"Successfully verified user {user_to_verify['id']} and linked GitHub account {github_username}.") + + # Verify complete: NOW delete the session to prevent replay + await store.delete(session_id) + return User(**updated_user_res.data[0]) - except Exception as e: - logger.error(f"Database error in find_user_by_session_and_verify: {e}", exc_info=True) + except Exception: + logger.exception("Database error in find_user_by_session_and_verify") raise async def cleanup_expired_tokens(): """ Clean up expired verification tokens from database. + Note: Redis sessions expire automatically via TTL. """ supabase = get_supabase_client() current_time = datetime.now().isoformat() @@ -157,20 +194,31 @@ async def get_verification_session_info(session_id: str) -> Optional[Dict[str, s """ Get information about a verification session. """ - _cleanup_expired_sessions() + try: + redis_client = await get_redis_client() + store = VerificationSessionStore(redis_client) - session_data = _verification_sessions.get(session_id) - if not session_data: - return None + session_data = await store.get(session_id) + if not session_data: + return None - discord_id, expiry_time = session_data + discord_id = session_data.get("discord_id") + expiry_time_str = session_data.get("expiry_time") + if not discord_id or not expiry_time_str: + logger.warning(f"Malformed session data for {session_id}") + await store.delete(session_id) + return None - if datetime.now() > expiry_time: - del _verification_sessions[session_id] - return None + expiry_time = datetime.fromisoformat(expiry_time_str) - return { - "discord_id": discord_id, - "expiry_time": expiry_time.isoformat(), - "time_remaining": str(expiry_time - datetime.now()) - } + # Calculate remaining time + now = datetime.now() + + return { + "discord_id": discord_id, + "expiry_time": expiry_time_str, + "time_remaining": str(expiry_time - now) + } + except Exception: + logger.exception("Error getting verification session info") + return None diff --git a/backend/github_mcp_server/Dockerfile b/backend/github_mcp_server/Dockerfile new file mode 100644 index 00000000..2a3b82b3 --- /dev/null +++ b/backend/github_mcp_server/Dockerfile @@ -0,0 +1,26 @@ +FROM python:3.10-slim + +WORKDIR /app + +# Install curl and dependencies in a single layer for optimization +# Install curl and dependencies for optimization +COPY requirements.txt . + +RUN apt-get update && apt-get install -y curl && \ + rm -rf /var/lib/apt/lists/* && \ + pip install --upgrade pip && \ + pip install --no-cache-dir torch --index-url https://download.pytorch.org/whl/cpu && \ + pip install --no-cache-dir -r requirements.txt + +# Copy the entire backend directory to /app +COPY . . + +# Expose the port the app runs on +EXPOSE 8001 + +# Health check (optional but recommended best practice) +HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \ + CMD curl -f http://localhost:8001/health || exit 1 + +# Start the server +CMD ["python", "start_github_mcp_server.py"] diff --git a/backend/github_mcp_server/requirements.txt b/backend/github_mcp_server/requirements.txt new file mode 100644 index 00000000..8fd44efd --- /dev/null +++ b/backend/github_mcp_server/requirements.txt @@ -0,0 +1,16 @@ +fastapi +uvicorn +requests +python-dotenv +pydantic +langgraph==0.4.8 +langchain==0.3.26 +langchain-google-genai==2.1.5 +langchain-core==0.3.66 +ddgs +sqlalchemy +langchain-community +sentence-transformers +weaviate-client +supabase +pydantic-settings diff --git a/backend/requirements.txt b/backend/requirements.txt index 59827539..b846525d 100644 --- a/backend/requirements.txt +++ b/backend/requirements.txt @@ -176,6 +176,7 @@ python-dotenv==1.1.1 pyvis==0.3.2 PyYAML==6.0.2 realtime==2.4.3 +redis>=5.0.0 referencing==0.36.2 regex==2024.11.6 requests==2.32.4 diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 00000000..f9383bb1 --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,38 @@ +version: '3.8' + +services: + github-mcp: + build: + context: ./backend + dockerfile: github_mcp_server/Dockerfile + container_name: github-mcp + ports: + - "5001:8001" + environment: + - GITHUB_TOKEN=${GITHUB_TOKEN} + - GITHUB_ORG=${GITHUB_ORG} + - SUPABASE_URL=${SUPABASE_URL} + - SUPABASE_KEY=${SUPABASE_KEY} + - REDIS_URL=${REDIS_URL} + volumes: + - ./backend:/app + depends_on: + redis: + condition: service_healthy + + redis: + image: redis:alpine + container_name: devr-redis + ports: + - "127.0.0.1:6379:6379" + restart: unless-stopped + volumes: + - redis_data:/data + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 30s + timeout: 10s + retries: 5 + +volumes: + redis_data: diff --git a/env.example b/env.example index 6ed55bcc..4c458376 100644 --- a/env.example +++ b/env.example @@ -5,6 +5,7 @@ TAVILY_API_KEY=your_tavily_api_key_here # Platform Integrations DISCORD_BOT_TOKEN=your_discord_bot_token_here GITHUB_TOKEN=your_github_token_here +GITHUB_ORG=your_github_org_here # Database SUPABASE_URL=your_supabase_url_here @@ -20,6 +21,7 @@ LANGSMITH_PROJECT=DevR_AI # RabbitMQ (optional - uses default if not set) RABBITMQ_URL=amqp://localhost:5672/ +REDIS_URL=redis://localhost:6379/0 # Agent Configuration (optional) DEVREL_AGENT_MODEL=gemini-2.5-flash