feat: Implement OAuth Dynamic Client Registration (DCR) and PKCE support

[manavgup]

Overview

This PR implements OAuth 2.0 Dynamic Client Registration (RFC 7591) and Proof Key for Code Exchange (RFC 7636) support for MCP Gateway, enabling seamless integration with OAuth-protected MCP servers.

Closes #979

🎯 What's Implemented

RFC 7591 - Dynamic Client Registration (DCR)

• ✅ AS metadata discovery (RFC 8414) with caching
• ✅ Automatic client registration when credentials are missing
• ✅ Support for both confidential and public OAuth clients
• ✅ Encrypted storage of registered client credentials
• ✅ Admin endpoints for managing DCR registrations
• ✅ Configurable issuer allowlist for security

RFC 7636 - Proof Key for Code Exchange (PKCE)

• ✅ SHA-256 code_challenge generation from code_verifier
• ✅ Secure code_verifier storage with state parameter
• ✅ PKCE parameters included in authorization URLs
• ✅ code_verifier validation during token exchange
• ✅ Replay attack prevention (single-use state)

Phase 1.4 - Integration

• ✅ Integrated DCR into existing oauth_router.py (no new router needed)
• ✅ Auto-triggers DCR when gateway has issuer but no client_id
• ✅ Automatically sets auth_type=oauth for registered clients
• ✅ Discovers AS endpoints if not manually configured
• ✅ Updates gateway configuration with DCR credentials

📊 Testing

All 53 tests passing (100%)

• 22 PKCE unit tests
• 23 DCR unit tests
• 8 integration tests

Manual validation completed with:

• systemprompt-mcp-server (Reddit MCP) - OAuth-protected real-world server
• Complete DCR + PKCE + OAuth flow tested end-to-end
• Tools successfully fetched from OAuth-protected MCP server

🔧 Configuration

New environment variables:

# Enable DCR
MCPGATEWAY_DCR_ENABLED=true
MCPGATEWAY_DCR_AUTO_REGISTER_ON_MISSING_CREDENTIALS=true

# DCR defaults
MCPGATEWAY_DCR_DEFAULT_SCOPES="mcp:read mcp:tools"
MCPGATEWAY_DCR_CLIENT_NAME_TEMPLATE="MCP Gateway - {gateway_name}"
MCPGATEWAY_DCR_TOKEN_ENDPOINT_AUTH_METHOD="none"  # or "client_secret_basic"

# Security
MCPGATEWAY_DCR_ALLOWED_ISSUERS="https://trusted-as.example.com,https://another-as.example.com"

# OAuth discovery
MCPGATEWAY_OAUTH_DISCOVERY_ENABLED=true
MCPGATEWAY_OAUTH_PREFERRED_CODE_CHALLENGE_METHOD="S256"

🎨 UI Enhancements

• Added "OAuth 2.0" option to auth_type dropdown
• Added oauth_issuer field for DCR configuration
• Made Authorization Code the default grant type (most common for OAuth servers)
• Added help text explaining DCR auto-registration for Client ID/Secret fields
• Backend auto-assembles oauth_config from individual form fields
• Auto-detects auth_type=oauth when OAuth fields are present

🗄️ Database Changes

New Tables:

• registered_oauth_clients - Stores DCR-registered client credentials
• Migration: 2f67b12600b4_add_registered_oauth_clients_table_for_.py

Schema Updates:

• Added code_verifier column to oauth_states table for PKCE
• Migration: 61ee11c482d6_add_code_verifier_to_oauth_states_for_.py

🔒 Security Features

• Client secrets encrypted at rest using Fernet encryption
• PKCE required for public clients (no client_secret)
• State parameter HMAC signing for CSRF protection
• Single-use state parameters (replay attack prevention)
• Code verifier never transmitted (stays on server)
• Configurable issuer allowlist

🚀 Usage Example

1. Register an OAuth-protected MCP server with DCR

# Via UI: Fill in OAuth fields, leave Client ID/Secret empty
# Via API:
curl -X POST http://localhost:4444/admin/gateways \
  -H "Authorization: Bearer $TOKEN" \
  -F "name=Reddit MCP Server" \
  -F "url=http://localhost:3003/mcp" \
  -F "transport=STREAMABLEHTTP" \
  -F "auth_type=oauth" \
  -F "oauth_grant_type=authorization_code" \
  -F "oauth_issuer=http://localhost:3003" \
  -F "oauth_token_url=http://localhost:3003/oauth/token" \
  -F "oauth_authorization_url=http://localhost:3003/oauth/authorize" \
  -F "oauth_redirect_uri=http://localhost:4444/oauth/callback" \
  -F "oauth_scopes=mcp:read mcp:tools"

2. Complete OAuth flow

# Visit the authorization URL
open http://localhost:4444/oauth/authorize/{gateway_id}

# What happens:
# 1. Gateway discovers AS metadata (if not provided)
# 2. Gateway registers client dynamically via DCR
# 3. PKCE parameters generated (code_verifier + code_challenge)
# 4. User redirected to OAuth authorization page
# 5. User authorizes
# 6. Callback exchanges code for token (with code_verifier)
# 7. Tokens stored (encrypted)
# 8. Tools fetched from MCP server

📝 API Endpoints Added

• GET /oauth/registered-clients - List all DCR-registered clients
• GET /oauth/registered-clients/{gateway_id} - Get client for specific gateway
• DELETE /oauth/registered-clients/{client_id} - Delete registered client

🐛 Bug Fixes

• Fixed OAuth field name inconsistencies across components
• Fixed JavaScript in admin.js that was incorrectly assembling oauth_config
• Handle public clients (no client_secret) throughout the flow
• Fixed token decryption for public clients
• Skip URL validation for OAuth servers (validation was too strict)
• Support STREAMABLEHTTP transport for OAuth servers

📚 Documentation

Test documentation files created:

• test_dcr_implementation.sh - Validation script
• All tests include comprehensive docstrings

✅ Checklist

• Implementation follows RFC 7591 (DCR)
• Implementation follows RFC 7636 (PKCE)
• Implementation follows RFC 8414 (AS Metadata)
• All 53 tests passing (100%)
• Manual testing completed with real OAuth server
• Database migrations created
• UI properly supports OAuth configuration
• Public client support (PKCE-only)
• Encrypted credential storage
• Admin endpoints for DCR management
• Integrated into existing OAuth router (no new router)
• Auto-detection of auth_type
• Comprehensive error handling

🔍 Testing Instructions

# Run all DCR/PKCE tests
pytest tests/unit/mcpgateway/services/test_oauth_manager_pkce.py \
       tests/unit/mcpgateway/services/test_dcr_service.py \
       tests/integration/test_dcr_flow_integration.py -v

# Expected: 53 passed

# Run validation script
./test_dcr_implementation.sh

# Manual test with Reddit MCP server
# See TEST_WITH_REDDIT_MCP.md for step-by-step guide

📸 Screenshots

Manual testing successfully completed - DCR registered public client with systemprompt-mcp-server, OAuth flow completed with PKCE, tools fetched successfully.