Fix PostgreSQL connection issues with proper pooling and retry logic

[bokelley]

Summary

Resolves production database connection errors that were causing A2A and MCP API calls to fail with psycopg2.OperationalError: server closed the connection unexpectedly errors.

Root Cause

The PostgreSQL database connection was configured without proper connection pooling and retry logic, causing connections to be dropped when idle or under load.

Database Engine Improvements

• PostgreSQL connection pooling: Added production-ready pool settings
  • pool_size=10 - Base connections in pool
  • max_overflow=20 - Additional connections beyond pool_size under load
  • pool_timeout=30 - Seconds to wait for connection from pool
  • pool_recycle=3600 - Recycle connections after 1 hour to prevent stale connections
  • pool_pre_ping=True - Test connections before use to detect disconnects
• SQLite compatibility: Removed unsupported pool options for development environment

Enhanced Error Handling & Retry Logic

• Add specific handling for OperationalError and DisconnectionError
• Implement exponential backoff retry logic with delays: 0.5s, 1s, 2s
• Better session cleanup on connection failures with db_session.remove()
• Distinguish between retryable connection errors and other SQLAlchemy errors

Authentication Layer Hardening

• Wrapped get_principal_from_token() with connection retry logic
• Added retry handling for get_principal_object() lookups
• Improved error logging for database failures with descriptive messages

Production Verification ✅

• Successfully deployed to production (Fly.io version 241)
• Database connection stabilized with proper pooling
• No more psycopg2.OperationalError messages in production logs
• All services (MCP server port 8080, A2A server port 8091, Admin UI port 8001) working correctly
• Database migrations completed successfully: "Database ready (3 tenant(s) configured)"

Test Plan

• Local development SQLite compatibility maintained
• Production PostgreSQL connection pooling tested
• AdCP contract compliance tests passing
• Pre-commit hooks passing (with expected mocking warning)
• Real-world production verification with API calls

Fixes the database connection issues reported in production where A2A and MCP endpoints were returning database connection errors instead of proper responses.