SQLAlchemy 2.0 Migration - Stage 1 & 2: Infrastructure and Helper Functions by bokelley · Pull Request #307 · prebid/salesagent

bokelley · 2025-10-07T09:47:07Z

Summary

Completes Stage 1 and Stage 2 of SQLAlchemy 2.0 migration for issue #304.

This PR migrates the database infrastructure layer to use SQLAlchemy 2.0 patterns without breaking any existing functionality.

Changes

Stage 1: Infrastructure Migration (Commit: `a59691c`)

✅ Migrated from declarative_base() to DeclarativeBase class
✅ Removed deprecated autocommit=False and autoflush=False parameters
✅ Updated 4 files:
- src/core/database/models.py - New DeclarativeBase style
- src/core/database/database_session.py - Clean sessionmaker
- src/services/gam_inventory_service.py - Clean sessionmaker
- src/services/gam_orders_service.py - Clean sessionmaker

Stage 2: Helper Function Migration (Commit: `6b9ad9a`)

✅ Migrated get_or_404() to use select() + session.scalars()
✅ Migrated get_or_create() to use select() + session.scalars()
✅ Added select import to database_session.py
✅ Pattern: session.query(Model) → select(Model) with scalars()

Documentation Update (Commit: `9aa269f`)

✅ Added SQLAlchemy 2.0 patterns section to CLAUDE.md
✅ Clear examples of correct vs incorrect patterns
✅ Guidelines: Always use 2.0 for new code, convert legacy when touched
✅ Common conversion patterns documented

Migration Pattern

Before (SQLAlchemy 1.x):
```python
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
instance = session.query(Model).filter_by(field=value).first()
```

After (SQLAlchemy 2.0):
```python
from sqlalchemy.orm import DeclarativeBase
class Base(DeclarativeBase):
pass
SessionLocal = sessionmaker(bind=engine)
stmt = select(Model).filter_by(field=value)
instance = session.scalars(stmt).first()
```

Test Results

✅ 449 unit tests pass
✅ 9 skipped (expected)
✅ No breaking changes
✅ All pre-commit hooks pass

Impact

Zero code changes required in application logic
Zero API changes - all existing code continues to work
Foundation complete for future query pattern migrations
~114 files remain with legacy session.query() patterns (future stages)
Developers now have clear guidance on SQLAlchemy patterns in CLAUDE.md

Next Steps (Future PRs)

Stage 3: Migrate application code (main.py + services)
Stage 4: Migrate Admin UI
Stage 5: Add Mapped[] type annotations and enable mypy plugin

- Replace declarative_base() with DeclarativeBase class in models.py - Remove deprecated autocommit=False and autoflush=False from sessionmaker() - Updated 3 files: database_session.py, gam_inventory_service.py, gam_orders_service.py - No breaking changes - all 449 unit tests pass This is the foundation for SQLAlchemy 2.0 migration per issue #304. Future stages will migrate query patterns to select() + scalars() syntax. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

- Migrated get_or_404() to use select() + session.scalars() - Migrated get_or_create() to use select() + session.scalars() - Added select import to database_session.py - Pattern: session.query(Model).filter_by() → select(Model).filter_by() with scalars() Changes: - Old: instance = session.query(model).filter_by(**kwargs).first() - New: stmt = select(model).filter_by(**kwargs); instance = session.scalars(stmt).first() All 449 unit tests pass ✅ Part of SQLAlchemy 2.0 migration (issue #304) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Document the correct SQLAlchemy 2.0 patterns with clear examples: - ✅ Correct: select() + session.scalars() - ❌ Wrong: session.query() (deprecated) Guidelines for developers: 1. Always use 2.0 patterns for new code 2. Convert legacy code when touching it 3. Common conversion patterns with examples This ensures consistent adoption of SQLAlchemy 2.0 patterns across the codebase as code is written or modified. Related: #304, PR #307 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Migrated 4 core files with 15 query patterns: - config_loader.py: 4 patterns (tenant lookups, subdomain/vhost resolution) - auth_utils.py: 5 patterns (principal/tenant auth lookups) - strategy.py: 5 patterns (strategy CRUD, simulation state management) - audit_logger.py: 2 patterns (tenant name lookups for notifications) All patterns converted from session.query() to select() + scalars(). Includes delete() pattern in strategy.py for reset() function. Related to PR #307 - SQLAlchemy 2.0 migration initiative. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Migrated database.py with 1 count pattern: - Converted session.query(Tenant).count() to select(func.count()).select_from(Tenant) - Used scalar() instead of execute() for count queries Total core files migrated: 5 files, 16 patterns converted. Related to PR #307 - SQLAlchemy 2.0 migration initiative. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

…chemy 2.0 Migrated 2 service files with 2 query patterns: - format_metrics_service.py: 1 pattern (upsert query for metrics) - dynamic_pricing_service.py: 1 pattern (query with chained filters) Converted query().filter() to select().where() with scalars().all()/first(). Related to PR #307 - SQLAlchemy 2.0 migration initiative. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

Migrated push_notification_service.py with 5 query patterns: - Converted all .query() calls to select() + scalars() - Patterns include: filter_by, filter with and_(), and in_() clauses - Fixed import order (ruff) All query patterns now use SQLAlchemy 2.0 style with explicit stmt variables. Related to PR #307 - SQLAlchemy 2.0 migration initiative. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

* Stage 5: Migrate src/core/ files to SQLAlchemy 2.0 Migrated 4 core files with 15 query patterns: - config_loader.py: 4 patterns (tenant lookups, subdomain/vhost resolution) - auth_utils.py: 5 patterns (principal/tenant auth lookups) - strategy.py: 5 patterns (strategy CRUD, simulation state management) - audit_logger.py: 2 patterns (tenant name lookups for notifications) All patterns converted from session.query() to select() + scalars(). Includes delete() pattern in strategy.py for reset() function. Related to PR #307 - SQLAlchemy 2.0 migration initiative. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Stage 5: Migrate src/core/database/ files to SQLAlchemy 2.0 Migrated database.py with 1 count pattern: - Converted session.query(Tenant).count() to select(func.count()).select_from(Tenant) - Used scalar() instead of execute() for count queries Total core files migrated: 5 files, 16 patterns converted. Related to PR #307 - SQLAlchemy 2.0 migration initiative. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Stage 5: Migrate format_metrics and dynamic_pricing services to SQLAlchemy 2.0 Migrated 2 service files with 2 query patterns: - format_metrics_service.py: 1 pattern (upsert query for metrics) - dynamic_pricing_service.py: 1 pattern (query with chained filters) Converted query().filter() to select().where() with scalars().all()/first(). Related to PR #307 - SQLAlchemy 2.0 migration initiative. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Stage 5: Migrate push_notification_service to SQLAlchemy 2.0 Migrated push_notification_service.py with 5 query patterns: - Converted all .query() calls to select() + scalars() - Patterns include: filter_by, filter with and_(), and in_() clauses - Fixed import order (ruff) All query patterns now use SQLAlchemy 2.0 style with explicit stmt variables. Related to PR #307 - SQLAlchemy 2.0 migration initiative. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Stage 5: Fix remaining .query() patterns in config_loader and update docs in database_session * Stage 5: Migrate GAM service files to SQLAlchemy 2.0 (24 patterns) Converted all `.query()` patterns in: - src/services/gam_inventory_service.py (17 patterns) - src/services/gam_orders_service.py (7 patterns) Changes: - Added `select`, `delete`, `update` imports - Converted `.query(Model)` → `stmt = select(Model)` - Converted `.filter()` → `.where()` - Converted `.all()/.first()` → `session.scalars(stmt).all()/.first()` - Converted `.update()` → `update(Model).values()` - Converted `.delete()` → `delete(Model).where()` - Converted `.count()` → `session.scalar(select(func.count()))` - Maintained readable formatting with proper variable names All conversions follow SQLAlchemy 2.0 best practices. * Stage 5: Migrate adapter files to SQLAlchemy 2.0 (18 patterns) Converted all `.query()` patterns in: - src/adapters/google_ad_manager.py (1 pattern) - src/adapters/mock_ad_server.py (1 pattern) - src/adapters/xandr.py (1 pattern) - src/adapters/gam_reporting_api.py (8 patterns) - src/adapters/gam/managers/sync.py (6 patterns) - src/adapters/gam/managers/workflow.py (1 pattern) Changes: - Added `select` imports (and `func` where needed for count operations) - Converted `.query(Model)` → `stmt = select(Model)` - Converted `.filter_by()` → `.filter_by()` (still supported in select) - Converted `.where()` for more complex filters - Converted `.all()/.first()` → `session.scalars(stmt).all()/.first()` - Converted `.count()` → `session.scalar(select(func.count()).select_from(Model).where(...))` - Used `replace_all=true` for repeated patterns in gam_reporting_api.py All conversions follow SQLAlchemy 2.0 best practices. * Stage 5: Migrate a2a_server file to SQLAlchemy 2.0 (4 patterns) Converted all `.query()` patterns in: - src/a2a_server/adcp_a2a_server.py (4 patterns) Changes: - Added `select` import from sqlalchemy - Converted `.query(DBPushNotificationConfig)` → `stmt = select(DBPushNotificationConfig)` - Converted `.filter_by()` → `.filter_by()` (chained with select) - Converted `.all()/.first()` → `db.scalars(stmt).all()/.first()` All push notification config queries now use SQLAlchemy 2.0 patterns. * Stage 5: Fix test to use scalars() mock for SQLAlchemy 2.0 Updated test_virtual_host_edge_cases.py to mock scalars() instead of query() to align with SQLAlchemy 2.0 patterns used in config_loader.py. * Stage 5: Complete test fixes for SQLAlchemy 2.0 Updated remaining test assertions in test_virtual_host_edge_cases.py: - Removed outdated mock.query assertion in SQL injection test - Updated corrupted tenant test to mock scalars() chain - Tests now align with SQLAlchemy 2.0 patterns (select + scalars) * Stage 5: Migrate google_ad_manager_original (legacy) to SQLAlchemy 2.0 (13 patterns) Converted all remaining .query() patterns in google_ad_manager_original.py: - Pattern 1 (line 405): Product lookup with filter_by - Pattern 2 (line 1834): CreativeFormat with filter, order_by, and in_() - Pattern 3 (line 2454): WorkflowStep with filter_by - Pattern 4 (line 2774): Tenant lookup with filter_by - Pattern 5 (line 2779): Product lookup with filter_by - Pattern 6 (line 2792): AdapterConfig with filter_by - Pattern 7 (line 2903): Product update with filter_by - Pattern 8 (line 2973): GAMInventory count with where - Pattern 9 (line 2989): GAMInventory all with where and and_() - Pattern 10 (line 3002): GAMInventory all with where and and_() - Pattern 11 (line 3046): GAMInventory all with where, and_(), and limit - Pattern 12 (line 3065): GAMInventory all with where, and_(), and limit - Pattern 13 (line 3089): GAMInventory column select with where and order_by (fixed typo: OrderBy → order_by) All patterns now use SQLAlchemy 2.0 API: - select() instead of query() - where() instead of filter() - scalars().first()/all() for entity queries - execute().first() for column-only queries - scalar() for count queries Note: This file appears to be legacy/deprecated (not imported anywhere). Migrated for completeness. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Stage 5: Fix syntax errors from automated test migration Fixed IndentationError in 5 test files caused by misplaced SQLAlchemy imports: - tests/smoke/test_smoke_critical_paths.py (line 482) - tests/integration/test_create_media_buy_v24.py (line 314) - tests/integration/test_gam_tenant_setup.py (line 227) - tests/integration/test_product_deletion.py (line 407) - tests/integration/test_tenant_management_api_integration.py (line 49) Also fixed SyntaxError in tests/manual/test_gam_automation_real.py: - Changed Product.tenant_id= to Product.tenant_id== in where() clauses Root cause: Automated migration script added imports in wrong location and used assignment operator instead of comparison operator in where(). All files now compile successfully. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>

…ctions (prebid#307) * Stage 1: SQLAlchemy 2.0 infrastructure migration - Replace declarative_base() with DeclarativeBase class in models.py - Remove deprecated autocommit=False and autoflush=False from sessionmaker() - Updated 3 files: database_session.py, gam_inventory_service.py, gam_orders_service.py - No breaking changes - all 449 unit tests pass This is the foundation for SQLAlchemy 2.0 migration per issue prebid#304. Future stages will migrate query patterns to select() + scalars() syntax. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Stage 2: SQLAlchemy 2.0 helper function migration - Migrated get_or_404() to use select() + session.scalars() - Migrated get_or_create() to use select() + session.scalars() - Added select import to database_session.py - Pattern: session.query(Model).filter_by() → select(Model).filter_by() with scalars() Changes: - Old: instance = session.query(model).filter_by(**kwargs).first() - New: stmt = select(model).filter_by(**kwargs); instance = session.scalars(stmt).first() All 449 unit tests pass ✅ Part of SQLAlchemy 2.0 migration (issue prebid#304) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Add SQLAlchemy 2.0 query patterns to CLAUDE.md Document the correct SQLAlchemy 2.0 patterns with clear examples: - ✅ Correct: select() + session.scalars() - ❌ Wrong: session.query() (deprecated) Guidelines for developers: 1. Always use 2.0 patterns for new code 2. Convert legacy code when touching it 3. Common conversion patterns with examples This ensures consistent adoption of SQLAlchemy 2.0 patterns across the codebase as code is written or modified. Related: prebid#304, PR prebid#307 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>

* Stage 5: Migrate src/core/ files to SQLAlchemy 2.0 Migrated 4 core files with 15 query patterns: - config_loader.py: 4 patterns (tenant lookups, subdomain/vhost resolution) - auth_utils.py: 5 patterns (principal/tenant auth lookups) - strategy.py: 5 patterns (strategy CRUD, simulation state management) - audit_logger.py: 2 patterns (tenant name lookups for notifications) All patterns converted from session.query() to select() + scalars(). Includes delete() pattern in strategy.py for reset() function. Related to PR prebid#307 - SQLAlchemy 2.0 migration initiative. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Stage 5: Migrate src/core/database/ files to SQLAlchemy 2.0 Migrated database.py with 1 count pattern: - Converted session.query(Tenant).count() to select(func.count()).select_from(Tenant) - Used scalar() instead of execute() for count queries Total core files migrated: 5 files, 16 patterns converted. Related to PR prebid#307 - SQLAlchemy 2.0 migration initiative. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Stage 5: Migrate format_metrics and dynamic_pricing services to SQLAlchemy 2.0 Migrated 2 service files with 2 query patterns: - format_metrics_service.py: 1 pattern (upsert query for metrics) - dynamic_pricing_service.py: 1 pattern (query with chained filters) Converted query().filter() to select().where() with scalars().all()/first(). Related to PR prebid#307 - SQLAlchemy 2.0 migration initiative. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Stage 5: Migrate push_notification_service to SQLAlchemy 2.0 Migrated push_notification_service.py with 5 query patterns: - Converted all .query() calls to select() + scalars() - Patterns include: filter_by, filter with and_(), and in_() clauses - Fixed import order (ruff) All query patterns now use SQLAlchemy 2.0 style with explicit stmt variables. Related to PR prebid#307 - SQLAlchemy 2.0 migration initiative. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Stage 5: Fix remaining .query() patterns in config_loader and update docs in database_session * Stage 5: Migrate GAM service files to SQLAlchemy 2.0 (24 patterns) Converted all `.query()` patterns in: - src/services/gam_inventory_service.py (17 patterns) - src/services/gam_orders_service.py (7 patterns) Changes: - Added `select`, `delete`, `update` imports - Converted `.query(Model)` → `stmt = select(Model)` - Converted `.filter()` → `.where()` - Converted `.all()/.first()` → `session.scalars(stmt).all()/.first()` - Converted `.update()` → `update(Model).values()` - Converted `.delete()` → `delete(Model).where()` - Converted `.count()` → `session.scalar(select(func.count()))` - Maintained readable formatting with proper variable names All conversions follow SQLAlchemy 2.0 best practices. * Stage 5: Migrate adapter files to SQLAlchemy 2.0 (18 patterns) Converted all `.query()` patterns in: - src/adapters/google_ad_manager.py (1 pattern) - src/adapters/mock_ad_server.py (1 pattern) - src/adapters/xandr.py (1 pattern) - src/adapters/gam_reporting_api.py (8 patterns) - src/adapters/gam/managers/sync.py (6 patterns) - src/adapters/gam/managers/workflow.py (1 pattern) Changes: - Added `select` imports (and `func` where needed for count operations) - Converted `.query(Model)` → `stmt = select(Model)` - Converted `.filter_by()` → `.filter_by()` (still supported in select) - Converted `.where()` for more complex filters - Converted `.all()/.first()` → `session.scalars(stmt).all()/.first()` - Converted `.count()` → `session.scalar(select(func.count()).select_from(Model).where(...))` - Used `replace_all=true` for repeated patterns in gam_reporting_api.py All conversions follow SQLAlchemy 2.0 best practices. * Stage 5: Migrate a2a_server file to SQLAlchemy 2.0 (4 patterns) Converted all `.query()` patterns in: - src/a2a_server/adcp_a2a_server.py (4 patterns) Changes: - Added `select` import from sqlalchemy - Converted `.query(DBPushNotificationConfig)` → `stmt = select(DBPushNotificationConfig)` - Converted `.filter_by()` → `.filter_by()` (chained with select) - Converted `.all()/.first()` → `db.scalars(stmt).all()/.first()` All push notification config queries now use SQLAlchemy 2.0 patterns. * Stage 5: Fix test to use scalars() mock for SQLAlchemy 2.0 Updated test_virtual_host_edge_cases.py to mock scalars() instead of query() to align with SQLAlchemy 2.0 patterns used in config_loader.py. * Stage 5: Complete test fixes for SQLAlchemy 2.0 Updated remaining test assertions in test_virtual_host_edge_cases.py: - Removed outdated mock.query assertion in SQL injection test - Updated corrupted tenant test to mock scalars() chain - Tests now align with SQLAlchemy 2.0 patterns (select + scalars) * Stage 5: Migrate google_ad_manager_original (legacy) to SQLAlchemy 2.0 (13 patterns) Converted all remaining .query() patterns in google_ad_manager_original.py: - Pattern 1 (line 405): Product lookup with filter_by - Pattern 2 (line 1834): CreativeFormat with filter, order_by, and in_() - Pattern 3 (line 2454): WorkflowStep with filter_by - Pattern 4 (line 2774): Tenant lookup with filter_by - Pattern 5 (line 2779): Product lookup with filter_by - Pattern 6 (line 2792): AdapterConfig with filter_by - Pattern 7 (line 2903): Product update with filter_by - Pattern 8 (line 2973): GAMInventory count with where - Pattern 9 (line 2989): GAMInventory all with where and and_() - Pattern 10 (line 3002): GAMInventory all with where and and_() - Pattern 11 (line 3046): GAMInventory all with where, and_(), and limit - Pattern 12 (line 3065): GAMInventory all with where, and_(), and limit - Pattern 13 (line 3089): GAMInventory column select with where and order_by (fixed typo: OrderBy → order_by) All patterns now use SQLAlchemy 2.0 API: - select() instead of query() - where() instead of filter() - scalars().first()/all() for entity queries - execute().first() for column-only queries - scalar() for count queries Note: This file appears to be legacy/deprecated (not imported anywhere). Migrated for completeness. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Stage 5: Fix syntax errors from automated test migration Fixed IndentationError in 5 test files caused by misplaced SQLAlchemy imports: - tests/smoke/test_smoke_critical_paths.py (line 482) - tests/integration/test_create_media_buy_v24.py (line 314) - tests/integration/test_gam_tenant_setup.py (line 227) - tests/integration/test_product_deletion.py (line 407) - tests/integration/test_tenant_management_api_integration.py (line 49) Also fixed SyntaxError in tests/manual/test_gam_automation_real.py: - Changed Product.tenant_id= to Product.tenant_id== in where() clauses Root cause: Automated migration script added imports in wrong location and used assignment operator instead of comparison operator in where(). All files now compile successfully. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>

bokelley and others added 3 commits October 6, 2025 20:03

bokelley merged commit bb50957 into main Oct 7, 2025
8 checks passed

This was referenced Oct 7, 2025

Fix authorized domains display showing character-by-character #305

Merged

SQLAlchemy 2.0 Migration - Stage 3: Application Code #308

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SQLAlchemy 2.0 Migration - Stage 1 & 2: Infrastructure and Helper Functions#307

SQLAlchemy 2.0 Migration - Stage 1 & 2: Infrastructure and Helper Functions#307
bokelley merged 3 commits intomainfrom
bokelley/sqlalchemy-2.0-migration

bokelley commented Oct 7, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

bokelley commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Stage 1: Infrastructure Migration (Commit: a59691c)

Stage 2: Helper Function Migration (Commit: 6b9ad9a)

Documentation Update (Commit: 9aa269f)

Migration Pattern

Test Results

Impact

Next Steps (Future PRs)

Related

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

bokelley commented Oct 7, 2025 •

edited

Loading

Stage 1: Infrastructure Migration (Commit: `a59691c`)

Stage 2: Helper Function Migration (Commit: `6b9ad9a`)

Documentation Update (Commit: `9aa269f`)