Add support for the psycopg3 postgres driver

[Dev-iL]

Motivation

At the time of writing, the postgres driver psycopg2 (hereinafter "PPG2") hasn't been updated for ~9 months. There exists a solution in the form of a new version, psycopg v3 (hereinafter "PPG3"):

Psycopg 3 is the new implementation of Psycopg 2. It emerges from the experience of more than 10 years of development and support of psycopg2. It embraces the new possibilities offered by the more modern generations of the Python language and the PostgreSQL database and addresses the challenges offered by the current patterns in software development and deployment.

After merging #52233, sqlalchemy ("SQLA") v2.x became the default version on python 3.13, which made it possible to experiment with PPG3. SQLA2 supports both PPG2 and PPG3, though it offers some additional features, most notably an async engine, on PPG3.

Summary

This PR adds the optional psycopg[binary] dependency to the postgres provider. As a result, when PPG3 and SQLA2 are both detected, new code branches for business-logic and tests are used. If PPG3 is uninstalled or SQLA1.4 is detected - the existing PPG2-based implementation is used.

It is not currently possible to force the use of PPG2 when SQLA2+PPG3 are present.

Test refactoring

Tests were modified to support dual psycopg versions where possible. In situations that needed significant restructuring between the versions, a version of the test was created under each of the corresponding <TestClass>PPG2 and <TestClass>PPG3 classes.

In a somewhat tangential effort, most of the postgres provider tests were restructured to use the mocker fixture instead of unittest.mock, and the use of caplog was removed in favor of mocking a logger object.

related: #44452

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[Dev-iL]

Most workflows are failing because SQLA1.4 doesn't recognize psycopg v3 as a valid connector. This could change once #52233 is merged, which has the side effect of installing SQLA2 by default on 3.13, meaning the necessary prerequisite to moving to psycopg v3 would be satisfied on that python version. Then we can have the new functionality behind a python version check. I'm not sure if we want to proceed with such dependency divergence, but it does make some sense having a "clean slate" in terms of the dependencies in python 3.13. Then, support of earlier python could either be backported in subsequent PRs or ignored until they're EOL and irrelevant anyway.

[potiuk]

Most workflows are failing because SQLA1.4 doesn't recognize psycopg v3 as a valid connector. This could change once #52233 is merged, which has the side effect of installing SQLA2 by default on 3.13, meaning the necessary prerequisite to moving to psycopg v3 would be satisfied on that python version. Then we can have the new functionality behind a python version check. I'm not sure if we want to proceed with such dependency divergence, but it does make some sense having a "clean slate" in terms of the dependencies in python 3.13. Then, support of earlier python could either be backported in subsequent PRs or ignored until they're EOL and irrelevant anyway.

I think that would really have to wait until FAB 5 is released and we move to SQLA 2 for everything.

[Copilot]

Pull Request Overview

This PR adds support for the psycopg3 PostgreSQL driver as an alternative to psycopg2, enabling compatibility with SQLAlchemy v2 and modernizing the Postgres provider. The implementation automatically detects which driver to use based on availability and SQLAlchemy version.

• Adds psycopg[binary]>=3.2.9 as a dependency alongside existing psycopg2-binary requirements
• Implements dual-driver support with automatic detection logic (USE_PSYCOPG3 flag)
• Refactors tests to support both psycopg2 and psycopg3 with version-specific test classes

Reviewed Changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
providers/postgres/hooks/postgres.py	Core implementation adding psycopg3 support with compatibility layer and protocol definition
providers/postgres/tests/unit/postgres/hooks/test_postgres.py	Comprehensive test refactoring with version-specific test classes and fixture updates
providers/postgres/pyproject.toml	Adds psycopg[binary]>=3.2.9 dependency
providers/google/cloud/transfers/postgres_to_gcs.py	Updates transfer operator to handle psycopg3 differences in cursor handling and JSON types
airflow-core/utils/db.py	Modifies database locking to use set_config() for psycopg3 compatibility
airflow-core/settings.py	Adjusts engine arguments to exclude psycopg2-specific options when using psycopg3

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Dev-iL]

@mik-laj @dabla @olegkachur-e
Is any of you available to review my changes?

[potiuk]

LGTM - one comment about deps.

[github-actions]

Backport failed to create: v3-0-test. View the failure log Run details

Status	Branch	Result
❌	v3-0-test

You can attempt to backport this manually by running:

cherry_picker 7a3c9a9 v3-0-test

This should apply the commit to the v3-0-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

[potiuk]

No backport needed.

[dabla]

Looking forward to test this PR good job!

[potiuk]

Looking forward to test this PR good job!

Indeed :)