Add CI support for SQLAlchemy 2.0 #52233
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:dev-tools
backport-to-v3-1-test
Dev-iL
force-pushed
the
Dev-iL/2506/sqla-v2-tests
branch
from
96d2e8d to
482fff4
Compare
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a new entrypoint function, check_upgrade_sqlalchemy, to handle upgrading SQLAlchemy for testing SQLA v2 while also addressing several shellcheck violations.
- Adds check_upgrade_sqlalchemy implementations to both scripts/docker/entrypoint_ci.sh and Dockerfile.ci.
- Updates Breeze parameters, commands, and CI workflow to include an upgrade_sqlalchemy flag.
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/docker/entrypoint_ci.sh | Adds check_upgrade_sqlalchemy function with minor quoting adjustments. |
| scripts/ci/docker-compose/devcontainer.env | Introduces UPGRADE_SQLALCHEMY environment variable. |
| dev/breeze/src/airflow_breeze/params/shell_params.py | Adds new boolean parameter upgrade_sqlalchemy. |
| dev/breeze/src/airflow_breeze/commands/developer_commands.py | Adds upgrade_sqlalchemy option to developer commands. |
| dev/breeze/src/airflow_breeze/commands/common_options.py | Adds a new click option for upgrade_sqlalchemy with help text. |
| Dockerfile.ci | Adds check_upgrade_sqlalchemy function similar to the entrypoint script. |
| .github/workflows/run-unit-tests.yml | Adds upgrade-sqlalchemy input to the CI workflow. |
Comments suppressed due to low confidence (1)
dev/breeze/src/airflow_breeze/commands/common_options.py:157
- [nitpick] The help text for the upgrade_sqlalchemy option mentions an 'experimental supported version'; consider revising it to clearly indicate that it upgrades SQLAlchemy to the latest version required for running tests, ensuring consistency with the function output.
option_upgrade_sqlalchemy = click.option(
d3dd43d to
d55a749
Compare
Collaborator
Author
|
The tests are failing as expected on SQLA v2 incompatibilities. Now what? Should I suggest in #28723 that developers rebase to my branch? |
Member
|
Yes. And make it a part of your PR and we can continuously rebase the PRs on top of yours. This is going to be a little involved and require rebasing here and there. |
Collaborator
Author
|
Should we perhaps decide on a single "main" branch (e.g. your one) and target the various SQLA2 PRs into it? Just to save the effort of rebasing across multiple forks... |
Dev-iL
force-pushed
the
Dev-iL/2506/sqla-v2-tests
branch
2 times, most recently
from
7bde1ac to
34e7502
Compare
Dev-iL
force-pushed
the
Dev-iL/2506/sqla-v2-tests
branch
5 times, most recently
from
5efe8cb to
9cb0ba2
Compare
Dev-iL
force-pushed
the
Dev-iL/2506/sqla-v2-tests
branch
6 times, most recently
from
67c8353 to
657cce1
Compare
Dev-iL
added a commit
to Dev-iL/airflow
that referenced
this pull request
Aug 10, 2025
This is needed for SQLA2 to work with MySQL 8.4 on python 3.13. Split from apache#52233 for easier review.
eladkal
removed
the
backport-to-v3-1-test
+ Fix some SQLA2 incompatibilities + Skip multiple tests that rely on FAB
Might help with the MySQL 8.4 + Py3.13 combination
- Make some tests more specific so as not trip on irrelevant things. - Add a couple of fixtures to ensure a predictable engine state. - Replace uses of unittest.mock with the mocker fixture. - Add tests for `AutocommitEngineForMySQL`.
Dev-iL
force-pushed
the
Dev-iL/2506/sqla-v2-tests
branch
from
657cce1 to
f44a247
Compare
potiuk
approved these changes
Aug 10, 2025
Member
|
LGTM. The MyPy fix has been fixed separated (leaving some TODOs: in type checking #54321 |
RoyLee1224
pushed a commit
to RoyLee1224/airflow
that referenced
this pull request
Aug 15, 2025
…pache#54316) This is needed for SQLA2 to work with MySQL 8.4 on python 3.13. Split from apache#52233 for easier review.
RoyLee1224
pushed a commit
to RoyLee1224/airflow
that referenced
this pull request
Aug 15, 2025
* CI: Add SQLA2 core & providers workflows * Remove the upper limit of SQLA + Fix some SQLA2 incompatibilities + Skip multiple tests that rely on FAB * Add a 30s start period to all containers to reduce test flakiness * Bump mysql-connector-python to 9.1.0 Might help with the MySQL 8.4 + Py3.13 combination * Set MySQL's `isolation_level` to `AUTOCOMMIT` in SQLA2 during DDL CREATE * Update the `db.py` tests - Make some tests more specific so as not trip on irrelevant things. - Add a couple of fixtures to ensure a predictable engine state. - Replace uses of unittest.mock with the mocker fixture. - Add tests for `AutocommitEngineForMySQL`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes: #48953, #52663
related: #28723, #50960 (prerequisite that's included in this PR)
check_upgrade_sqlalchemycontainer entrypoint for testing SQLA v2 on CI^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.