Create FAB's user/role tables on migration, not only on initdb #54227
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
eladkal
reviewed
Aug 7, 2025
.../fab/src/airflow/providers/fab/migrations/versions/0001_1_4_0_create_ab_tables_if_missing.py
Show resolved
Hide resolved
vincbeck
approved these changes
Aug 7, 2025
ashb
commented
Aug 7, 2025
.../fab/src/airflow/providers/fab/migrations/versions/0001_1_4_0_create_ab_tables_if_missing.py
Show resolved
Hide resolved
ashb
added a commit
to astronomer/airflow
that referenced
this pull request
Aug 7, 2025
There were two things blocking this: 1) The revision heads map didn't have any 2.11.x versions in it, so the previous implementation of `_get_version_revision` was only looking within the same <major.minor> pathc version. We change it to rely on the fact that our pre-commit checks ensure this map is ordered, and iterate over the dictionary reversed, and when we find the first thing less than the target revision we use that (direct equal is handled already above) 2) The `ab_*` tables not existing were blocking the migration. Part of this is now fixable manually with apache#54227, but I have decided that since FAB was required and the only option in 2.x, so I have decided to just create the tables if they are missing In order to try and cope with possible future changes I create the tables at the latest version and then downgrade to the oldest known revision. This is all handled in a `reset_to_2_x()` method on the FABDBManager, with a fallback to just blindly create the tables from the ORM for versions of the provider that don't yet have that function.
ashb
added a commit
to astronomer/airflow
that referenced
this pull request
Aug 8, 2025
There were two things blocking this: 1) The revision heads map didn't have any 2.11.x versions in it, so the previous implementation of `_get_version_revision` was only looking within the same <major.minor> pathc version. We change it to rely on the fact that our pre-commit checks ensure this map is ordered, and iterate over the dictionary reversed, and when we find the first thing less than the target revision we use that (direct equal is handled already above) 2) The `ab_*` tables not existing were blocking the migration. Part of this is now fixable manually with apache#54227, but I have decided that since FAB was required and the only option in 2.x, so I have decided to just create the tables if they are missing In order to try and cope with possible future changes I create the tables at the latest version and then downgrade to the oldest known revision. This is all handled in a `reset_to_2_x()` method on the FABDBManager, with a fallback to just blindly create the tables from the ORM for versions of the provider that don't yet have that function.
ashb
added a commit
to astronomer/airflow
that referenced
this pull request
Aug 11, 2025
There were two things blocking this: 1) The revision heads map didn't have any 2.11.x versions in it, so the previous implementation of `_get_version_revision` was only looking within the same <major.minor> pathc version. We change it to rely on the fact that our pre-commit checks ensure this map is ordered, and iterate over the dictionary reversed, and when we find the first thing less than the target revision we use that (direct equal is handled already above) 2) The `ab_*` tables not existing were blocking the migration. Part of this is now fixable manually with apache#54227, but I have decided that since FAB was required and the only option in 2.x, so I have decided to just create the tables if they are missing In order to try and cope with possible future changes I create the tables at the latest version and then downgrade to the oldest known revision. This is all handled in a `reset_to_2_x()` method on the FABDBManager, with a fallback to just blindly create the tables from the ORM for versions of the provider that don't yet have that function.
ashb
added a commit
to astronomer/airflow
that referenced
this pull request
Aug 11, 2025
There were two things blocking this: 1) The revision heads map didn't have any 2.11.x versions in it, so the previous implementation of `_get_version_revision` was only looking within the same <major.minor> pathc version. We change it to rely on the fact that our pre-commit checks ensure this map is ordered, and iterate over the dictionary reversed, and when we find the first thing less than the target revision we use that (direct equal is handled already above) 2) The `ab_*` tables not existing were blocking the migration. Part of this is now fixable manually with apache#54227, but I have decided that since FAB was required and the only option in 2.x, so I have decided to just create the tables if they are missing In order to try and cope with possible future changes I create the tables at the latest version and then downgrade to the oldest known revision. This is all handled in a `reset_to_2_x()` method on the FABDBManager, with a fallback to just blindly create the tables from the ORM for versions of the provider that don't yet have that function.
ashb
added a commit
that referenced
this pull request
Aug 11, 2025
* Allow downgrading to 2.11 from 3.x There were two things blocking this: 1) The revision heads map didn't have any 2.11.x versions in it, so the previous implementation of `_get_version_revision` was only looking within the same <major.minor> pathc version. We change it to rely on the fact that our pre-commit checks ensure this map is ordered, and iterate over the dictionary reversed, and when we find the first thing less than the target revision we use that (direct equal is handled already above) 2) The `ab_*` tables not existing were blocking the migration. Part of this is now fixable manually with #54227, but I have decided that since FAB was required and the only option in 2.x, so I have decided to just create the tables if they are missing In order to try and cope with possible future changes I create the tables at the latest version and then downgrade to the oldest known revision. This is all handled in a `reset_to_2_x()` method on the FABDBManager, with a fallback to just blindly create the tables from the ORM for versions of the provider that don't yet have that function. * Remove `downgrade` from the RunDBManager interface This never made sense, and wasn't actually called as part of the `airflow db downgrade` CLI calls. The reason it doesn't make sense is that the version you pass is either the Airflow version (but external DB managers are installed and versioned separately) or the migration revision ID for the Airflow Core meta db. For FAB specifically there is the `airflow fab-db` CLI command to manage things, so "checking RunDBManager doesn't run Fab migrations" doesn't make sense as a test now (as the code that _could_ do it is removed), so I've removed the test too.
ashb
force-pushed
the
create-fab-tables-on-existing-db
branch
from
41d3278 to
fe12fb6
Compare
ashb
added a commit
to astronomer/airflow
that referenced
this pull request
Aug 11, 2025
There were two things blocking this: 1) The revision heads map didn't have any 2.11.x versions in it, so the previous implementation of `_get_version_revision` was only looking within the same <major.minor> pathc version. We change it to rely on the fact that our pre-commit checks ensure this map is ordered, and iterate over the dictionary reversed, and when we find the first thing less than the target revision we use that (direct equal is handled already above) 2) The `ab_*` tables not existing were blocking the migration. Part of this is now fixable manually with apache#54227, but I have decided that since FAB was required and the only option in 2.x, so I have decided to just create the tables if they are missing In order to try and cope with possible future changes I create the tables at the latest version and then downgrade to the oldest known revision. This is all handled in a `reset_to_2_x()` method on the FABDBManager, with a fallback to just blindly create the tables from the ORM for versions of the provider that don't yet have that function.
ashb
added a commit
that referenced
this pull request
Aug 12, 2025
* Allow downgrading to 2.11 from 3.x There were two things blocking this: 1) The revision heads map didn't have any 2.11.x versions in it, so the previous implementation of `_get_version_revision` was only looking within the same <major.minor> pathc version. We change it to rely on the fact that our pre-commit checks ensure this map is ordered, and iterate over the dictionary reversed, and when we find the first thing less than the target revision we use that (direct equal is handled already above) 2) The `ab_*` tables not existing were blocking the migration. Part of this is now fixable manually with #54227, but I have decided that since FAB was required and the only option in 2.x, so I have decided to just create the tables if they are missing In order to try and cope with possible future changes I create the tables at the latest version and then downgrade to the oldest known revision. This is all handled in a `reset_to_2_x()` method on the FABDBManager, with a fallback to just blindly create the tables from the ORM for versions of the provider that don't yet have that function. * Remove `downgrade` from the RunDBManager interface This never made sense, and wasn't actually called as part of the `airflow db downgrade` CLI calls. The reason it doesn't make sense is that the version you pass is either the Airflow version (but external DB managers are installed and versioned separately) or the migration revision ID for the Airflow Core meta db. For FAB specifically there is the `airflow fab-db` CLI command to manage things, so "checking RunDBManager doesn't run Fab migrations" doesn't make sense as a test now (as the code that _could_ do it is removed), so I've removed the test too.
18 tasks
ashb
added a commit
to astronomer/airflow
that referenced
this pull request
Aug 12, 2025
* Allow downgrading to 2.11 from 3.x There were two things blocking this: 1) The revision heads map didn't have any 2.11.x versions in it, so the previous implementation of `_get_version_revision` was only looking within the same <major.minor> pathc version. We change it to rely on the fact that our pre-commit checks ensure this map is ordered, and iterate over the dictionary reversed, and when we find the first thing less than the target revision we use that (direct equal is handled already above) 2) The `ab_*` tables not existing were blocking the migration. Part of this is now fixable manually with apache#54227, but I have decided that since FAB was required and the only option in 2.x, so I have decided to just create the tables if they are missing In order to try and cope with possible future changes I create the tables at the latest version and then downgrade to the oldest known revision. This is all handled in a `reset_to_2_x()` method on the FABDBManager, with a fallback to just blindly create the tables from the ORM for versions of the provider that don't yet have that function. * Remove `downgrade` from the RunDBManager interface This never made sense, and wasn't actually called as part of the `airflow db downgrade` CLI calls. The reason it doesn't make sense is that the version you pass is either the Airflow version (but external DB managers are installed and versioned separately) or the migration revision ID for the Airflow Core meta db. For FAB specifically there is the `airflow fab-db` CLI command to manage things, so "checking RunDBManager doesn't run Fab migrations" doesn't make sense as a test now (as the code that _could_ do it is removed), so I've removed the test too. (cherry picked from commit 1d04f09)
ashb
added a commit
to astronomer/airflow
that referenced
this pull request
Aug 12, 2025
* Allow downgrading to 2.11 from 3.x There were two things blocking this: 1) The revision heads map didn't have any 2.11.x versions in it, so the previous implementation of `_get_version_revision` was only looking within the same <major.minor> pathc version. We change it to rely on the fact that our pre-commit checks ensure this map is ordered, and iterate over the dictionary reversed, and when we find the first thing less than the target revision we use that (direct equal is handled already above) 2) The `ab_*` tables not existing were blocking the migration. Part of this is now fixable manually with apache#54227, but I have decided that since FAB was required and the only option in 2.x, so I have decided to just create the tables if they are missing In order to try and cope with possible future changes I create the tables at the latest version and then downgrade to the oldest known revision. This is all handled in a `reset_to_2_x()` method on the FABDBManager, with a fallback to just blindly create the tables from the ORM for versions of the provider that don't yet have that function. * Remove `downgrade` from the RunDBManager interface This never made sense, and wasn't actually called as part of the `airflow db downgrade` CLI calls. The reason it doesn't make sense is that the version you pass is either the Airflow version (but external DB managers are installed and versioned separately) or the migration revision ID for the Airflow Core meta db. For FAB specifically there is the `airflow fab-db` CLI command to manage things, so "checking RunDBManager doesn't run Fab migrations" doesn't make sense as a test now (as the code that _could_ do it is removed), so I've removed the test too. (cherry picked from commit 1d04f09)
ashb
added a commit
to astronomer/airflow
that referenced
this pull request
Aug 12, 2025
* Allow downgrading to 2.11 from 3.x There were two things blocking this: 1) The revision heads map didn't have any 2.11.x versions in it, so the previous implementation of `_get_version_revision` was only looking within the same <major.minor> pathc version. We change it to rely on the fact that our pre-commit checks ensure this map is ordered, and iterate over the dictionary reversed, and when we find the first thing less than the target revision we use that (direct equal is handled already above) 2) The `ab_*` tables not existing were blocking the migration. Part of this is now fixable manually with apache#54227, but I have decided that since FAB was required and the only option in 2.x, so I have decided to just create the tables if they are missing In order to try and cope with possible future changes I create the tables at the latest version and then downgrade to the oldest known revision. This is all handled in a `reset_to_2_x()` method on the FABDBManager, with a fallback to just blindly create the tables from the ORM for versions of the provider that don't yet have that function. * Remove `downgrade` from the RunDBManager interface This never made sense, and wasn't actually called as part of the `airflow db downgrade` CLI calls. The reason it doesn't make sense is that the version you pass is either the Airflow version (but external DB managers are installed and versioned separately) or the migration revision ID for the Airflow Core meta db. For FAB specifically there is the `airflow fab-db` CLI command to manage things, so "checking RunDBManager doesn't run Fab migrations" doesn't make sense as a test now (as the code that _could_ do it is removed), so I've removed the test too. (cherry picked from commit 1d04f09)
If we only create these tables on initdb when the DB is empty, it makes it impossible to switch from SimpleAuth manager to FAB Auth manager. This migration was autogenerated by taking the tables created and letting alembic do it for us.
ashb
force-pushed
the
create-fab-tables-on-existing-db
branch
from
fe12fb6 to
141508a
Compare
ashb
added
the
backport-to-v3-1-test
Backport successfully created: v3-0-test
|
ashb
removed
the
backport-to-v3-1-test
54 tasks
RoyLee1224
pushed a commit
to RoyLee1224/airflow
that referenced
this pull request
Aug 15, 2025
* Allow downgrading to 2.11 from 3.x There were two things blocking this: 1) The revision heads map didn't have any 2.11.x versions in it, so the previous implementation of `_get_version_revision` was only looking within the same <major.minor> pathc version. We change it to rely on the fact that our pre-commit checks ensure this map is ordered, and iterate over the dictionary reversed, and when we find the first thing less than the target revision we use that (direct equal is handled already above) 2) The `ab_*` tables not existing were blocking the migration. Part of this is now fixable manually with apache#54227, but I have decided that since FAB was required and the only option in 2.x, so I have decided to just create the tables if they are missing In order to try and cope with possible future changes I create the tables at the latest version and then downgrade to the oldest known revision. This is all handled in a `reset_to_2_x()` method on the FABDBManager, with a fallback to just blindly create the tables from the ORM for versions of the provider that don't yet have that function. * Remove `downgrade` from the RunDBManager interface This never made sense, and wasn't actually called as part of the `airflow db downgrade` CLI calls. The reason it doesn't make sense is that the version you pass is either the Airflow version (but external DB managers are installed and versioned separately) or the migration revision ID for the Airflow Core meta db. For FAB specifically there is the `airflow fab-db` CLI command to manage things, so "checking RunDBManager doesn't run Fab migrations" doesn't make sense as a test now (as the code that _could_ do it is removed), so I've removed the test too.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If we only create these tables on initdb when the DB is empty, it makes it
impossible to switch from SimpleAuth manager to FAB Auth manager.
This migration was autogenerated by taking the tables created and letting
alembic do it for us.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.