Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed rows count in the migration script #34348

Merged
merged 5 commits into from
Oct 3, 2023
Merged

Fixed rows count in the migration script #34348

merged 5 commits into from
Oct 3, 2023

Conversation

alartum
Copy link
Contributor

@alartum alartum commented Sep 13, 2023

Changes:

  • replaced .count() method with an already existing get_query_count method;

Motivation:

While performing the Apache Airflow update from 2.2.3 to 2.7.1 I faced an issue with the migration:

$ airflow db migrate
...
File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/__main__.py", line 60, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/commands/db_command.py", line 65, in upgradedb
    migratedb(args)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/cli.py", line 113, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/providers_configuration_loader.py", line 56, in wrapped_function
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/cli/commands/db_command.py", line 104, in migratedb
    db.upgradedb(
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/session.py", line 77, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/db.py", line 1616, in upgradedb
    for err in _check_migration_errors(session=session):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/db.py", line 1499, in _check_migration_errors
    yield from check_fn(session=session)
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/utils/db.py", line 1460, in check_bad_references
    invalid_row_count = bad_rows_query.count()
AttributeError: 'Select' object has no attribute 'count'

The cause of the error is that SQLAlchemy changed the behavior of the select statement. It has already been addressed in other places using func.count(), so the problematic line looks more like a typo/oversight.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg
Copy link

boring-cyborg bot commented Sep 13, 2023

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@alartum alartum marked this pull request as ready for review September 14, 2023 06:55
@@ -1446,7 +1446,7 @@ class BadReferenceConfig:

dangling_table_name = _format_airflow_moved_table_name(source_table.name, change_version, "dangling")
if dangling_table_name in existing_table_names:
invalid_row_count = bad_rows_query.count()
invalid_row_count = get_query_count(bad_rows_query, session=session)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a test to protect future changes here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I will add it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ephraimbuddy I've finally found the time to add the test, although it required more work than expected :)

@eladkal eladkal added this to the Airflow 2.7.2 milestone Oct 1, 2023
@eladkal eladkal added the type:bug-fix Changelog: Bug Fixes label Oct 1, 2023
@@ -0,0 +1 @@
Fixed ``AttributeError: 'Select' object has no attribute 'count'`` during the ``airflow db migrate`` command
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice to have but not required

@ephraimbuddy ephraimbuddy merged commit f349fda into apache:main Oct 3, 2023
@boring-cyborg
Copy link

boring-cyborg bot commented Oct 3, 2023

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

@alartum alartum deleted the fix-migration-sql-count branch October 3, 2023 11:10
ephraimbuddy pushed a commit that referenced this pull request Oct 5, 2023
* Fixed row count for SQLAlchemy 1.4+

* Updated newsfragments

* Fixed typo

* Added newline

* Added test for `check_bad_references`

(cherry picked from commit f349fda)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:bug-fix Changelog: Bug Fixes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants