-
Notifications
You must be signed in to change notification settings - Fork 14.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Do not reset DB in CI tests if not needed #44084
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
potiuk
requested review from
eladkal,
Copilot,
kaxil,
gopidesupavan,
romsharon98 and
shahar1
November 16, 2024 04:07
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Copilot reviewed 10 out of 18 changed files in this pull request and generated no suggestions.
Files not reviewed (8)
- dev/breeze/doc/images/output_shell.txt: Language not supported
- dev/breeze/doc/images/output_start-airflow.txt: Language not supported
- dev/breeze/doc/images/output_testing_core-integration-tests.txt: Language not supported
- dev/breeze/doc/images/output_testing_core-tests.txt: Language not supported
- dev/breeze/doc/images/output_testing_providers-integration-tests.txt: Language not supported
- dev/breeze/doc/images/output_testing_providers-tests.txt: Language not supported
- dev/breeze/doc/images/output_testing_system-tests.txt: Language not supported
- scripts/ci/testing/run_unit_tests.sh: Language not supported
Comments skipped due to low confidence (1)
dev/breeze/src/airflow_breeze/commands/testing_commands.py:1116
- Ensure that the new behavior introduced by 'db_reset=db_reset if not skip_db_tests else False' is covered by tests.
db_reset=db_reset if not skip_db_tests else False
potiuk
force-pushed
the
do-not-reset-db-in-non-db-tests
branch
from
November 16, 2024 04:17
6bf3be4
to
ee8f039
Compare
The apache#43979 refactoring of tests caused unnecessary database reset attempts in tests that did not require it or had no database set. This caused unnecessary `airflow db reset` in collection-only tests with removed non-ARM packages, but also it caused the error printed in non-DB tests: ``` Resetting the DB [2024-11-16T03:50:37.812+0000] {cli_parser.py:67} ERROR - Failed to load CLI commands from executor: LocalExecutor Traceback (most recent call last): File "/opt/airflow/airflow/cli/cli_parser.py", line 64, in <module> executor, _ = ExecutorLoader.import_executor_cls(executor_name) File "/opt/airflow/airflow/executors/executor_loader.py", line 285, in import_executor_cls return _import_and_validate(executor_name.module_path), executor_name.connector_source File "/opt/airflow/airflow/executors/executor_loader.py", line 282, in _import_and_validate cls.validate_database_executor_compatibility(executor) File "/opt/airflow/airflow/executors/executor_loader.py", line 327, in validate_database_executor_compatibility if engine.dialect.name == "sqlite": AttributeError: 'NoneType' object has no attribute 'dialect' [2024-11-16T03:50:37.813+0000] {cli_parser.py:68} ERROR - Ensure all dependencies are met and try again. If using a Celery based executor install a 3.3.0+ version of the Celery provider. If using a Kubernetes executor, install a 7.4.0+ version of the CNCF provider Traceback (most recent call last): File "/usr/local/bin/airflow", line 8, in <module> sys.exit(main()) File "/opt/airflow/airflow/__main__.py", line 62, in main args.func(args) File "/opt/airflow/airflow/cli/cli_config.py", line 49, in command return func(*args, **kwargs) File "/opt/airflow/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function return func(*args, **kwargs) File "/opt/airflow/airflow/cli/commands/db_command.py", line 63, in resetdb print(f"DB: {settings.engine.url!r}") AttributeError: 'NoneType' object has no attribute 'url' Database has been reset ``` The fix is to add `--no-db-reset` in collection tests and force db_reset = False in case `skip_db_tests` is set to True.
potiuk
force-pushed
the
do-not-reset-db-in-non-db-tests
branch
from
November 16, 2024 10:23
ee8f039
to
b622202
Compare
jscheffl
approved these changes
Nov 16, 2024
kandharvishnu
pushed a commit
to kandharvishnu/airflow
that referenced
this pull request
Nov 19, 2024
The apache#43979 refactoring of tests caused unnecessary database reset attempts in tests that did not require it or had no database set. This caused unnecessary `airflow db reset` in collection-only tests with removed non-ARM packages, but also it caused the error printed in non-DB tests: ``` Resetting the DB [2024-11-16T03:50:37.812+0000] {cli_parser.py:67} ERROR - Failed to load CLI commands from executor: LocalExecutor Traceback (most recent call last): File "/opt/airflow/airflow/cli/cli_parser.py", line 64, in <module> executor, _ = ExecutorLoader.import_executor_cls(executor_name) File "/opt/airflow/airflow/executors/executor_loader.py", line 285, in import_executor_cls return _import_and_validate(executor_name.module_path), executor_name.connector_source File "/opt/airflow/airflow/executors/executor_loader.py", line 282, in _import_and_validate cls.validate_database_executor_compatibility(executor) File "/opt/airflow/airflow/executors/executor_loader.py", line 327, in validate_database_executor_compatibility if engine.dialect.name == "sqlite": AttributeError: 'NoneType' object has no attribute 'dialect' [2024-11-16T03:50:37.813+0000] {cli_parser.py:68} ERROR - Ensure all dependencies are met and try again. If using a Celery based executor install a 3.3.0+ version of the Celery provider. If using a Kubernetes executor, install a 7.4.0+ version of the CNCF provider Traceback (most recent call last): File "/usr/local/bin/airflow", line 8, in <module> sys.exit(main()) File "/opt/airflow/airflow/__main__.py", line 62, in main args.func(args) File "/opt/airflow/airflow/cli/cli_config.py", line 49, in command return func(*args, **kwargs) File "/opt/airflow/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function return func(*args, **kwargs) File "/opt/airflow/airflow/cli/commands/db_command.py", line 63, in resetdb print(f"DB: {settings.engine.url!r}") AttributeError: 'NoneType' object has no attribute 'url' Database has been reset ``` The fix is to add `--no-db-reset` in collection tests and force db_reset = False in case `skip_db_tests` is set to True.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The #43979 refactoring of tests caused unnecessary database reset attempts in tests that did not require it or had no database set.
This caused unnecessary
airflow db reset
in collection-only tests with removed non-ARM packages, but also it caused the error printed in non-DB tests:The fix is to add
--no-db-reset
in collection tests and force db_reset = False in caseskip_db_tests
is set to True.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.