Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ban lazy imports from airflow and airflow.models in codebase #34600

Closed

Conversation

Taragolis
Copy link
Contributor

Refers to:

This PR show what we should change before we could merge it.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@Taragolis Taragolis added the full tests needed We need to run full set of tests for this PR to merge label Sep 25, 2023
@Taragolis Taragolis changed the title Ban airflow in airflow.models lazy imports in airflow codebase Ban lazy imports from airflow and airflow.models in codebase Sep 25, 2023
@uranusjr
Copy link
Member

I think importing from airflow.models is mostly fine actually, since every one of those models eventually need to pull in most of the models anyway due to how SQLAlchemy’s ORM works. But if the check is simple why not, I guess.

@ephraimbuddy
Copy link
Contributor

I think where we have issue is in importing from models when we are inside the models but every other imports in other places should work but of course, it’s better to ban it everywhere in the codebase, so this is good

Copy link
Member

@hussein-awala hussein-awala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1000

@Taragolis
Copy link
Contributor Author

There are 500+ occurrence lazy imports in airflow codebase, without from airflow import __version__

In general every import should be excluded by up to 3 PRs:

  1. Exclude from core + core tests + example dags
  2. Exclude from providers + providers tests
  3. Ban import in [tool.ruff.flake8-tidy-imports.banned-api]

What I have done with airflow.AirflowException locally:

  1. Create in branch and in first commit was banned in[tool.ruff.flake8-tidy-imports.banned-api]
  2. pre-commit run ruff --all-files
  3. Resolve all cases in core, create second commit
  4. pre-commit run ruff --all-files
  5. Resolve all cases in providers, create third commit(s)
  6. Check again that there is no any static checks failed
  7. Create 3 branches from main and cherry pick changes
  8. Create 3 PRs

Copy link
Contributor

@vincbeck vincbeck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Nov 10, 2023
@github-actions github-actions bot closed this Nov 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
full tests needed We need to run full set of tests for this PR to merge stale Stale PRs per the .github/workflows/stale.yml policy file
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants