Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[airflow] Extend airflow context parameter check for BaseOperator.execute (AIR302) #15713

Merged
merged 9 commits into from
Jan 27, 2025

Conversation

Lee-W
Copy link
Contributor

@Lee-W Lee-W commented Jan 24, 2025

Summary

  • feat
    • add is_execute_method_inherits_from_airflow_operator for checking the removed context key in the execute method
  • refactor: rename
    • is_airflow_task as is_airflow_task_function_def
    • in_airflow_task as in_airflow_task_function_def
    • removed_in_3 as airflow_3_removal_expr
    • removed_in_3_function_def as airflow_3_removal_function_def
  • test:
    • reorganize test cases

Test Plan

a test fixture has been updated

@Lee-W
Copy link
Contributor Author

Lee-W commented Jan 24, 2025

as a heads up, I'll be less active next week as it's the lunar new year in Taiwan. Thanks for helping out!

Copy link
Contributor

github-actions bot commented Jan 24, 2025

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

@dhruvmanila
Copy link
Member

dhruvmanila commented Jan 24, 2025

refactor: rename

  • is_airflow_task as is_decorated_by_airflow_task
  • check_function_parameters as check_parameters_in_function_def
  • removed_in_3 as removed_expr_in_3
  • removed_in_3_function_def as removed_funciton_def_in_3

Can you provide the motivation for these renames? We usually prefer to have active voice instead of passive (at least regarding the first two renames) (https://github.com/astral-sh/ruff/blob/2b3550c85fcc55d93216c4d4ef64c5f0a5766053/crates/ruff_python_semantic/src/analyze/visibility.rs). I'm a bit torn on the last two renames because it indicates to me that some expression / function definition has been removed but that's not the case. I think we should rename them to airflow_3_removal (similar to numpy_2_0_deprecation) and add suffix _expr / _function_def to indicate the node that's being checked. Happy to hear your thoughts on this.

@dhruvmanila
Copy link
Member

as a heads up, I'll be less active next week as it's the lunar new year in Taiwan. Thanks for helping out!

Thanks for the heads up. Have a great week! Do you know if there's more work required to complete #14626 ?

Copy link
Member

@dhruvmanila dhruvmanila left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thank you.

I've a minor concern about the renames.

crates/ruff_linter/src/rules/airflow/rules/removal_in_3.rs Outdated Show resolved Hide resolved
@dhruvmanila dhruvmanila added rule Implementing or modifying a lint rule preview Related to preview mode features labels Jan 24, 2025
@Lee-W Lee-W force-pushed the extend-airflow-context-check branch from 9eb6715 to 25653cc Compare January 24, 2025 15:12
@Lee-W
Copy link
Contributor Author

Lee-W commented Jan 24, 2025

refactor: rename

  • is_airflow_task as is_decorated_by_airflow_task
  • check_function_parameters as check_parameters_in_function_def
  • removed_in_3 as removed_expr_in_3
  • removed_in_3_function_def as removed_funciton_def_in_3

Can you provide the motivation for these renames? We usually prefer to have active voice instead of passive (at least regarding the first two renames) (https://github.com/astral-sh/ruff/blob/2b3550c85fcc55d93216c4d4ef64c5f0a5766053/crates/ruff_python_semantic/src/analyze/visibility.rs). I'm a bit torn on the last two renames because it indicates to me that some expression / function definition has been removed but that's not the case. I think we should rename them to airflow_3_removal (similar to numpy_2_0_deprecation) and add suffix _expr / _function_def to indicate the node that's being checked. Happy to hear your thoughts on this.

The motivation was to make it more accurate. I tried to change it a bit. Please take a look. Thanks!

@Lee-W
Copy link
Contributor Author

Lee-W commented Jan 24, 2025

as a heads up, I'll be less active next week as it's the lunar new year in Taiwan. Thanks for helping out!

Thanks for the heads up. Have a great week! Do you know if there's more work required to complete #14626 ?

There're 3 cases I can think of as of now.

  1. context.get("conf") or context["get"] in BaseOperator.execute
  2. In PythonOperator(..., python_callable=func), the func needs to be checked
  3. templated variable (e.g., params="{{ conf }}"

The first should be easy, the second might need some exploration, and the third one is a bit tricky, and I'm not even sure whether we should check it.
Whether a variable can be templated is somewhat flexible, but on the other hand, it's also uncommon to have a string happen to be in that format

@Lee-W Lee-W requested a review from dhruvmanila January 24, 2025 16:14
@dylwil3 dylwil3 changed the title [airflow] Extend airflow context parameter check in BaseOperator.execute (AIR302) [airflow] Extend airflow context parameter check for BaseOperator.execute (AIR302) Jan 25, 2025
@dhruvmanila dhruvmanila merged commit c161e4f into astral-sh:main Jan 27, 2025
21 checks passed
dhruvmanila added a commit that referenced this pull request Jan 27, 2025
dcreager added a commit that referenced this pull request Jan 29, 2025
* main:
  [red-knot] Extend instance-attribute tests (#15808)
  Fix formatter warning message for `flake8-quotes` option (#15788)
  [`flake8-bugbear`] Exempt `NewType` calls where the original type is immutable (`B008`) (#15765)
  Add missing config docstrings (#15803)
  [`refurb`] Do not emit diagnostic when loop variables are used outside loop body (`FURB122`) (#15757)
  [`ruff`] Check for shadowed `map` before suggesting fix (`RUF058`) (#15790)
  [red-knot] Do not use explicit `knot_extensions.Unknown` declaration (#15787)
  Preserve quotes in generated byte strings (#15778)
  [minor] Simplify some `ExprStringLiteral` creation logic (#15775)
  Preserve quote style in generated code (#15726)
  Rename internal helper functions (#15771)
  [`airflow`] Extend airflow context parameter check for `BaseOperator.execute` (`AIR302`) (#15713)
  Implement tab autocomplete for `ruff config` (#15603)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
preview Related to preview mode features rule Implementing or modifying a lint rule
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants