Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cache_ok flag to sqlalchemy TypeDecorators. #24499

Merged
merged 1 commit into from
Jul 5, 2022

Conversation

jkinkead
Copy link
Contributor

@jkinkead jkinkead commented Jun 16, 2022

This fixes #22647, which was marked closed due to lack of activity / feedback.

SQLAlchemy added the cache_ok flag in 1.4.14. If this is unset in a TypeDecorator subclass, it emits a log warning when the class is used. The fix is to set the value to True or False to suppress the warning.

I've set the value to True for the three TypeDecorator classes in the airflow.utils.sqlalchemy module, as these classes have no state and are safe to cache. Setting it to False would also suppress the warning, and leave behavior unchanged.

@boring-cyborg
Copy link

boring-cyborg bot commented Jun 16, 2022

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@jkinkead
Copy link
Contributor Author

Fixes new issue #24504 .

Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. But @ephraimbuddy @dstandish or @ashb co-review is needed

@github-actions
Copy link

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Jun 19, 2022
Copy link
Member

@uranusjr uranusjr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think at least ExtendedJSON should use cache_ok = False since the value is not cachable? (Not sure about relativedelta.)

@jkinkead
Copy link
Contributor Author

The cache_ok flag is confusing: It doesn't describe whether you can cache prepared values (the return values of process_bind_param), it describes whether you can use the instance of the type itself in caching the SQL statement the type is referenced in. Since these types are stateless and have static configuration, they're fine to use in the statement cache (they all prepare bind params identically).

See this section of the SQLAlchemy docs on how cache keys are generated.

Using a False here would exclude any statements with these columns from the SQLAlchemy statement cache, so it would have some performance implications.

@potiuk
Copy link
Member

potiuk commented Jul 4, 2022

Makes sense . I rebased to see if it will be fine. also it might be necesary after #24819 is merged as we will be upgrading to sqlalchemy > 1.4.39 that will start emit this warning.

@potiuk
Copy link
Member

potiuk commented Jul 4, 2022

Just in case - we should rebase this one after #24819 gets merged to test its behaviour anyway.

@potiuk potiuk merged commit d969473 into apache:main Jul 5, 2022
@boring-cyborg
Copy link

boring-cyborg bot commented Jul 5, 2022

Awesome work, congrats on your first merged pull request!

potiuk pushed a commit that referenced this pull request Jul 5, 2022
@potiuk potiuk added this to the Airflow 2.3.3 milestone Jul 5, 2022
@ephraimbuddy ephraimbuddy added the type:bug-fix Changelog: Bug Fixes label Jul 5, 2022
ephraimbuddy pushed a commit that referenced this pull request Jul 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
full tests needed We need to run full set of tests for this PR to merge type:bug-fix Changelog: Bug Fixes
Projects
None yet
5 participants