Move Hive macros to the provider

[potiuk]

The Hive macros are now moved to the provider.

Fixes: #19445

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

This one will remove I think last "real" dependency of Airflow code on providers.

I am not 100% sure on two things:

• will that actually work (do we have to add some code to discover such hive macros ?
• do we care?
• should we add a min-hive-provider version to "apache.hive" airflow extra ?

[eladkal]

wasn't able to link the issue but this closes #19445

[Taragolis]

ds_add, ds_format, datetime_diff_for_humans commonly use by users. Also them listed into the Templates reference: https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#macros

[Taragolis]

And unfortunetly airflow macros could only distributed by Plugins, not by Providers

[potiuk]

And unfortunetly airflow macros could only distributed by Plugins, not by Providers

Provider can also deliver plugins - it's just the matter of implementing plugin endpoint.

[potiuk]

But yeah. Obviously I moved too much out of the "macros" package :). Bringing it back

[Taragolis]

Provider can also deliver plugins - it's just the matter of implementing plugin endpoint.

If it stored into Provider it can't automatically injected into Jinja Environment.

[Taragolis]

Or you mean we could add additional entrypoint into provider package?

[potiuk]

Or you mean we could add additional entrypoint into provider package?

Yep. Already done so

[Taragolis]

Whoa!!! I never thought from this side.

That mean potentially we could have CeleryExecutor into celery package and K8S Executors inside cncf.kubernetes package?

[potiuk]

I am not sure (tired?) I thought the "airflow.macros" macros also belong to hive and I moved them together with hive ones (reverted now).

I believe with the approach I proposed now - we do not even need to add any deprecations. If we properly expose the hive macros via additional entrypoint in the hive provider, we can just move "macros.hive" entrely to the provider - because users interact with them through JINJA only an the macros will be available as hive.max_partition for example.

I think the only thing left (besides testing of course) is to make sure that we document this as potentially breaking change in the docs (users will need to upgrade hive provider) and add >= <new_provider_version> in the hive extra for airlfow (and the docs of course mentioned by @eladkal ).

[potiuk]

Whoa!!! I never thought from this side.

That mean potentially we could have CeleryExecutor into celery package and K8S Executors inside cncf.kubernetes package?

Executors are not plugins - you can run your own executors without doclaring them via plugin - you need just to configure airflow to use them via configuration. So we could move them regardless - but I am not really sure we want to move them to providers because they are so "essential" to airflow. However after AIP-51 it actually would be possible.

[Taragolis]

Oh... I just more interested in a hypothetical possibility rather than move existed one or implement new one.

Anyway this approach could add ability of add provider specific filters and macros

[potiuk]

Oh... I just more interested in a hypothetical possibility rather than move existed one or implement new one.

Anyway this approach could add ability of add provider specific filters and macros

Very much so. That's why I wanted to try it out finally - it's been on my list for quite some time

[potiuk]

Still need to sort out versioning, and test it, but it's close to be ready IMHO

[potiuk]

Ok . I tested it and it works well now. Including installing the new provider on earlier version of airflow that has hive macros built-in and the macros from the provider nicely override the hive macros that come from Airflow.

I've added >= 5.1.0 exception for Hive in airflow's [hive] extra so we can be sure that the new version of Airflow pulls the new version of hive when installed with extrra.

The only potentially dangerous scenario is when someone will upgrade just airflow to > 2.6.0 (I assume this change will be out in 2.6.0) and still has the old hive provider installed. This should geenerally not happen, because of constraints and it has an easy upgrede mechanism if it happens (I added newsfragment describing it).

[eladkal]

The only potentially dangerous scenario is when someone will upgrade just airflow to > 2.6.0 (I assume this change will be out in 2.6.0) and still has the old hive provider installed. This should geenerally not happen, because of constraints and it has an easy upgrede mechanism if it happens (I added newsfragment describing it).

constraints + newsfragment is great.

It is tricky though usualy we deprecate first but I'm not sure if there is a way to raise deprecation warning for macros?

[potiuk]

It is tricky though usualy we deprecate first but I'm not sure if there is a way to raise deprecation warning for macros?

Technically it's not "deprecation" - it should work identically to old way but only when the provider is installed. This is yet another case where things are not breaking - as long as we upgrade dependencies.

We could - potentially raise a warning here - it but not just yet. After releasing Hive Provider 5.1.0 (but only after that, because otherwise the user would not have a way to get rid of the warning) we could raise such warnings - but this will have to be in Airflow 2.5.n "airflow.macros.hive" - methods - and the way to raise the warning would be to ask the user to upgrade the Hive Provider to 5.1.0+. It will have to be done directly in v2-5-test branch though.