acryl-datahub-airflow-plugin airflow 2.7 compatibility

[grumbler]

Describe the bug

acryl-datahub-airflow-plugin 0.11.0.1 seems to have compatibility issues with airflow 2.7.1

plugin can't be installed with constraints file, when installed separately pip check complains

pydantic-core 2.6.3 has requirement typing-extensions!=4.7.0,>=4.6.0, but you have typing-extensions 4.5.0.

To Reproduce
Steps to reproduce the behavior:

Tested with python 3.10.3

pip install 'apache-airflow==2.7.1' --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.1/constraints-3.10.txt"
pip install 'apache-airflow==2.7.1' 'acryl-datahub-airflow-plugin==0.11.0.1'
pip check

Expected behavior

I have not yet tested if this conflict causes any issues. Though, for previous airflow versions we have always installed dependencies with the constraints file and it worked fine.

Desktop (please complete the following information):

• OS: MacOs
• Browser -
• Version -

Additional context

Two issues that could be related:
#8829
#8859

[hsheth2]

The typing extensions issue is going to be fixed as part of #8853.

For the pydantic issue, you're right that #8829 will be required to be compatible with Airflow's precise constraints. However, based on apache/airflow#32366, it seems like Airflow is compatible with pydantic 1 and 2, so we should work fine.

[Perados]

Hi, I don't think the plugin is compatible with Airflow 2.7.1 yet because of pydantic 2:

→ pip install acryl-datahub-airflow-plugin==0.11.0.3 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.7.1/constraints-3.11.txt
...
INFO: pip is looking at multiple versions of acryl-datahub[datahub-rest] to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install acryl-datahub-airflow-plugin==0.11.0.3 and acryl-datahub[datahub-rest]==0.11.0.3 because these package versions have conflicting dependencies.

The conflict is caused by:
    acryl-datahub-airflow-plugin 0.11.0.3 depends on pydantic>=1.5.1
    acryl-datahub[datahub-rest] 0.11.0.3 depends on pydantic!=1.10.3, <2 and >=1.5.1
    The user requested (constraint) pydantic==2.3.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Any thoughts on this?

[hsheth2]

@Perados so for airflow 2.7, we're compatible with the dependencies declared by apache-airflow, but not the constraints file. However, because Airflow is actually compatible with both pydantic 1.x and 2.x, you won't have any issues running the two together (and we test this in CI). You're right that installing with the airflow constraints file will not work though.

[steveipkis]

Would you welcome contributions to make the acryl-datahub-airflow-plugin conform to the airflow constraints file?

It would make lives easier if users didn't have to go through dependency hell when installing new plugin providers for airflow. Could even open a PR to pin acryl-datahub-airflow-plugin per airflow release

[hsheth2]

@steveipkis yes we definitely would accept PRs. For what it's worth, we are now compatible with the airflow 2.7 / python 3.11 constraints file (because of #9434).

In the immediate term, the most helpful thing would actually be PRs around testing with Airflow 2.8