Make Kedro Compatible with Airflow 2.4.2

[rxm7706]

Description

Airflow is one of the primary deployment solutions for Kedro
Airflow 2.4.2 has a dependency of attrs>=22.1.0
Kedro requirements uses attrs~=21.3
https://github.com/kedro-org/kedro/blob/c40873a5d4adfbc7b5970cb8b0c1133d53463bbf/dependency/requirements.txt#L2

Context

Unable to install Kedro & Airflow using conda in the same environment

Steps to Reproduce

1. mamba create -n kedro-airflow "python=3.7.12" "airflow=2.4.2" "kedro>=0.18.3"
   Encountered problems while solving:

• package airflow-2.4.2-py310h51d5547_0 requires attrs >=22.1.0, but none of the providers can be installed

Expected Result

With just the change of attrs - from attrs~=21.3 to "attrs >=22.1.0,<22.2"

Kedro, Airflow and MLFlow are compatible

mamba create -n kedro-airflow "python=3.7.12" "anyconfig >=0.10.0,<0.11.0" "attrs >=21.3,<22.2" "cachetools >=4.1,<5.0" "click <9.0" "cookiecutter >=2.1.1,<3.0" "dynaconf >=3.1.2, <4.0" "fsspec >=2021.4,<=2022.5.0" "gitpython >=3.0,<4.0" "importlib_metadata >=3.6" "importlib_resources >=1.3" "jmespath >=0.9.5,<1.0" "jupyter_client >=5.1,<7.0" "pip-tools >=6.5,<7.0" "pluggy >=1.0,<1.1" "python >=3.7,<3.11" "python-json-logger >=2.0.0,<3.0.0" "pyyaml >=4.2,<7.0" "rich >=12.0,<13.0" "rope >=0.21.0,<0.22.0" "setuptools >=38.0" "toml >=0.10,<0.11" "toposort >=1.5,<2.0" "airflow=2.4.2" "mlflow >=1.0.0,<2.0.0"

Actual Result

Draft Pull for conda-feedstock shows build is possible - Restoring compatibility with Airflow & MLFlow
conda-forge/kedro-feedstock#25

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• Kedro version used : 0.18.3
• Python version used : Python 3.7
• Operating system and version: Linux 64
• Constrained by a Monolith Conda Environment : Requiring Kedro, Airflow, MLFlow, PySpark and Great-Expectations installed in a single conda environment

[rxm7706]

@merelcht @deepyaman I've created a PR that seems to pass all required tests for Kedro

Once Merged - It appears that

kedro-org/kedro#2030
&
#52

together will address all Airflow / Kedro compatibility issues.

[noklam]

I am curious if this happens for pip too? Does it stops you completely from installing the packages?

I expect the older version of attrs will still work but it happens that kedro pinned a older version.

[rxm7706]

@noklam It should give the same error with pip - Quick Test gives the following error

INFO: pip is looking at multiple versions of to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of apache-airflow to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install apache-airflow==2.4.2 and kedro==0.18.3 because these package versions have conflicting dependencies.

The conflict is caused by:
apache-airflow 2.4.2 depends on attrs>=22.1.0
kedro 0.18.3 depends on attrs~=21.3

To fix this you could try to:

1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts