Run pip check in CI pipelines

[turbaszek]

Description

We should consider adding pip check command in our CI system to verify that dependencies that we use (together with constrains) are fully compatible. This should be run for biggest possible user-installation (apache-airflow[all]).

Use case / motivation

Successful pip check is often a business requirement for releasing new version of Airflow. It assure us that all dependencies we use to build and run Airflow are compatible and we won't face any unexpected errors.

Related Issues

N/A

[turbaszek]

CC @potiuk @mhenc @anitakar

[ashb]

We'll have to fix the conflicts first ;)

snowflake-connector-python 2.2.10 has requirement azure-storage-blob<13.0.0,>=12.0.0; python_version >= "3.5.2", but you have azure-storage-blob 2.1.0.
snowflake-connector-python 2.2.10 has requirement cryptography<3.0.0,>=2.5.0, but you have cryptography 3.0.
snowflake-connector-python 2.2.10 has requirement idna<2.10, but you have idna 2.10.
snowflake-connector-python 2.2.10 has requirement requests<2.24.0, but you have requests 2.24.0.
moto 1.3.14 has requirement idna<2.9,>=2.5, but you have idna 2.10.
cfn-lint 0.35.0 has requirement importlib-resources~=1.4; python_version < "3.7" and python_version != "3.4", but you have importlib-resources 3.0.0.
botocore 1.17.44 has requirement docutils<0.16,>=0.10, but you have docutils 0.16.
astroid 2.4.2 has requirement lazy-object-proxy==1.4.*, but you have lazy-object-proxy 1.5.1.

[ashb]

(That list is a lot better than it used to be)

[turbaszek]

We'll have to fix the conflicts first ;)

Yes and that's not all 🗻

[turbaszek]

Running on latest docker image:

➜ docker run -it --entrypoint /bin/bash apache/airflow
airflow@2efb9f6adf36:/opt/airflow$ pip check
slackclient 1.3.2 has requirement websocket-client<0.55.0,>=0.35, but you have websocket-client 0.57.0.
botocore 1.17.44 has requirement docutils<0.16,>=0.10, but you have docutils 0.16.

[potiuk]

We'll have to fix the conflicts first ;)
Yep. Isssue waiting to be done #10854

[mik-laj]

Done:

airflow/scripts/docker/install_airflow.sh

Line 63 in e25eea0

pip check