Transitive dependencies are too specific in install_requires

[abfreiheit]

Python SDK client library is not compatible with the most popular DS installations in GCP.
I was using !pip install feast together with standard Jupyter notebook installations for TensorFlow 1.15.0 and 2.2.0.

In both cases the installations end up with the following errors:

Collecting feast
  Downloading feast-0.6.2-py3-none-any.whl (116 kB)
     |████████████████████████████████| 116 kB 7.1 MB/s eta 0:00:01
Collecting google-cloud-bigquery==1.18.*
  Downloading google_cloud_bigquery-1.18.1-py2.py3-none-any.whl (146 kB)
     |████████████████████████████████| 146 kB 21.2 MB/s eta 0:00:01
Requirement already satisfied: googleapis-common-protos==1.* in /opt/conda/lib/python3.7/site-packages (from feast) (1.51.0)
Requirement already satisfied: Click==7.* in /opt/conda/lib/python3.7/site-packages (from feast) (7.1.2)
Collecting google
  Downloading google-3.0.0-py2.py3-none-any.whl (45 kB)
     |████████████████████████████████| 45 kB 4.3 MB/s  eta 0:00:01
Collecting google-api-core==1.20.*
  Downloading google_api_core-1.20.1-py2.py3-none-any.whl (90 kB)
     |████████████████████████████████| 90 kB 9.2 MB/s  eta 0:00:01
Collecting kafka-python==1.*
  Downloading kafka_python-1.4.7-py2.py3-none-any.whl (266 kB)
     |████████████████████████████████| 266 kB 16.6 MB/s eta 0:00:01
Requirement already satisfied: tabulate==0.8.* in /opt/conda/lib/python3.7/site-packages (from feast) (0.8.7)
Requirement already satisfied: google-auth<2.0dev,>=1.14.0 in /opt/conda/lib/python3.7/site-packages (from feast) (1.19.2)
Collecting google-cloud-storage==1.20.*
  Downloading google_cloud_storage-1.20.0-py2.py3-none-any.whl (70 kB)
     |████████████████████████████████| 70 kB 10.7 MB/s eta 0:00:01
Collecting google-cloud-bigquery-storage==0.7.*
  Downloading google_cloud_bigquery_storage-0.7.0-py2.py3-none-any.whl (55 kB)
     |████████████████████████████████| 55 kB 5.3 MB/s  eta 0:00:01
Requirement already satisfied: pandas~=1.0.0 in /opt/conda/lib/python3.7/site-packages (from feast) (1.0.5)
Collecting google-cloud-core==1.0.*
  Downloading google_cloud_core-1.0.3-py2.py3-none-any.whl (26 kB)
Requirement already satisfied: protobuf>=3.10 in /opt/conda/lib/python3.7/site-packages (from feast) (3.12.3)
Requirement already satisfied: grpcio==1.* in /opt/conda/lib/python3.7/site-packages (from feast) (1.30.0)
Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from feast) (1.19.1)
Requirement already satisfied: toml==0.10.* in /opt/conda/lib/python3.7/site-packages (from feast) (0.10.1)
Collecting confluent-kafka
  Downloading confluent_kafka-1.5.0-cp37-cp37m-manylinux1_x86_64.whl (8.1 MB)
     |████████████████████████████████| 8.1 MB 15.8 MB/s eta 0:00:01
Collecting PyYAML==5.1.*
  Downloading PyYAML-5.1.2.tar.gz (265 kB)
     |████████████████████████████████| 265 kB 74.1 MB/s eta 0:00:01
Requirement already satisfied: tqdm==4.* in /opt/conda/lib/python3.7/site-packages (from feast) (4.48.0)
Collecting fastavro<0.23,>=0.22.11
  Downloading fastavro-0.22.13-cp37-cp37m-manylinux2010_x86_64.whl (1.3 MB)
     |████████████████████████████████| 1.3 MB 43.8 MB/s eta 0:00:01
Collecting pandavro==1.5.*
  Downloading pandavro-1.5.2.tar.gz (3.8 kB)
Collecting pyarrow<0.16.0,>=0.15.1
  Downloading pyarrow-0.15.1-cp37-cp37m-manylinux2010_x86_64.whl (59.2 MB)
     |████████████████████████████████| 59.2 MB 2.4 MB/s eta 0:00:011:00:08.2 MB 2.4 MB/s eta 0:00:06
Collecting google-resumable-media<0.5.0dev,>=0.3.1
  Downloading google_resumable_media-0.4.1-py2.py3-none-any.whl (38 kB)
Requirement already satisfied: beautifulsoup4 in /opt/conda/lib/python3.7/site-packages (from google->feast) (4.9.1)
Requirement already satisfied: setuptools>=34.0.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core==1.20.*->feast) (49.2.0.post20200712)
Requirement already satisfied: six>=1.10.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core==1.20.*->feast) (1.15.0)
Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core==1.20.*->feast) (2.24.0)
Requirement already satisfied: pytz in /opt/conda/lib/python3.7/site-packages (from google-api-core==1.20.*->feast) (2020.1)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.14.0->feast) (4.1.1)
Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3" in /opt/conda/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.14.0->feast) (4.6)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.14.0->feast) (0.2.7)
Requirement already satisfied: python-dateutil>=2.6.1 in /opt/conda/lib/python3.7/site-packages (from pandas~=1.0.0->feast) (2.8.1)
Requirement already satisfied: soupsieve>1.2 in /opt/conda/lib/python3.7/site-packages (from beautifulsoup4->google->feast) (2.0.1)
Requirement already satisfied: chardet<4,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core==1.20.*->feast) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core==1.20.*->feast) (1.25.10)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core==1.20.*->feast) (2020.6.20)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core==1.20.*->feast) (2.10)
Requirement already satisfied: pyasn1>=0.1.3 in /opt/conda/lib/python3.7/site-packages (from rsa<5,>=3.1.4; python_version >= "3"->google-auth<2.0dev,>=1.14.0->feast) (0.4.8)
Building wheels for collected packages: PyYAML, pandavro
  Building wheel for PyYAML (setup.py) ... done
  Created wheel for PyYAML: filename=PyYAML-5.1.2-cp37-cp37m-linux_x86_64.whl size=44104 sha256=f3ae30f6e563d48459717b27bfca01eedcda098554cceaa7816e747de448fc76
  Stored in directory: /home/jupyter/.cache/pip/wheels/23/b9/73/57aaccb6957d94ed63f474b51a9f7f992c5eff4635052c0557
  Building wheel for pandavro (setup.py) ... done
  Created wheel for pandavro: filename=pandavro-1.5.2-py3-none-any.whl size=2954 sha256=d02a6dea6e304befccbc4432336798a567f3de4ef90cf593906c37701fb9d3c5
  Stored in directory: /home/jupyter/.cache/pip/wheels/33/3f/96/9f1b46a9f7f6043ff4741b1aa1a7b249ba33be4dc1d08843e4
Successfully built PyYAML pandavro
ERROR: tfx 0.22.1 has requirement pyarrow<0.17,>=0.16, but you'll have pyarrow 0.15.1 which is incompatible.
ERROR: tfx-bsl 0.22.1 has requirement pyarrow<0.17,>=0.16.0, but you'll have pyarrow 0.15.1 which is incompatible.
ERROR: tensorflow-model-analysis 0.22.2 has requirement pyarrow<0.17,>=0.16, but you'll have pyarrow 0.15.1 which is incompatible.
ERROR: tensorflow-model-analysis 0.22.2 has requirement scipy==1.4.1; python_version >= "3", but you'll have scipy 1.5.1 which is incompatible.
ERROR: tensorflow-data-validation 0.22.2 has requirement joblib<0.15,>=0.12, but you'll have joblib 0.16.0 which is incompatible.
ERROR: tensorflow-data-validation 0.22.2 has requirement pyarrow<0.17,>=0.16, but you'll have pyarrow 0.15.1 which is incompatible.
ERROR: google-cloud-translate 2.0.1 has requirement google-cloud-core<2.0dev,>=1.1.0, but you'll have google-cloud-core 1.0.3 which is incompatible.
ERROR: google-cloud-logging 1.15.0 has requirement google-cloud-core<2.0dev,>=1.1.0, but you'll have google-cloud-core 1.0.3 which is incompatible.
ERROR: apache-beam 2.22.0 has requirement httplib2<0.18.0,>=0.8, but you'll have httplib2 0.18.1 which is incompatible.
ERROR: apache-beam 2.22.0 has requirement mock<3.0.0,>=1.0.1, but you'll have mock 4.0.2 which is incompatible.
Installing collected packages: google-api-core, google-cloud-core, google-resumable-media, google-cloud-bigquery, google, kafka-python, google-cloud-storage, google-cloud-bigquery-storage, confluent-kafka, PyYAML, fastavro, pandavro, pyarrow, feast
  Attempting uninstall: google-api-core
    Found existing installation: google-api-core 1.22.0
    Uninstalling google-api-core-1.22.0:
      Successfully uninstalled google-api-core-1.22.0
  Attempting uninstall: google-cloud-core
    Found existing installation: google-cloud-core 1.3.0
    Uninstalling google-cloud-core-1.3.0:
      Successfully uninstalled google-cloud-core-1.3.0
  Attempting uninstall: google-resumable-media
    Found existing installation: google-resumable-media 0.5.1
    Uninstalling google-resumable-media-0.5.1:
      Successfully uninstalled google-resumable-media-0.5.1
  Attempting uninstall: google-cloud-bigquery
    Found existing installation: google-cloud-bigquery 1.26.0
    Uninstalling google-cloud-bigquery-1.26.0:
      Successfully uninstalled google-cloud-bigquery-1.26.0
  Attempting uninstall: google-cloud-storage
    Found existing installation: google-cloud-storage 1.29.0
    Uninstalling google-cloud-storage-1.29.0:
      Successfully uninstalled google-cloud-storage-1.29.0
  Attempting uninstall: PyYAML
    Found existing installation: PyYAML 5.3.1
ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

The reason is simple - the range of supported libraries in: https://github.com/feast-dev/feast/blob/v0.6-branch/sdk/python/setup.py is too small. This contradicts the best practices of maintaining dependencies described in this article (Indicate transitive dependencies using install_requires):
https://medium.com/knerd/best-practices-for-python-dependency-management-cc8d1913db82

Listing only direct dependencies and permitting the largest possible range of dependencies will simplify the feast client installation process a lot.

[woop]

Thanks for raising this @abfreiheit, I think this is a known issue. It's also becoming a problem when using the statistics API where the use of Google libraries can cause issues that aren't always noticed. Let's keep this issue open until we have relaxed the dependencies.

[adchia]

I think this largely has been addressed now that we use pip-tools and have also broadened dependencies. Marking as closed