SNOW-237424 Declare dependency on setuptools

[groodt]

The code makes use of modules that are not bundled with the Python standard library.

Ensure that "setuptools" is part of install_requires

[sfc-gh-mkeller]

Thanks @groodt
This must be the reason why the pyarrow version checking did not work on AWS Glue.
Do you use AWS Glue? I wonder if this additional dependency would break our compatibility with glue altogether.

[groodt]

@sfc-gh-mkeller We actually didn't see this error in Glue, we see it in CI when our tests fail.

The error we see is:

ModuleNotFoundError: No module named 'pkg_resources'

We probably see it because we run with a reasonably hermetic Python setup with Bazel.

It's probably similar with AWS Glue. It seems they want you to zip up site-packages. https://aws.amazon.com/premiumsupport/knowledge-center/glue-job-use-external-python-libraries/

I don't think the extra dependency should cause problems. Even though pip and setuptools are not part of the Python stdlib, it is still relatively difficult to create an environment to install packages without them.

gunicorn includes it for example: https://github.com/benoitc/gunicorn/blob/94ab2091173c6037b504f94e56f4e88816d540bf/setup.py#L71-L77

If there are any tools which calculate the transitive closure of dependencies by fully traversing the dependency requirements (Bazel, perhaps conda too). Then it's more correct to declare the dependency.

Demo:

/tmp
❯ python --version
Python 3.7.8

/tmp
❯ python -m venv .venv

/tmp
❯ source .venv/bin/activate

/tmp
.venv ❯ pip install gunicorn pipdeptree

/tmp
.venv ❯ pipdeptree
gunicorn==20.0.4
  - setuptools [required: >=3.0, installed: 47.1.0]
pipdeptree==1.0.0
  - pip [required: >=6.0.0, installed: 20.1.1]

/tmp
.venv ❯ pip install snowflake-connector-python

/tmp 18s
.venv ❯ pipdeptree
gunicorn==20.0.4
  - setuptools [required: >=3.0, installed: 47.1.0]
pipdeptree==1.0.0
  - pip [required: >=6.0.0, installed: 20.1.1]
snowflake-connector-python==2.3.6
  - asn1crypto [required: >0.24.0,<2.0.0, installed: 1.4.0]
  - azure-common [required: <2.0.0, installed: 1.1.26]
  - azure-storage-blob [required: >=12.0.0,<13.0.0, installed: 12.6.0]
    - azure-core [required: >=1.9.0,<2.0.0, installed: 1.9.0]
      - requests [required: >=2.18.4, installed: 2.23.0]
        - certifi [required: >=2017.4.17, installed: 2020.11.8]
        - chardet [required: >=3.0.2,<4, installed: 3.0.4]
        - idna [required: >=2.5,<3, installed: 2.10]
        - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.11]
      - six [required: >=1.6, installed: 1.15.0]
    - cryptography [required: >=2.1.4, installed: 3.2.1]
      - cffi [required: >=1.8,!=1.11.3, installed: 1.14.4]
        - pycparser [required: Any, installed: 2.20]
      - six [required: >=1.4.1, installed: 1.15.0]
    - msrest [required: >=0.6.10, installed: 0.6.19]
      - certifi [required: >=2017.4.17, installed: 2020.11.8]
      - isodate [required: >=0.6.0, installed: 0.6.0]
        - six [required: Any, installed: 1.15.0]
      - requests [required: ~=2.16, installed: 2.23.0]
        - certifi [required: >=2017.4.17, installed: 2020.11.8]
        - chardet [required: >=3.0.2,<4, installed: 3.0.4]
        - idna [required: >=2.5,<3, installed: 2.10]
        - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.11]
      - requests-oauthlib [required: >=0.5.0, installed: 1.3.0]
        - oauthlib [required: >=3.0.0, installed: 3.1.0]
        - requests [required: >=2.0.0, installed: 2.23.0]
          - certifi [required: >=2017.4.17, installed: 2020.11.8]
          - chardet [required: >=3.0.2,<4, installed: 3.0.4]
          - idna [required: >=2.5,<3, installed: 2.10]
          - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.11]
  - boto3 [required: >=1.4.4,<1.16, installed: 1.15.18]
    - botocore [required: >=1.18.18,<1.19.0, installed: 1.18.18]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
      - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.15.0]
      - urllib3 [required: >=1.20,<1.26, installed: 1.25.11]
    - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
    - s3transfer [required: >=0.3.0,<0.4.0, installed: 0.3.3]
      - botocore [required: >=1.12.36,<2.0a.0, installed: 1.18.18]
        - jmespath [required: >=0.7.1,<1.0.0, installed: 0.10.0]
        - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1]
          - six [required: >=1.5, installed: 1.15.0]
        - urllib3 [required: >=1.20,<1.26, installed: 1.25.11]
  - certifi [required: <2021.0.0, installed: 2020.11.8]
  - cffi [required: >=1.9,<1.15, installed: 1.14.4]
    - pycparser [required: Any, installed: 2.20]
  - cryptography [required: >=2.5.0,<4.0.0, installed: 3.2.1]
    - cffi [required: >=1.8,!=1.11.3, installed: 1.14.4]
      - pycparser [required: Any, installed: 2.20]
    - six [required: >=1.4.1, installed: 1.15.0]
  - oscrypto [required: <2.0.0, installed: 1.2.1]
    - asn1crypto [required: >=1.0.0, installed: 1.4.0]
  - pycryptodomex [required: >=3.2,<4.0.0,!=3.5.0, installed: 3.9.9]
  - pyjwt [required: <2.0.0, installed: 1.7.1]
  - pyOpenSSL [required: >=16.2.0,<21.0.0, installed: 19.1.0]
    - cryptography [required: >=2.8, installed: 3.2.1]
      - cffi [required: >=1.8,!=1.11.3, installed: 1.14.4]
        - pycparser [required: Any, installed: 2.20]
      - six [required: >=1.4.1, installed: 1.15.0]
    - six [required: >=1.5.2, installed: 1.15.0]
  - pytz [required: <2021.0, installed: 2020.4]
  - requests [required: <2.24.0, installed: 2.23.0]
    - certifi [required: >=2017.4.17, installed: 2020.11.8]
    - chardet [required: >=3.0.2,<4, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.10]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.11]
  - urllib3 [required: >=1.20,<1.26.0, installed: 1.25.11]

You can see above, that snowflake-connector-python does not declare a dependency on setuptools, even though it does depend on it at runtime.

For what it's worth, this error surfaced for us when snowflake-connector-python released version 2.3.4. We've currently pinned to version 2.3.3 because of this issue.

[sfc-gh-mkeller]

🚢 🚢