"chalice package" fails in download_all_dependencies()

[iscooter]

Found during CI build and fixed it by reverting setuptools: python3 -m pip install setuptools==59.8.0. I'm assuming there's a dependency with distutils/setuptools that was introduced in 60.0.0 that's breaking the downloader.

Chalice version used:

bash-4.2# chalice --version
+ chalice --version
chalice 1.26.1, python 3.7.10, linux 5.10.47-linuxkit

Error with setuptools 60.0.0 or greater:

bash-4.2# chalice --debug package --stage temp --pkg-format terraform .
+ chalice --debug package --stage temp --pkg-format terraform .
2021-12-20 20:20:15,081 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2021-12-20 20:20:15,083 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2021-12-20 20:20:15,084 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2021-12-20 20:20:15,086 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2021-12-20 20:20:15,086 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2021-12-20 20:20:15,087 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2021-12-20 20:20:15,088 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2021-12-20 20:20:15,091 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2021-12-20 20:20:15,091 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2021-12-20 20:20:15,092 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2021-12-20 20:20:15,092 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2021-12-20 20:20:15,093 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/endpoints.json
/usr/local/lib/python3.7/site-packages/_distutils_hack/__init__.py:35: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Unsupported service for auto policy generation: apigateway
2021-12-20 20:20:15,733 botocore.credentials [INFO] Found credentials in environment variables.
Creating deployment package.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/chalice/cli/__init__.py", line 636, in main
    return cli(obj={})
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/chalice/cli/__init__.py", line 555, in package
    packager.package_app(config, out, stage)
  File "/usr/local/lib/python3.7/site-packages/chalice/package.py", line 1255, in package_app
    config, chalice_stage_name)
  File "/usr/local/lib/python3.7/site-packages/chalice/package.py", line 117, in construct_resources
    self._build_stage.execute(config, resources)
  File "/usr/local/lib/python3.7/site-packages/chalice/deploy/deployer.py", line 652, in execute
    step.handle(config, resource)
  File "/usr/local/lib/python3.7/site-packages/chalice/deploy/deployer.py", line 419, in handle
    handler(config, resource)
  File "/usr/local/lib/python3.7/site-packages/chalice/deploy/deployer.py", line 454, in handle_deploymentpackage
    config.project_dir, config.lambda_python_version)
  File "/usr/local/lib/python3.7/site-packages/chalice/deploy/packager.py", line 301, in create_deployment_package
    site_packages_dir=tmpdir)
  File "/usr/local/lib/python3.7/site-packages/chalice/deploy/packager.py", line 278, in _build_python_dependencies
    abi, requirements_filepath, site_packages_dir)
  File "/usr/local/lib/python3.7/site-packages/chalice/deploy/packager.py", line 784, in build_site_packages
    abi, tempdir, requirements_filepath)
  File "/usr/local/lib/python3.7/site-packages/chalice/deploy/packager.py", line 637, in _download_dependencies
    requirements_filename, directory)
  File "/usr/local/lib/python3.7/site-packages/chalice/deploy/packager.py", line 566, in _download_all_dependencies
    self._pip.download_all_dependencies(requirements_filename, directory)
  File "/usr/local/lib/python3.7/site-packages/chalice/deploy/packager.py", line 1050, in download_all_dependencies
    raise PackageDownloadError(error)
chalice.deploy.packager.PackageDownloadError: Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 963, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 906, in _find_spec
  File "/usr/local/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 83, in find_spec
    return method()
  File "/usr/local/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 104, in spec_for_pip
    if self.pip_imported_during_build():
  File "/usr/local/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 117, in pip_imported_during_build
    for frame, line in traceback.walk_stack(None)
  File "/usr/local/lib/python3.7/site-packages/_distutils_hack/__init__.py", line 117, in <genexpr>
    for frame, line in traceback.walk_stack(None)
KeyError: '__file__'

Fixed with setuptools 59.8.0 or lower:

bash-4.2# chalice --debug package --stage temp --pkg-format terraform .
+ chalice --debug package --stage temp --pkg-format terraform .
2021-12-20 20:22:50,000 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2021-12-20 20:22:50,003 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2021-12-20 20:22:50,004 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2021-12-20 20:22:50,005 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2021-12-20 20:22:50,005 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2021-12-20 20:22:50,006 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2021-12-20 20:22:50,007 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2021-12-20 20:22:50,009 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2021-12-20 20:22:50,010 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2021-12-20 20:22:50,010 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2021-12-20 20:22:50,010 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2021-12-20 20:22:50,011 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/endpoints.json
Unsupported service for auto policy generation: apigateway
2021-12-20 20:22:50,748 botocore.credentials [INFO] Found credentials in environment variables.
Creating deployment package.
2021-12-20 20:22:57,085 botocore.credentials [INFO] Found credentials in environment variables.

[redduck3]

I meet the same problem which led to the failure of application deployment. Now I cannot update the chalice-based application.

[aalvrz]

Also experiencing this issue. Can confirm that pinning to setuptools==59.8.0 is a workaround.

[jamesls]

Seems like this is due to pypa/setuptools#2941, which is fixed in 60.0.3. Confirmed that the latest setuptools no longer has this issue so people should no longer be running into this issue (if we wanted to be safe we could explicitly exclude 60.0.0-60.0.2 in our install_requires).

[iscooter]

verified 60.0.3 does work now 👍