Lambda load error: importlib_metadata.PackageNotFoundError: importlib_metadata

[lbustelo]

Unable to get Lambda loading with the configuration below:

versions

Framework Core: 1.59.3
Plugin: 3.2.5
SDK: 2.2.1
Components Core: 1.1.2
Components CLI: 1.4.0

requirements.txt

boto3==1.9.42
Flask==1.1.1
flask-cors==3.0.8
flask-marshmallow==0.10.1
flask-restplus==0.13.0
marshmallow==3.2.1
python-json-logger==0.1.11
requests==2.22.0
webargs==5.5.1

# bert and TF
tensorflow==1.13.1
bert-tensorflow==1.0.1
beautifulsoup4==4.8.1

serverless.yml snippet

plugins:
  - serverless-python-requirements
  - serverless-wsgi

custom:
  wsgi:
    app: app.app
    packRequirements: false
  pythonRequirements:
    dockerImage: lambci/lambda:build-python3.6
    # dockerizePip: true
    zip: true
    slim: false
    noDeploy:
      - boto3
      - botocore
      - docutils
      - jmespath
      - pip
      - python-dateutil
      - s3transfer
      - setuptools
      - six
      - tensorboard

ERROR on Lambda Load

    from .model import Model
  File "/tmp/sls-py-req/flask_restplus/model.py", line 15, in <module>
    from jsonschema import Draft4Validator
  File "/tmp/sls-py-req/jsonschema/__init__.py", line 33, in <module>
    import importlib_metadata as metadata
  File "/tmp/sls-py-req/importlib_metadata/__init__.py", line 554, in <module>
    __version__ = version(__name__)
  File "/tmp/sls-py-req/importlib_metadata/__init__.py", line 516, in version
    return distribution(distribution_name).version
  File "/tmp/sls-py-req/importlib_metadata/__init__.py", line 489, in distribution
    return Distribution.from_name(distribution_name)
  File "/tmp/sls-py-req/importlib_metadata/__init__.py", line 194, in from_name
    raise PackageNotFoundError(name)
importlib_metadata.PackageNotFoundError: importlib_metadata
START RequestId: 263c5ed9-a9ac-4d2a-a61d-84a52a023d75 Version: $LATEST
module initialization error: Unable to import app.app

END RequestId: 263c5ed9-a9ac-4d2a-a61d-84a52a023d75
REPORT RequestId: 263c5ed9-a9ac-4d2a-a61d-84a52a023d75	Duration: 108.10 ms	Billed Duration: 200 ms	Memory Size: 1024 MB	Max Memory Used: 483 MB	Init Duration: 4198.33 ms	
module initialization error
Unable to import app.app
Traceback (most recent call last):
  File "/tmp/sls-py-req/jsonschema/__init__.py", line 31, in <module>
    from importlib import metadata
ImportError: cannot import name 'metadata'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/task/wsgi_handler.py", line 44, in import_app
    wsgi_module = importlib.import_module(wsgi_fqn_parts[-1])
  File "/var/lang/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/task/app.py", line 6, in <module>
    from industry_classifier.app import create_app
  File "/var/task/industry_classifier/app.py", line 6, in <module>
    from industry_classifier import api
  File "/var/task/industry_classifier/api/__init__.py", line 1, in <module>
    from . import views
  File "/var/task/industry_classifier/api/views.py", line 2, in <module>
    from flask_restplus import Api
  File "/tmp/sls-py-req/flask_restplus/__init__.py", line 4, in <module>
    from . import fields, reqparse, apidoc, inputs, cors
  File "/tmp/sls-py-req/flask_restplus/reqparse.py", line 19, in <module>
    from .model import Model
  File "/tmp/sls-py-req/flask_restplus/model.py", line 15, in <module>
    from jsonschema import Draft4Validator
  File "/tmp/sls-py-req/jsonschema/__init__.py", line 33, in <module>
    import importlib_metadata as metadata
  File "/tmp/sls-py-req/importlib_metadata/__init__.py", line 554, in <module>
    __version__ = version(__name__)
  File "/tmp/sls-py-req/importlib_metadata/__init__.py", line 516, in version
    return distribution(distribution_name).version
  File "/tmp/sls-py-req/importlib_metadata/__init__.py", line 489, in distribution
    return Distribution.from_name(distribution_name)
  File "/tmp/sls-py-req/importlib_metadata/__init__.py", line 194, in from_name
    raise PackageNotFoundError(name)
importlib_metadata.PackageNotFoundError: importlib_metadata

[ConstantinoSchillebeeckxSimpleRose]

The issue here (I think) is that slim: true is removing .dist-info directories that is needed by importlib_metadata to function properly.

Here's a workaround: modify the pythonRequirements section of your serverless.yml to include both slimPatternsAppendDefaults and slimPatterns (the important part here is to exclude the removal of **/*.dist-info*):

pythonRequirements:
  slim: true
  slimPatternsAppendDefaults: false
  slimPatterns:
    - '**/*.py[c|o]'
    - '**/__pycache__*'

Also note that if you were using previously cached pip files (which is on by default), you may also want to disable cache before deploying:

pythonRequirements:
  useDownloadCache: false
  useStaticCache: false
  slim: true
  slimPatternsAppendDefaults: false
  slimPatterns:
    - '**/*.py[c|o]'
    - '**/__pycache__*'

Ultimately the .dist-info you aren't deleting should contribute only minimally to the deployed .zip size; in fact, also adding '**/docs/*' and '**/tests/*' to slimPatterns should save you even more MBs. Though be warned that removing those two directories may have other side effects (for example boto3 has a docs directory as part of their code base).

[jacksgt]

I just bumped into the same error after adding a dependency and then having to figure out that one of the transitive dependencies is relying on dist-info files.

I would also vote for removing the *.dist-info pattern from the default exclusion list, since the size of them is pretty negligible, while at the same time omitting these files can break a lot of applications.
(For reference, one of my deployment packages grew from 8.5MB to 8.7MB after including the dist-info files)

[cbcoutinho]

I just ran into this problem as well; however, in my case, I'm using the poetry support from this plugin to package the dependencies of my project.

Poetry also adds the *.dist-info directory to the project virtualenv when installing the package or when building a wheel; however, this plugin neglects the package metadata when using poetry to generate the requirements.txt file. This results in the same error mentioned by OP when dependencies such as importlib_metadata rely on the package metadata to function.

I'm currently working around with the following snippet in my package __init__.py file. I'm not a fan of it either

import logging
from importlib_metadata import version

logger = logging.getLogger(__name__)

try:
    __version__ = version(__package__)
except:
    # The serverless-python-requirements plugin installs package dependencies
    # without installing this package, which leaves out the package metadata
    # from the generated zip file
    logger.warning("Unable to get package version - setting to 0.0.0")
    __version__ = "0.0.0"

Is my problem different enough from this to warrant a new issue?

[tsalama]

The issue here (I think) is that slim: true is removing .dist-info directories that is needed by importlib_metadata to function properly.

Here's a workaround: modify the pythonRequirements section of your serverless.yml to include both slimPatternsAppendDefaults and slimPatterns (the important part here is to exclude the removal of **/*.dist-info*):

pythonRequirements:
  slim: true
  slimPatternsAppendDefaults: false
  slimPatterns:
    - '**/*.py[c|o]'
    - '**/__pycache__*'

Also note that if you were using previously cached pip files (which is on by default), you may also want to disable cache before deploying:

pythonRequirements:
  useDownloadCache: false
  useStaticCache: false
  slim: true
  slimPatternsAppendDefaults: false
  slimPatterns:
    - '**/*.py[c|o]'
    - '**/__pycache__*'

Ultimately the .dist-info you aren't deleting should contribute only minimally to the deployed .zip size; in fact, also adding '**/docs/*' and '**/tests/*' to slimPatterns should save you even more MBs. Though be warned that removing those two directories may have other side effects (for example boto3 has a docs directory as part of their code base).

Thanks a lot for this, it fixed my issue! One thing though -- if I set the caches back to true after disabling them, I hit the issue again. Were you ever able to get it to work with the caches or did you have to keep them disabled? Thanks!