Open
Description
It appears that the serverless-python-requirements plugin is exhibiting some odd behavior. It doesn't appear to be slimming the contents of the package that's deployed to Lambda, and includes the entire contents of the venv directory in the package, irrespective of whether or not athe zip option was selected.
Here's what our serverless.yml looks like:
plugins:
- serverless-python-requirements
custom:
pythonRequirements:
useDownloadCache: false
useStaticCache: false
invalidateCaches: true
zip: true
slim: true
slimPatternsAppendDefaults: false # we need to overwrite default dist-info
slimPatterns:
# "normal" exclusions:
- '**/tests/*'
# default dist-info with !(lib_name*) - matches only if name is not present:
- "**/!(lib_name*)*.dist-info/*"
# the rest of defaults:
- '**/*.py[c|o]'
- '**/__pycache__*'
stage: 'dev'
package:
individually: true
provider:
name: aws
runtime: python3.8
timeout: 20
region: us-west-2
functions:
train-model:
handler: src/python/aws/lambda/train_model.handler
The reason for the slimPatterns is the following comment:
I have found a workaround, sadly it's a bit inconvenient to use At least it works
slim: true
slimPatternsAppendDefaults: false # we need to overwrite default dist-info
slimPatterns:
# "normal" exclusions:
- '**/tests/*'
# default dist-info with !(lib_name*) - matches only if name is not present:
- "**/!(lib_name*)*.dist-info/*"
# the rest of defaults:
- '**/*.py[c|o]'
- '**/__pycache__*'
Okay fine, but even that doesn't seem to work. Now with the zip: true above, I do get the .requirements.zip. Woo hoo! It's ~59 megs. BUT, here's my Serverless package:
$ ls -alh train-model.zip
-rw-r--r-- 1 ryancammer ryancammer 281M Dec 8 20:27 train-model.zip
And why is it so huge?
$ unzip train-model.zip
inflating: venv/lib64/python3.8/site-packages/websocket/tests/__pycache__/test_cookiejar.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/websocket/tests/__pycache__/test_websocket.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/websocket/tests/data/header01.txt
inflating: venv/lib64/python3.8/site-packages/websocket/tests/data/header02.txt
inflating: venv/lib64/python3.8/site-packages/websocket/tests/test_cookiejar.py
inflating: venv/lib64/python3.8/site-packages/websocket/tests/test_websocket.py
inflating: venv/lib64/python3.8/site-packages/websocket_client-0.56.0.dist-info/INSTALLER
inflating: venv/lib64/python3.8/site-packages/websocket_client-0.56.0.dist-info/LICENSE
inflating: venv/lib64/python3.8/site-packages/websocket_client-0.56.0.dist-info/METADATA
inflating: venv/lib64/python3.8/site-packages/websocket_client-0.56.0.dist-info/RECORD
inflating: venv/lib64/python3.8/site-packages/websocket_client-0.56.0.dist-info/WHEEL
inflating: venv/lib64/python3.8/site-packages/websocket_client-0.56.0.dist-info/top_level.txt
inflating: venv/lib64/python3.8/site-packages/yaml/__init__.py
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/__init__.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/composer.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/constructor.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/cyaml.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/dumper.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/emitter.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/error.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/events.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/loader.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/nodes.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/parser.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/reader.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/representer.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/resolver.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/scanner.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/serializer.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/__pycache__/tokens.cpython-38.pyc
inflating: venv/lib64/python3.8/site-packages/yaml/composer.py
inflating: venv/lib64/python3.8/site-packages/yaml/constructor.py
inflating: venv/lib64/python3.8/site-packages/yaml/cyaml.py
inflating: venv/lib64/python3.8/site-packages/yaml/dumper.py
inflating: venv/lib64/python3.8/site-packages/yaml/emitter.py
inflating: venv/lib64/python3.8/site-packages/yaml/error.py
inflating: venv/lib64/python3.8/site-packages/yaml/events.py
inflating: venv/lib64/python3.8/site-packages/yaml/loader.py
inflating: venv/lib64/python3.8/site-packages/yaml/nodes.py
inflating: venv/lib64/python3.8/site-packages/yaml/parser.py
inflating: venv/lib64/python3.8/site-packages/yaml/reader.py
inflating: venv/lib64/python3.8/site-packages/yaml/representer.py
inflating: venv/lib64/python3.8/site-packages/yaml/resolver.py
inflating: venv/lib64/python3.8/site-packages/yaml/scanner.py
inflating: venv/lib64/python3.8/site-packages/yaml/serializer.py
inflating: venv/lib64/python3.8/site-packages/yaml/tokens.py
So basically, the plugin is including the venv directory anyway, .pyc and all. Using dockerizePip: true, this is the result on OSX:
✗ ls -h
-rw-r--r-- 1 cory staff 64M Dec 8 20:46 .requirements.zip
-rw-r--r-- 1 cory staff 76M Dec 8 20:46 train-model.zip
Again, that train-model.zip file was chock full of .pyc files. I've tried with and without the slimPatterns, with and without zip: true, but no matter what, the end result is a massive lambda zip that's full of .pyc files.