Prevent plugin packaging when deploying non-python functions #179

icj217 · 2018-04-20T20:01:14Z

It would be nice if the plugin could either ignore or not be invoked when deploying a non-python function. For those of us with mixed-runtime projects, this will speed up testing and deployment.

Right now, when deploying a node function, all the python packages we have defined for our python packages get included in the node function's ZIP.

I've had to define our provider-level runtime to python3.6 and then at the function-level I repeat the runtime definitions (python3.6, node6.10, etc). Here's an example:

service: my-mixed-runtime-service

plugins:
  - serverless-python-requirements
custom:
  pythonRequirements:
    dockerizePip: non-linux
    fileName: requirements.txt

provider:
  name: aws
  runtime: python3.6
  stage: test
  region: us-east-1

functions:
  node-function:
    handler: functions/my_node_function.handler
    runtime: nodejs6.10
    timeout: 10
  python-function:
    handler: functions/my_python_function.handler
    runtime: python3.6
    timeout: 10
    package: 
      excludes: ['node_modules/']

Then, when running a sls deploy --function node-function, I get the following output. 42MB is accurate for our python packages but we would expect our node_modules to be around 9-10MB.

Serverless: Installing required Python packages with python3.6...
Serverless: Docker Image: lambci/lambda:build-python3.6
Serverless: Linking required Python packages...
Serverless: Packaging function: node-function...
Serverless: Excluding development dependencies...
Serverless: Unlinking required Python packages...
Serverless: Uploading function: node-function (42.49 MB)...
Serverless: Successfully deployed function: node-function
Serverless: Successfully updated function: node-function

And here's our requirements.txt and package.json for good measure:

Requirements.txt

appdirs==1.4.3
cached-property==1.4.0
certifi==2018.1.18
chardet==3.0.4
defusedxml==0.5.0
idna==2.6
isodate==0.6.0
lxml==4.2.1
pytz==2018.3
requests==2.18.4
requests-toolbelt==0.8.0
six==1.11.0
urllib3==1.22
zeep==2.5.0

Package.json

{
  "name": "my-mixed-runtime-service",
  "version": "1.0.0",
  "description": "Blah",
  "main": "README.md",
  "dependencies": {
    "@slack/client": "^4.1.0",
    "async": "^2.6.0",
    "csv-parse": "^2.2.0",
    "detect-csv": "^1.1.0",
    "moment": "^2.22.0",
    "pg": "^7.4.1",
    "request": "^2.85.0",
    "request-promise": "^4.2.2",
    "serverless-pseudo-parameters": "^1.4.2",
    "serverless-python-requirements": "^3.3.1",
    "serverless-step-functions": "^1.4.0"
  },
  "devDependencies": {},
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "blah"
  },
  "author": "",
  "license": "ISC",
  "homepage": "blah"
}

The text was updated successfully, but these errors were encountered:

bsamuel-ui · 2018-04-20T20:21:56Z

I believe you want to set individually: true per this guidance.

icj217 · 2018-04-23T13:39:39Z

@bsamuel-ui Ok so I've modified my serverless.yml like so and it fixed the issue with my node.js files being packaged by the plugin. But now my python function is complaining about No module named 'zeep' when invoking in the Lambda console.

Custom:

Custom:
  pythonRequirements:
    dockerizePip: non-linux
    individually: true

Python Function:

  python-function:
    handler: functions/my_python_function/my_python_function.handler
    module: functions/my_python_function
    runtime: python3.6
    timeout: 10
    package: 
      excludes: ['node_modules/']

(when I set handler to my_python_function.handler it would complain about not being able to find the handler)

And my files look like:

serverless.yml
functions
   my-python-function
      py-python-function.py
      requirements.txt

dschep · 2018-04-23T14:01:18Z

@icj217 where is your requirements.txt located? It should be stored at functions/my_python_function/requirements.txt when using package: individually IIRC

icj217 · 2018-04-23T14:03:21Z

@dschep Correct that's where it's located.

icj217 · 2018-04-23T14:14:44Z

I've identified part of the issue. I was including the following in the custom.pythonRequirements section rather than under package.

package:
  individually: true

Now I am getting the following error:

Serverless: Installing requirements of requirements.txt in .serverless...
Serverless: Docker Image: lambci/lambda:build-python3.6
Serverless: Installing requirements of functions/my-python-function/requirements.txt in .serverless/functions/my-python-function...
Serverless: Docker Image: lambci/lambda:build-python3.6
Serverless: Invoke package:function
Serverless: Packaging function: my-python-function...
Serverless: Excluding development dependencies...
Serverless: Injecting required Python packages to package...

  Type Error ---------------------------------------------

  Cannot read property 'artifact' of undefined

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

TypeError: Cannot read property 'artifact' of undefined
    at BbPromise.resolve.filter.map.map.func (/Users/craigburdulis/Documents/Projects/my-mixed-runtime-service/node_modules/serverless-python-requirements/lib/inject.js:97:40)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:372:22)
    at PluginManager.spawn (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:390:17)
    at Deploy.BbPromise.bind.then.then (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:118:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:113:10)
    at BbPromise.reduce (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:372:55)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:372:22)
    at PluginManager.run (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:403:17)
    at variables.populateService.then (/usr/local/lib/node_modules/serverless/lib/Serverless.js:102:33)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)
From previous event:
    at Serverless.run (/usr/local/lib/node_modules/serverless/lib/Serverless.js:89:74)
    at serverless.init.then (/usr/local/lib/node_modules/serverless/bin/serverless:42:50)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           6.11.0
     Serverless Version:     1.26.1

icj217 · 2018-04-23T14:18:25Z

I'm not sure what const artifact = func.package.artifact; is supposed to be pulling (unless that's an internal SLS attribute). There's no package attribute in my functions.

dschep · 2018-04-23T14:28:11Z

Dohh.. you've run into #161. Does it work when you deploy the whole stack? If I get a chance to do so, I'll take a stab at serverless/serverless#4898 which is the root cause of the bug.

icj217 · 2018-04-23T14:32:00Z

It does work when I deploy the whole stack.

icj217 · 2018-04-23T14:35:48Z

If I try a deploy of a node.js function it still fails because it looks like the python functions are still getting packaged?

(venv) craigburdulis@Craig-Burdulis-MacBook-Air:~/Documents/Projects/my-mixed-runtime-service$ sls deploy --function my-node-function
Serverless: Installing requirements of requirements.txt in .serverless...
Serverless: Docker Image: lambci/lambda:build-python3.6
Serverless: Installing requirements of functions/historical-validation/requirements.txt in .serverless/functions/my-python-function...
Serverless: Docker Image: lambci/lambda:build-python3.6
Serverless: Packaging function: my-node-function...
Serverless: Injecting required Python packages to package...

  Type Error ---------------------------------------------

  Cannot read property 'artifact' of undefined

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           6.11.0
     Serverless Version:     1.26.1

fixes #161 and fixes #179

@AndrewFarley

* Better mixed runtime & function deploy handling fixes #161 and fixes #179 * another tweak * fix again? * Fix corrupted zip archive in case of same module * Do not try to install requirements for non-python runtime * Fix lint * format * update test for merging #181 * @AndrewFarley's fix * huh. depcheck sucks. * fixix syntax error

dschep added a commit that referenced this issue Apr 23, 2018

Better mixed runtime & function deploy handling

832b7d6

fixes #161 and fixes #179

dschep mentioned this issue Apr 23, 2018

Better mixed runtime & function deploy handling #180

Merged

dschep added a commit that referenced this issue Apr 23, 2018

Better mixed runtime & function deploy handling

f4390b2

fixes #161 and fixes #179

dschep closed this as completed in #180 Apr 26, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prevent plugin packaging when deploying non-python functions #179

Prevent plugin packaging when deploying non-python functions #179

icj217 commented Apr 20, 2018 •

edited

Loading

bsamuel-ui commented Apr 20, 2018

icj217 commented Apr 23, 2018 •

edited

Loading

dschep commented Apr 23, 2018

icj217 commented Apr 23, 2018

icj217 commented Apr 23, 2018

icj217 commented Apr 23, 2018

dschep commented Apr 23, 2018

icj217 commented Apr 23, 2018

icj217 commented Apr 23, 2018

Prevent plugin packaging when deploying non-python functions #179

Prevent plugin packaging when deploying non-python functions #179

Comments

icj217 commented Apr 20, 2018 • edited Loading

bsamuel-ui commented Apr 20, 2018

icj217 commented Apr 23, 2018 • edited Loading

dschep commented Apr 23, 2018

icj217 commented Apr 23, 2018

icj217 commented Apr 23, 2018

icj217 commented Apr 23, 2018

dschep commented Apr 23, 2018

icj217 commented Apr 23, 2018

icj217 commented Apr 23, 2018

icj217 commented Apr 20, 2018 •

edited

Loading

icj217 commented Apr 23, 2018 •

edited

Loading