Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load Project and Site-Packages from S3 #510

Closed
mcrowson opened this issue Dec 2, 2016 · 4 comments
Closed

Load Project and Site-Packages from S3 #510

mcrowson opened this issue Dec 2, 2016 · 4 comments

Comments

@mcrowson
Copy link
Collaborator

mcrowson commented Dec 2, 2016

Currently, the entire application and site packages are uploaded to Lambda. This issue proposes uploading the project contents and deps to S3 instead, and having the zappa handler load the project files.

Proposed Changes:

  • Zappa CLI: Zip up minimal handler application with Zappa and dependencies for lambda
  • Zappa CLI: Send project zip to S3
  • Handler: On handler initialization, load the project zip in S3 into /tmp/{project_name}
  • Handler: Add the /tmp/{project_name} to sys path.

The first time the application runs it will likely be slower pulling from S3, but if the application uses the keep_warm appropriately, then the subsequent loads shouldn't have the same slowdown due to lambda keeping /tmp around between function calls.

The benefit of this change would be allowing much larger applications to run on Lambda.

Thoughts? Am I missing something?
Credits to @mathom for suggesting P3 via S3 this way.

@mathom
Copy link
Collaborator

mathom commented Dec 2, 2016

You probably want to check the file hash somehow (s3 etag) in case you push an update to s3 and need to pull down over an already unzipped project in a warm handler. Not sure if that's really an issue depending on how long your stuff lives but it might be a while with the keep warm setting.

It should be pretty obvious when the application is actually having to download and unzip in the timings, so it will be easy to keep an eye on.

@mcrowson
Copy link
Collaborator Author

mcrowson commented Dec 5, 2016

Thanks @machbio for wanting to help (as per Slack convo). Here is my fork for getting going. https://github.com/mcrowson/Zappa/tree/project-to-s3

There are a few things I'm working through now.

  • We will need to keep the project in S3. The current zappa paradigm cleans up after itself in S3, so we might need a new settings for how many of these to retain and where.
  • As this approach is only needed for projects over 50 MB, we might want this feature to only take place if the size of the packaged zip is > 50 MB. This way smaller projects can avoid the additional startup time on cold-starts.

@mathom
Copy link
Collaborator

mathom commented Dec 6, 2016

A cool side effect of keeping your project in a bucket is you could move all the lambda management stuff over to cloud formation, too. Not sure if that's worth pursuing at the moment but it's def an option to simplify some of the update stuff.

@machbio
Copy link
Contributor

machbio commented Dec 6, 2016

Sorry about doubting '/tmp' space guarantee - its pretty clear that '/tmp' space is 512MB as given in the Limit - http://docs.aws.amazon.com/lambda/latest/dg/limits.html#limits-list

Ephemeral disk capacity ("/tmp" space) 512 MB

and also on the FAQ page for Lambda - https://aws.amazon.com/lambda/faqs/

Q: What if I need scratch space on disk for my AWS Lambda function?
Each Lambda function receives 500MB of non-persistent disk space in its own /tmp directory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants