AWS Python Serverless

An easy, step-by-step guide to deploy Python code to AWS Lambda using Serverless Framework.

1. Serverless Setup

Make sure Node has been installed before installing Serverless. Otherwise, go to Node official page and follow the instructions on how to install Node.
Verify that both Node and npm have been installed on your computer.
```
node --version
```
Example Output
```
v16.15.0
```
```
npm --version
```
Example Output
```
8.5.5
```
Install Serverless globally
```
npm install -g serverless
```

Verify that Serverless has been successfully installed

serverless --version

Example Output

Framework Core: 3.0.1 (standalone)
Plugin: 6.0.0
SDK: 4.3.0

Initialize Serverless Python project
```
serverless
```
Select any Python template that you are interested in. If you are new to AWS and Serverless, "AWS - Python - Starter" may be a good starting point.
Next, name your project (ideally no whitespace between words). Then, Serverless will automatically create a directory with given name and include necessary boilerplate and template files.
Select no when you are prompted to login or deploy the project.
Change to project directory and start developing your Serverless project.
```
cd <PROJECT_DIR>
```

2. Lambda Function Setup

3. How to Add Third-Party Packages

Two of the most common ways to add third-party packages to Serverless Python project.

pip + virtualenv + serverless-python-requirements
pipenv + serverless-python-requirements (recommended)

I personally prefer the second option for the following reasons.

pipenv allows developer to split development and production dependencies (similar to package.json in Node project)
Similar to package.json, pipenv will store the project configuration in Pipfile. The file allows granular configuration and control, which is apparently not available in requirements.txt file.
pipenv locks the version of the dependencies and store the details in Pipfile.lock to guarantee consistent, manageable dependencies.
serverless-python-requirements supports pipenv and will only pick up the production dependencies upon packaging and deployment. Hence, it significantly reduces the size of Lambda functions and layers.
The dependencies are installed in $HOME/.cache/pip/wheel/ directory, outside project directory. It means your project directory size remains small and is not bloated with dependencies.

However, feel free to choose any option that works best for you.

3.1. Option 1: pip + virtualenv + serverless-python-requirements

3.1.1. Windows

Create a new virtual environment inside project folder, e.g. venv

Command
```
python -m venv <VIRTUAL_ENVIRONMENT_NAME>
```
Example

Create a new virtual environment, venv inside project folder
```
python -m venv .\venv
```
Activate the virtual environment

Command
```
.\<VIRTUAL_ENVIRONMENT_NAME>\Scripts\activate
```
Activate virtual environment named venv

Example
```
.\venv\Scripts\activate
```
Check whether pip is installed

Command
```
pip -V
```
(Optional) Upgrade pip to the latest version
```
python -m pip install --upgrade pip
```
Note

If pip is accidentally removed, or not installed, it can be re-installed using the following command
```
python -m ensurepip --upgrade
```
Check the list of installed Python packages
```
pip list
```
Note

The list should only contain minimum installation, most likely pip and setuptools. If it contains any third-party packages, it indicates that the virtual environment has not been activated.
Create requirements.txt file and list the needed third-party packages

Example
```
requests
pyyaml
numpy
```
Note

The file may also contain requirement specifiers if you want to narrow down or select specific version to be installed in the virtual environment.
Install these packages by running the following command
```
pip install -r requirements.txt
```
Note

-r flag tells pip to install recursively (in case of listing multi packages in the requirements)
Check whether these packages are successfully installed
```
pip list
```
Example output

If requests package is successfully installed, it will be listed on the output
```
Package            Version
------------------ ---------
...                 ...
requests            2.26.0
...                 ...
```
Initialize Node application
```
npm init
```
Note

package.json should exist in the project directory if the initialization is successful.
Install serverless-python-requirements plugin learn more
```
npm install --save serverless-python-requirements
```

Add these lines to our serverless.yml after installation is complete

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: true

3.2. Option 2: pipenv + serverless-python-requirements (recommended)

Install pipenv
```
pip install pipenv
```
Check for pipenv version to ensure the package is successfully installed.
```
python -m pipenv --version
```
Example Output
```
pipenv, version 2022.5.2
```
(Optional) Install pipenv inside virtual environment to enable pipenv command
```
python -m pipenv install pipenv --dev
```
Note

As this is the first third-party package installation, pipenv will automatically generate Pipfile and Pipfile.lock. The --dev option tells pipenv to install the local pipenv as development dependency. It will be excluded during deployment.
Activate virtual environment
```
python -m pipenv shell
```
Check the packages installed inside the virtual environment
```
pip list
```
Alternatively, pipenv also comes with a feature to check dependency graph.
```
pipenv graph
```
If the virtual environment is configured properly, it should only contain pip, setuptools, pipenv and their dependencies.
Install all necessary dependencies.

Production dependency
```
pipenv install <PACKAGE_NAME>
```
Development dependency
```
pipenv install <PACKAGE_NAME> --dev
```
Initialize Node application
```
npm init
```
Note

package.json should exist in the project directory if the initialization is successful.
Install serverless-python-requirements plugin learn more
```
npm install --save serverless-python-requirements
```

Add the following config to serverless.yml after installation is complete

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: true

4. (Optional) Lambda Layer Configuration

If the Lambda functions of Serverless application use the same dependencies, you can take full advantage of Lambda layer to further reduce the function size by packing it as centralized dependencies.

Before

┌─────────────────────┐         ┌─────────────────────┐
│ Function 1          │         │ Function n          │
├─────────────────────┤         ├─────────────────────┤
│ Main function  5kB  │         │ Main function 10kB  │
├─────────────────────┤   ...   ├─────────────────────┤
│ Dependencies   2MB  │         │ Dependencies   2MB  │
└─────────────────────┘         └─────────────────────┘
 Total size 2.005MB               Total size 2.010MB

After

┌─────────────────────┐         ┌─────────────────────┐
│ Function 1          │         │ Function n          │
├─────────────────────┤         ├─────────────────────┤
│ Main function  5kB  │   ...   │ Main function 10kB  │
└─────────────────────┘         └─────────────────────┘
 Total size 5kB                   Total size 10kB
         │                                 │
         └─────────────────┬───────────────┘
                           │
                ┌─────────────────────┐
                | Lambda Layer        |
                ├─────────────────────┤
                │ Dependencies   2MB  │
                └─────────────────────┘

Layer configuration in serverless.yml

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: true
    layer:
      name: ${self:provider.stage}-pythonLambdaLayer
      description: Python requirements lambda layer
      #...

# explicitly filter and select necessary files and directories
package:
  individually: true
  excludeDevDependencies: true
  patterns:
    - "!./**" # exclude all files and directories
    - "./src/**" # select only 'src' directory and its files and subdirectories

# link lambda function to dependency layer
functions:
  myhandler:
    handler: src/path/to/my.handler
    #..
    layers:
      - Ref: PythonRequirementsLambdaLayer

5. Deployment to AWS

Command

serverless deploy [--aws-profile <AWS_PROFILE_NAME>] [--stage <DEPLOYMENT_STAGE>]

Note

Both aws-profile and stage are optional, however, you are expected to have at least 1 profile (default profile) in your machine. Otherwise, Serverless will never know where to deploy nor gain access to your AWS account programmatically.

Below is how Serverless looks for the AWS profile, sorted by priority (high to low):

Command line arguments (--aws-profile <AWS_PROFILE_NAME>)
Deployment script (typically serverless.yml)
Default AWS profile
- Linux & MacOS: $HOME/.aws/{credentials, config}
- Windows: C:\Users\<USER_NAME>\.aws\{credentials, config}

6. (Optional) Final Check

To make sure the third-party package has been installed and deployed, check the .serverless directory and it should have similar content as shown below.

your-function-name
├── .serverless
│   ├── requirements
│   │    ├── ...
│   │    └── ...
│   ├── cloudformation-template-create-stack.json
│   ├── cloudformation-template-update-stack.json
│   ├── pythonRequirements.zip
│   ├── function_1.zip
│   ├── ...
│   ├── function_n.zip
│   ├── requirements.txt
│   └── serverless-state.json
├── node_modules
│   ├── ...
│   └── ...
├── venv                 # virtual environment config + local dependencies
│   ├── ...              # (not exist if configuration uses pipenv)
│   └── ...
├── ...
├── src
│   ├── function_1
│   │    ├── ...
│   │    └── handler.py
│   ├── ...
│   └── function_n
│        ├── ...
│        └── handler.py
├── handler.py
├── ...
├── package.json
├── Pipfile              # (not exist if configuration uses pip + venv)
├── Pipfile.lock         # (not exist if configuration uses pip + venv)
├── requirements.txt     # (not exist if configuration uses pipenv)
└── serverless.yml

License

dvsu/aws-python-serverless

ErrorLooks like something went wrong!

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Error
Looks like something went wrong!

Packages