diff --git a/DEVELOPMENT_GUIDE.md b/DEVELOPMENT_GUIDE.md index b399682ae2..994c1e3220 100644 --- a/DEVELOPMENT_GUIDE.md +++ b/DEVELOPMENT_GUIDE.md @@ -1,5 +1,4 @@ -DEVELOPMENT GUIDE -================= +# AWS SAM CLI Development Guide **Welcome hacker!** @@ -8,11 +7,8 @@ development environment, IDEs, tests, coding practices, or anything that will help you be more productive. If you found something is missing or inaccurate, update this guide and send a Pull Request. -**Note**: `pyenv` currently only supports macOS and Linux. If you are a -Windows users, consider using [pipenv](https://docs.pipenv.org/). +## 1-Click Ready to Hack IDE (this section might be outdated, to be verified) -1-Click Ready to Hack IDE -------------------------- For setting up a local development environment, we recommend using Gitpod - a service that allows you to spin up an in-browser Visual Studio Code-compatible editor, with everything set up and ready to go for development on this project. Just click the button below to create your private workspace: [![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/awslabs/aws-sam-cli) @@ -21,125 +17,216 @@ This will start a new Gitpod workspace, and immediately kick off a build of the Gitpod is free for 50 hours per month - make sure to stop your workspace when you're done (you can always resume it later, and it won't need to run the build again). -Environment Setup ------------------ +## Environment Setup +### 1. Prerequisites (Python Virtual Environment) -### 1. Install Python Versions +AWS SAM CLI is mainly written in Python 3 and we support Python 3.6, 3.7 and 3.8. +So having a Python environment with aforementioned versions is required. -We support 3.6 and 3.7 versions. Our CI/CD pipeline is setup to run -unit tests against both Python versions. So make sure you test it -with both versions before sending a Pull Request. -See [Unit testing with multiple Python versions](#unit-testing-with-multiple-python-versions). +Having a dedicated Python virtual environment ensures it won't "pollute" or get "polluted" +by other python packages. Here we introduce two ways of setting up a Python virtual environment: +(1) Python's built in [`venv`](https://docs.python.org/3/tutorial/venv.html) and (2) [`pyenv`](https://github.com/pyenv/pyenv). -[pyenv](https://github.com/pyenv/pyenv) is a great tool to -easily setup multiple Python versions. - -> Note: For Windows, type -> `export PATH="/c/Users//.pyenv/libexec:$PATH"` to add pyenv to -> your path. - -1. Install PyEnv - - `curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash` -2. `pyenv install 3.6.8` -3. `pyenv install 3.7.2` -4. Make Python versions available in the project: - `pyenv local 3.6.8 3.7.2` - -### 2. Install Additional Tooling -#### Black -We format our code using [Black](https://github.com/python/black) and verify the source code is black compliant -in Appveyor during PRs. Black will be installed automatically with `make init`. - -After installing, you can run our formatting through our Makefile by `make black` or integrating Black directly in your favorite IDE (instructions -can be found [here](https://black.readthedocs.io/en/stable/editor_integration.html)) - -##### (workaround) Integrating Black directly in your favorite IDE -Since black is installed in virtualenv, when you follow [this instruction](https://black.readthedocs.io/en/stable/editor_integration.html), `which black` might give you this +**Note**: `pyenv` currently only supports macOS and Linux. If you are a +Windows users, consider using [pyenv-win](https://github.com/pyenv-win/pyenv-win). -```bash -(samcli37) $ where black -/Users//.pyenv/shims/black +| | `venv` | `pyenv` | +| -- | -------- | ------------ | +| Pick if you want ... | Easy setup | You want to develop and test SAM CLI in different Python versions | + + +#### `venv` setup + +```sh +python3 -m venv .venv # one time setup: create a virtual environment to directory .venv +source .venv/bin/activate # activate the virtual environment ``` +#### `pyenv` setup -However, IDEs such PyChaim (using FileWatcher) will have a hard time invoking `/Users//.pyenv/shims/black` -and this will happen: +Install `pyenv` and [`pyenv-virtualenv` plugin](https://github.com/pyenv/pyenv-virtualenv) +On macOS with [Homebrew](https://brew.sh/) + +```sh +brew install pyenv +brew install pyenv-virtualenv ``` -pyenv: black: command not found -The `black' command exists in these Python versions: - 3.7.2/envs/samcli37 - samcli37 -``` +or using [pyenv-installer](https://github.com/pyenv/pyenv-installer) and git -A simple workaround is to use `/Users//.pyenv/versions/samcli37/bin/black` -instead of `/Users//.pyenv/shims/black`. +```sh +curl https://pyenv.run | bash # https://github.com/pyenv/pyenv-installer +exec $SHELL # restart your shell so the path changes take effect +git clone https://github.com/pyenv/pyenv-virtualenv.git $(pyenv root)/plugins/pyenv-virtualenv +exec $SHELL # restart your shell to enable pyenv-virtualenv +``` + +Next, setup a virtual environment and activate it: -#### Pre-commit -If you don't wish to manually run black on each pr or install black manually, we have integrated black into git hooks through [pre-commit](https://pre-commit.com/). -After installing pre-commit, run `pre-commit install` in the root of the project. This will install black for you and run the black formatting on -commit. +```sh +# Assuming you want to develop AWS SAM CLI in Python 3.8.9 +pyenv install 3.8.9 # install Python 3.8.9 using pyenv +pyenv virtualenv 3.8.9 samcli38 # create a virtual environment using 3.8.9 named "samcli38" +pyenv activate samcli38 # activate the virtual environment +``` -### 3. Activate Virtualenv +### 2. Initialize dependencies and create `samdev` available in `$PATH` -Virtualenv allows you to install required libraries outside of the -Python installation. A good practice is to setup a different virtualenv -for each project. [pyenv](https://github.com/pyenv/pyenv) comes with a -handy plugin that can create virtualenv. +Clone the AWS SAM CLI repository to your local machine if you haven't done that yet. -Depending on the python version, the following commands would change to -be the appropriate python version. +```sh +# Using SSH +git clone git@github.com:aws/aws-sam-cli.git +``` +or +```sh +# Using HTTPS +git clone https://github.com/aws/aws-sam-cli.git +``` -1. `pyenv virtualenv 3.7.2 samcli37` -2. `pyenv activate samcli37` for Python3.7 +(make sure you have virtual environment activated) -### 4. Install dev version of SAM CLI +```sh +cd aws-sam-cli +make init # this will put a file `samdev` available in $PATH +``` -We will install a development version of SAM CLI from source into the -virtualenv for you to try out the CLI as you make changes. We will -install in a command called `samdev` to keep it separate from a global -SAM CLI installation, if any. +Now you can verify whether the dev AWS SAM CLI is available: -1. Activate Virtualenv: `pyenv activate samcli37` -2. Install dev CLI: `make init` -3. Make sure installation succeeded: `which samdev` +```sh +samdev --version # this will print something like "SAM CLI, version x.xx.x" +``` -### 5. (Optional) Install development version of SAM Transformer +#### Try out to make change to AWS SAM CLI (Optional) -If you want to run the latest version of [SAM -Transformer](https://github.com/awslabs/serverless-application-model/), -you can clone it locally and install it in your pyenv. This is useful if -you want to validate your templates against any new, unreleased SAM -features ahead of time. +```sh +# Change the AWS SAM CLI version to 123.456.789 +echo '__version__ = "123.456.789"' >> samcli/__init__.py +samdev --version # this will print "SAM CLI, version 123.456.789" +``` + +### 3. (Optional) Install development version of SAM Transformer -This step is optional and will use the specified version of -aws-sam-transformer from PyPi by default. +If you want to run the latest version of [SAM Transformer](https://github.com/aws/serverless-application-model/) +or work on it at the same time, you can clone it locally and install it in your virtual environment. +This is useful if you want to validate your templates against any new, unreleased SAM features ahead of time. -``cd ~/projects (cd into the directory where you usually place projects)`` +```sh +# Make sure it is not in AWS SAM CLI repository -``git clone https://github.com/awslabs/serverless-application-model/`` +# clone the AWS SAM repo +git clone git@github.com:aws/serverless-application-model.git +# or using HTTPS: git clone https://github.com/aws/serverless-application-model.git -``git checkout develop `` +cd serverless-application-model +``` -Install the SAM Transformer in editable mode so that all changes you make to the SAM Transformer locally are immediately picked up for SAM CLI. +Make sure you are in the same virtual environment as the one you are using with SAM CLI. +```sh +source /.venv/bin/activate # if you chose to use venv to setup the virtual environment +# or +pyenv activate samcli38 # if you chose to use pyenv to setup the virtual environment +``` -``pip install -e . `` +Install the SAM Transformer in editable mode so that +all changes you make to the SAM Transformer locally are immediately picked up for SAM CLI. + +```sh +pip install -e . +``` Move back to your SAM CLI directory and re-run init, If necessary: open requirements/base.txt and replace the version number of aws-sam-translator with the ``version number`` specified in your local version of `serverless-application-model/samtranslator/__init__.py` -``cd ../aws-sam-cli`` - -``make init`` +```sh +# Make sure you are back to your SAM CLI directory +make init +``` + +## Making a Pull Request + +Above demonstrates how to setup the environment, which is enough +to play with the AWS SAM CLI source code. However, if you want to +contribute to the repository, there are a few more things to consider. + +### Make Sure AWS SAM CLI Work in Multiple Python Versions + +We support 3.6, 3.7 and 3.8 versions. Our CI/CD pipeline is setup to run +unit tests against all Python versions. So make sure you test it +with all versions before sending a Pull Request. +See [Unit testing with multiple Python versions](#unit-testing-with-multiple-python-versions). -Running Tests -------------- +If you chose to use `pyenv` in the previous session, setting up a +different Python version should be easy: -### Unit testing with one Python version +(assuming you are in virtual environment `samcli38`) + +```sh +# Your shell now should looks like "(samcli38) $" +pyenv deactivate samcli38 # "(samcli38)" will disappear +pyenv install 3.7.10 # one time setup +pyenv virtualenv 3.7.10 samcli37 # one time setup +pyenv activate samcli37 +# Your shell now should looks like "(samcli37) $" + +# You can verify the version of Python +python --version # Python 3.7.10 + +make init # one time setup, this will put a file `samdev` available in $PATH +``` + +### Format Python Code + +We format our code using [Black](https://github.com/python/black) and verify the source code is +black compliant in AppVeyor during PRs. Black will be installed automatically with `make init`. + +There are generally 3 options to make sure your change is compliant with our formatting standard: + +#### (Option 1) Run `make black` + +```sh +make black +``` + +#### (Option 2) Integrating Black directly in your favorite IDE + +Since black is installed in virtualenv, when you follow [this instruction](https://black.readthedocs.io/en/stable/editor_integration.html), `which black` might give you this + +``` +/Users//.pyenv/shims/black +``` + +However, IDEs such PyChaim (using FileWatcher) will have a hard time +invoking `/Users//.pyenv/shims/black` +and this will happen: + +``` +pyenv: black: command not found + +The `black' command exists in these Python versions: + 3.8.9/envs/samcli38 + samcli38 +``` + +A simple workaround is to use `/Users//.pyenv/versions/samcli37/bin/black` +instead of `/Users//.pyenv/shims/black`. + +#### (Option 3) Pre-commit + +We have integrated black into git hooks through [pre-commit](https://pre-commit.com/). +After installing pre-commit, run `pre-commit install` in the root of the project. This will install black for you and run the black formatting on commit. + +### Do a Local PR Check + +This commands will run the AWS SAM CLI code through various checks including +lint, formatter, unit tests, function tests, and so on. +```sh +make pr +``` -If you're trying to do a quick run, it's ok to use the current python version. Run `make pr`. +We also suggest to run `make pr` in all Python versions. -### Unit testing with multiple Python versions +#### Unit Testing with Multiple Python Versions (Optional) Currently, SAM CLI only supports Python3 versions (see setup.py for exact versions). For the most part, code that works in Python3.6 will work in Python3.7. You only run into problems if you are @@ -148,7 +235,7 @@ will not work in Python3.6). If you want to test in many versions, you can creat each version and flip between them (sourcing the activate script). Typically, we run all tests in one python version locally and then have our ci (appveyor) run all supported versions. -### Integration Test +#### Integration Test (Optional) `make integ-test` - To run integration test against global SAM CLI installation. It looks for a command named `sam` in your shell. @@ -158,8 +245,8 @@ development version of SAM CLI. This is useful if you are making changes to the CLI and want to verify that it works. It is a good practice to run integration tests before submitting a pull request. -Code Conventions ----------------- +## Other Topics +### Code Conventions Please follow these code conventions when making your changes. This will align your code to the same conventions used in rest of the package and @@ -198,8 +285,7 @@ conventions are best practices that we have learnt over time. comments. -Testing -------- +### Our Testing Practices We need thorough test coverage to ensure the code change works today, and continues to work in future. When you make a code change, use the @@ -230,8 +316,7 @@ following framework to decide the kinds of tests to write: calling AWS APIs, spinning up Docker containers, mutating files etc. -Design Document ---------------- +### Design Document A design document is a written description of the feature/capability you are building. We have a [design document