From 4abf4ddf49a3481ad88db664fec067a85fe8b699 Mon Sep 17 00:00:00 2001 From: Simon Kok Date: Wed, 10 Jan 2024 17:31:22 +0100 Subject: [PATCH] New install process **Why?** With the upcoming release of ADF, CDK v2 support is added. The dependencies that go with it, unfortunately increased the deployment size beyond the limit that is supported by the Serverless Application Repository. Hence a new installation mechanism had to be developed. **What?** This change proposes to use make as the deployment mechanism. I chose make, as this eases support queries on deployments of old version and future versions. For example, by default, the `make` command will install the latest `aws-sam-cli` and `yq` dependencies. However, if these appear to be incompatible, we can instruct to use a specific version of the make install script: ```bash make update_makefile make UPDATE_VERSION=make/2.0 update_makefile ``` Similarly, we can instruct using the AWS SAM CLI used at the time the ADF version was developed by running: ```bash make clean deps docker version_number git_ignore sam_build post_build ``` In case someone runs into an issue while building or deploying ADF, they can generate a report of their environment with: ```bash make version_report ``` This generates a report of all the versions of the build environment. _Long story short:_ The proposed Makefile enables various paths to resolve and investigate issues. **Changes** * Updated the Makefile to act as the new deployment mechanism. * Moved tox related tasks to test and lint the code to Makefile.tox. As these need to run in the virtual environment managed by Tox, not the one managed by the new Makefile. * Updated the docs to support ADF administrators updating and installing using the new Makefile. * Tested with v3.2.0 deployments and the upcoming release of ADF. * Added support to generate version numbers on customized ADF installations. * Updated the contribution docs to clarify how to deploy and test. * Added support to update the Makefile in the future, such that a deployment issue in a specific ADF version can be fixed by updating the Makefile. * Updated GitHub workflows accordingly, plus added new tests to ensure the dependencies installed by ADF are compatible with each other. * Added a new step in the GitHub workflow to test a full build of ADF. * Updated the Bug Issue template to include the version report and to allow the reporter to indicate that they are working on a fix already. --- .github/ISSUE_TEMPLATE/bug-template.yaml | 16 +- .github/workflows/adf.yml | 13 +- .gitignore | 3 + CONTRIBUTING.md | 71 +- Makefile | 251 ++++++- Makefile.tox | 24 + README.md | 12 +- docs/admin-guide.md | 53 +- docs/installation-guide.md | 676 ++++++++++++++++-- docs/serverless-application-repo.md | 135 ---- requirements-dev.txt | 1 + .../initial_commit/initial_commit.py | 2 +- .../initial_commit/initial_commit.py | 2 +- src/template.yml | 2 +- tox.ini | 3 +- 15 files changed, 965 insertions(+), 299 deletions(-) create mode 100644 Makefile.tox delete mode 100644 docs/serverless-application-repo.md diff --git a/.github/ISSUE_TEMPLATE/bug-template.yaml b/.github/ISSUE_TEMPLATE/bug-template.yaml index 04e283e1a..0366d6da2 100644 --- a/.github/ISSUE_TEMPLATE/bug-template.yaml +++ b/.github/ISSUE_TEMPLATE/bug-template.yaml @@ -59,7 +59,7 @@ body: attributes: label: Possible Solution description: | - Suggest a fix/reason for the bug + Suggest a fix/reason for the bug. validations: required: false - type: textarea @@ -70,6 +70,12 @@ body: Anything else that might be relevant for troubleshooting this bug. Providing context helps us come up with a solution that is most useful in the real world. + + If you are facing a build or deployment issue, please include details + about the environment you are working in. You can run the following + command to capture all relevant details in one report: + + make version_report validations: required: false - type: input @@ -79,3 +85,11 @@ body: description: The ADF version you are running. validations: required: true + - type: checkboxes + attributes: + label: Contributing a fix? + description: > + Are you are able to contribute and resolve this issue? + options: + - label: Yes, I am working on a fix to resolve this issue + required: false diff --git a/.github/workflows/adf.yml b/.github/workflows/adf.yml index 8698087ce..e96f43068 100644 --- a/.github/workflows/adf.yml +++ b/.github/workflows/adf.yml @@ -19,10 +19,13 @@ jobs: - name: Set the correct Node version using nvm shell: bash -l {0} run: nvm install - - name: Install dependencies - run: make init + - name: Test compatibility of source dependencies + run: | + make clean src_deps + echo "Source dependencies are compatible!" - name: Run tox - # Run tox using the version of Python in `PATH` run: | - tox --version - tox + make clean tox + - name: Build fully + run: | + make clean build diff --git a/.gitignore b/.gitignore index 8380271e1..a26a7493a 100644 --- a/.gitignore +++ b/.gitignore @@ -18,9 +18,12 @@ master-deploy.yml shared_layer.zip .aws-sam samconfig.toml +samconfig.yml +samconfig.yaml pipeline.json template-sam.yml deploy.sh +Makefile.new # Byte-compiled / optimized / DLL files __pycache__ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0fe9d7ecd..899b14ecc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -16,33 +16,49 @@ features. When filing an issue, please check [existing open](https://github.com/awslabs/aws-deployment-framework/issues), and [recently closed](https://github.com/awslabs/aws-deployment-framework/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20) -issues to make sure somebody else hasn't already reported the issue. +issues to make sure somebody else has not already reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: - A reproducible test case or series of steps - The version of our code being used +- For deployment related bugs, please include the details returns by the + following command as executed on the development machine: + `make report_versions` - Any modifications you have made relevant to the bug - Anything unusual about your environment or deployment ## Running tests locally -In order to run the tests locally you need a virtual environment that is used -by [tox](https://pypi.org/project/tox/). +In order to run the tests locally you need install the development requirements +in the python virtual environment that is used. -1. Create a virtual environment: `virtualenv .tox/env` -2. Activate the virtual environment: `source .tox/env/bin/activate` -3. Install dependencies: `make init` -4. To run the tests, execute: `tox` +To ensure test execute consistently, ADF relies on +[tox](https://pypi.org/project/tox/). + +The easiest way to install and run these tests is to run: + +```bash +make tox +``` + +Alternatively, you can also install the required dependencies in your +python virtual environment by running: +```bash +pip install -r requirements-dev.txt +``` + +To run the tests, simply execute next: `tox`. +This will create a virtual environment managed by tox to run the tests in. ## Running linters locally You need to have NodeJS and Docker installed on your computer to run MegaLinter locally with MegaLinter Runner. -You can run mega-linter-runner without installation by using `npx` (Run from the -root of the repository!). +You can run mega-linter-runner without installation by using `npx`. +Make sure to execute this from the root of the repository. ```sh npx mega-linter-runner @@ -54,6 +70,13 @@ Some linters can automatically fix findings by running the command below. npx mega-linter-runner --fix ``` +## Deploy your changes + +To deploy your changes, make sure to commit them in your local repository +first. Once committed, follow the [installation +guide](./docs/installation-guide.md) to deploy them into your management +account. + ## Contributing via Pull Requests Contributions via pull requests are much appreciated. @@ -121,6 +144,36 @@ following account ids may be used: - `012345671234` - `123456789012` +## Resolving Build and deployment issues + +Please capture the environment the build/deployment issue occurs in: + +```bash +make version_report +``` + +Before you report an issue, please try again after cleaning the environment: + +```bash +make clean +``` + +Possibly, this issue is fixed already, please update your Makefile and try +again after: +```bash +make update_makefile +``` + +If that did not resolve the issue, please try running: +```bash +make clean deps_build +``` + +Or download an older version of the Makefile by running: +```bash +make UPDATE_VERSION=make/2.0 update_makefile +``` + ## Code of Conduct This project has adopted the diff --git a/Makefile b/Makefile index 1185c8d7d..07d0dfa05 100644 --- a/Makefile +++ b/Makefile @@ -1,28 +1,223 @@ -init: - pip install -r requirements.txt -r requirements-dev.txt - -test: - # Run unit tests - pytest src/lambda_codebase/account -vvv -s -c src/lambda_codebase/account/pytest.ini - pytest src/lambda_codebase/account_processing -vvv -s -c src/lambda_codebase/account_processing/pytest.ini - pytest src/lambda_codebase/initial_commit -vvv -s -c src/lambda_codebase/initial_commit/pytest.ini - pytest src/lambda_codebase/initial_commit/bootstrap_repository -vvv -s -c src/lambda_codebase/initial_commit/bootstrap_repository/pytest.ini - pytest src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase -vvv -s -c src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/pytest.ini - pytest src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/initial_commit -vvv -s -c src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/initial_commit/pytest.ini - pytest src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/determine_default_branch -vvv -s -c src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/determine_default_branch/pytest.ini - pytest src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers -vvv -s -c src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/pytest.ini - pytest src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python -vvv -s -c src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/pytest.ini - pytest src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk -vvv -s -c src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/pytest.ini - pytest src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared -vvv -s -c src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/pytest.ini - -lint: - # Linter performs static analysis to catch latent bugs - find src -iname "*.py" -not -path "src/.aws-sam/*" | xargs pylint --verbose --rcfile .pylintrc - find src -iname "*.yml" -o -iname "*.yaml" -not -path "src/.aws-sam/*" | xargs yamllint -c .yamllint.yml - cfn-lint - -build: - sam build -u - -deploy: - sam deploy --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Makefile versions +MAKEFILE_VERSION := 2.0 +UPDATE_VERSION := make/latest + +# Repository versions +SRC_VERSION := $(shell git describe --tags --match 'v[0-9]*') +SRC_VERSION_TAG_ONLY := $(shell git describe --tags --abbrev=0 --match 'v[0-9]*') + +# ADF Related URLs +SRC_URL_BASE := https://github.com/awslabs/aws-deployment-framework +RAW_URL_BASE := https://raw.githubusercontent.com/awslabs/aws-deployment-framework + +UPDATE_URL := "$(RAW_URL_BASE)/$(UPDATE_VERSION)/Makefile" +SRC_TAGGED_URL_BASE := "$(SRC_URL_BASE)/tree/$(SRC_VERSION_TAG_ONLY)" +MAKE_TAGGED_URL_BASE := "$(SRC_URL_BASE)/tree/make/$(MAKEFILE_VERSION)" + +# Command line colors +CLR_RED := $(shell echo -e "\033[0;31m") +CLR_GREEN := $(shell echo -e "\033[0;32m") +CLR_YELLOW := $(shell echo -e "\033[0;33m") +CLR_BLUE := $(shell echo -e "\033[0;34m") +CLR_END := $(shell echo -e "\033[0m") + +# Files to work with +SAM_VERSIONED_TEMPLATE := ./src/template-sam.yml +SAM_BUILD_DIR := ./.aws-sam/build +SRC_DIR := ./src + +REQUIREMENTS := $(shell find . -maxdepth 1 -name 'requirements.txt' -or -name 'requirements-dev.txt') +SRC_REQUIREMENTS := $(shell find $(SRC_DIR) -name 'requirements.txt' -or -name 'requirements-dev.txt') +PIP_INSTALL_REQUIREMENTS := $(addprefix -r ,$(REQUIREMENTS)) +PIP_INSTALL_SRC_REQUIREMENTS := $(addprefix -r ,$(SRC_REQUIREMENTS)) + +# Actions + +# Default action should come first, jump to build: +all: build + +# Which actions do not create an actual file like make expects: +.PHONY: all clean update_makefile +.PHONY: report_makefile_version report_versions version_report +.PHONY: build_debs deps src_deps docker version_number git_ignore docs tox +.PHONY: pre_build pre_deps_build sam_build post_build build deps_build +.PHONY: pre_deploy_msg pre_deploy sam_deploy post_deploy deploy + +.venv: .venv/is_ready + +.venv/is_ready: + ( \ + test -d .venv || python3 -m venv .venv; \ + touch .venv/is_ready; \ + ) + +clean: + test -e $(SAM_VERSIONED_TEMPLATE) && rm $(SAM_VERSIONED_TEMPLATE) || exit 0 + test -d $(SAM_BUILD_DIR) && rm -r $(SAM_BUILD_DIR) || exit 0 + test -d .venv && rm -r .venv || exit 0 + +update_makefile: report_makefile_version + @( \ + ( \ + which curl && curl -fsSL $(UPDATE_URL) -o ./Makefile.new \ + ) || ( \ + which wget && wget -q $(UPDATE_URL) -O ./Makefile.new \ + ) || ( \ + echo "$(CLR_RED)No curl or wget, please install and try again$(CLR_END)" && \ + exit 1 \ + ); \ + ) + @echo "Updated Makefile info:" + @make -f ./Makefile.new clean build_deps report_makefile_version + @mv ./Makefile.new ./Makefile + @echo "$(CLR_GREEN)Update complete$(CLR_END)" + +report_makefile_version: + @echo "Makefile: v$(MAKEFILE_VERSION) $$(shasum --algorithm 256 Makefile)" + +report_versions: report_makefile_version + @echo "$(CLR_YELLOW)*** Beginning of ADF Version Report ***$(CLR_END)" + @echo "ADF Source version: $(SRC_VERSION)" + @echo "ADF $$(cat src/template.yml | grep SemanticVersion | xargs)" + @echo "" + @echo "Hardware platform: $$(uname --hardware-platform || echo 'n/a')" + @echo "Kernel name: $$(uname --kernel-name || echo 'n/a')" + @echo "Kernel release: $$(uname --kernel-release || echo 'n/a')" + @echo "" + @test -e /etc/os-release && echo "OS Release:" && cat /etc/os-release + @echo "" + @echo "Disk:" + @df -h $$PWD || echo 'N/A' + @echo "" + @echo "Dependencies:" + @echo "docker: $$(docker --version || echo 'n/a')" + @echo "git: $$(git --version || echo 'n/a')" + @echo "make: $$(make --version || echo 'n/a')" + @( \ + source .venv/bin/activate; \ + pip --version; \ + pip list; \ + ) + @echo "" + git status + @echo "$(CLR_YELLOW)*** End of ADF Version Report ***$(CLR_END)" + +version_report: report_versions + +build_deps: .venv + ( \ + source .venv/bin/activate; \ + pip install aws-sam-cli yq; \ + ) + +deps: .venv + ( \ + source .venv/bin/activate; \ + pip install $(PIP_INSTALL_REQUIREMENTS); \ + ) + +src_deps: .venv + ( \ + source .venv/bin/activate; \ + pip install $(PIP_INSTALL_SRC_REQUIREMENTS); \ + ) + +tox: deps + # Run tests via tox + @( \ + source .venv/bin/activate; \ + tox --version; \ + tox; \ + ) + +docker: + @echo "Prepare docker to support all architectures..." + docker run --rm --privileged multiarch/qemu-user-static --reset -p yes + +version_number: .venv + @echo "Generate a new version number..." + cp $(SRC_DIR)/template.yml $(SAM_VERSIONED_TEMPLATE) + ( \ + source .venv/bin/activate; \ + BASE_ADF_VERSION=$$(cat $(SAM_VERSIONED_TEMPLATE) | yq '.Metadata."AWS::ServerlessRepo::Application".SemanticVersion' -r); \ + COMMIT_ADF_VERSION=$(SRC_VERSION); \ + sed "s/Version: $$BASE_ADF_VERSION/Version: $$COMMIT_ADF_VERSION/g" -i $(SAM_VERSIONED_TEMPLATE); \ + ) + +git_ignore: + mkdir -p $(SAM_BUILD_DIR)/InitialCommitHandler/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/initial_commit/pipelines_repository + cp $(SRC_DIR)/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/initial_commit/pipelines_repository/.gitignore $(SAM_BUILD_DIR)/InitialCommitHandler/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/initial_commit/pipelines_repository/.gitignore + cp $(SRC_DIR)/lambda_codebase/initial_commit/bootstrap_repository/.gitignore $(SAM_BUILD_DIR)/InitialCommitHandler/bootstrap_repository/.gitignore + +docs: + @echo "" + @echo "$(CLR_YELLOW)Please use the guides related to ADF $(SRC_VERSION_TAG_ONLY):$(CLR_END)" + @echo "" + @( \ + echo "$(SRC_VERSION_TAG_ONLY)" | grep -E 'v[0-3]\.' &> /dev/null && \ + echo "* $(CLR_BLUE)$(MAKE_TAGGED_URL_BASE)/docs/installation-guide.md$(CLR_END)" || \ + echo "* $(CLR_BLUE)$(SRC_TAGGED_URL_BASE)/docs/installation-guide.md$(CLR_END)"; \ + ) + @echo "" + @echo "* $(CLR_BLUE)$(SRC_TAGGED_URL_BASE)/docs/admin-guide.md$(CLR_END)" + @echo "" + @echo "* $(CLR_BLUE)$(SRC_TAGGED_URL_BASE)/docs/user-guide.md$(CLR_END)" + @echo "" + +pre_build: build_deps docker version_number git_ignore + +pre_deps_build: deps docker version_number git_ignore + +sam_build: + @( \ + source .venv/bin/activate; \ + sam build \ + --use-container \ + --template $(SAM_VERSIONED_TEMPLATE); \ + ) + +post_build: + @rm $(SAM_VERSIONED_TEMPLATE) + @echo "" + @echo "$(CLR_GREEN)ADF built successfully!$(CLR_END)" + @echo "$(CLR_GREEN)To deploy ADF, please run:$(CLR_END) make deploy" + @echo "" + +build: pre_build sam_build post_build + +deps_build: pre_deps_build sam_build post_build + +pre_deploy_msg: + @echo "" + @echo "" + @echo "$(CLR_GREEN)Thank you for deploying ADF, we are about to proceed$(CLR_END)" + @echo "" + @echo "$(CLR_YELLOW)In the next step, a few questions need to be answered.$(CLR_END)" + @echo "$(CLR_YELLOW)Please use the following guide to answer these:$(CLR_END)" + @echo "" + @( \ + echo "$(SRC_VERSION_TAG_ONLY)" | grep -E 'v[0-3]\.' &> /dev/null && \ + echo "$(CLR_BLUE)$(MAKE_TAGGED_URL_BASE)/docs/installation-guide.md$(CLR_END)" || \ + echo "$(CLR_BLUE)$(SRC_TAGGED_URL_BASE)/docs/installation-guide.md$(CLR_END)"; \ + ) + @echo "" + @echo "" + +pre_deploy: build_deps pre_deploy_msg + +sam_deploy: + @( \ + source .venv/bin/activate; \ + sam deploy \ + --guided \ + --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \ + --no-disable-rollback \ + --tags "ADF_VERSION=$(SRC_VERSION)"; \ + ) + +post_deploy: docs + +deploy: pre_deploy sam_deploy post_deploy diff --git a/Makefile.tox b/Makefile.tox new file mode 100644 index 000000000..2631836dc --- /dev/null +++ b/Makefile.tox @@ -0,0 +1,24 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Files to work with +SRC_DIR := ./src +TEST_CONFIGS := $(shell find $(SRC_DIR) -name 'pytest.ini') + +all: test lint + +.PHONY: all test lint + +test: + # # Run unit tests + ( \ + for config in $(TEST_CONFIGS); do \ + pytest $$(dirname $$config) -vvv -s -c $$config; \ + done \ + ) + +lint: + # Linter performs static analysis to catch latent bugs + find $(SRC_DIR) -iname "*.py" | xargs pylint --verbose --rcfile .pylintrc + find $(SRC_DIR) -iname "*.yml" -o -iname "*.yaml" | xargs yamllint -c .yamllint.yml + cfn-lint diff --git a/README.md b/README.md index 64fcba76f..546fcf2a1 100644 --- a/README.md +++ b/README.md @@ -27,12 +27,14 @@ template as its baseline. ## Quick Start -Launch ADF via the -[Serverless Application Repository](https://console.aws.amazon.com/lambda/home?region=us-east-1#/create/app?applicationId=arn:aws:serverlessrepo:us-east-1:112893979820:applications/aws-deployment-framework) -within the AWS Console. - - Refer to the [Installation Guide](docs/installation-guide.md) for Installation steps. -- Refer to the [User Guide](docs/user-guide.md) for using ADF once it is setup. +- Refer to the [Admin Guide](docs/admin-guide.md) on guidance how-to manage and + administrate ADF. +- Refer to the [User Guide](docs/user-guide.md) for using ADF to generate + and manage pipelines. +- Refer to the [Technical Guide](docs/technical-guide.md) if you want to learn + more about the inner workings of ADF. For example in case you want to + contribute or build on top of ADF. - Refer to the [Samples Guide](docs/samples-guide.md) for a detailed walk through of the provided samples. diff --git a/docs/admin-guide.md b/docs/admin-guide.md index 18c60a0f6..4f9b1a5e9 100644 --- a/docs/admin-guide.md +++ b/docs/admin-guide.md @@ -67,11 +67,11 @@ The configuration properties are synced into AWS Systems Manager Parameter Store and are used for certain orchestration options throughout your Organization. -Below is an example of an `adfconfig.yml` file. When you install ADF via the -Serverless Application Repository, some of the information entered at the time -of deployment will be passed into the `adfconfig.yml` that is committed to the -bootstrap repository as a starting point. You can always edit it and push it -back into the bootstrap repository to update any values. +Below is an example of an `adfconfig.yml` file. When you deploy ADF, +the information entered at the time of deployment will be passed into the +`adfconfig.yml` that is committed to the bootstrap repository as a starting +point. You can always edit it and push it back into the bootstrap repository +to update these values. ```yaml roles: @@ -320,7 +320,7 @@ that is moved into a specific Organizational Unit in AWS Organizations. Bootstrapping of AWS accounts is a convenient way to apply a baseline to an account or sub-set of accounts based on the structure of your AWS Organization. -When deploying ADF via the Serverless Application Repository, a CodeCommit +When deploying ADF, a CodeCommit repository titled `aws-deployment-framework-bootstrap` will also be created. This repository acts as an entry point for bootstrapping templates. The definition of which templates are applied to which Organization Unit are defined @@ -473,8 +473,8 @@ template as required, however, the default resources should not be removed. #### Bootstrapping Regions When you setup the initial configuration for the AWS Deployment Framework you -define your parameters in the Serverless Application Repository, some of these -details get placed into the [adfconfig.yml](#adfconfig). +define your initial parameters, a subset of these get placed into the +[adfconfig.yml](#adfconfig). This file defines the regions you will use for not only bootstrapping but which regions will later be used as targets for deployment pipelines. Be sure you read @@ -922,42 +922,21 @@ account in us-east-1. Check the CloudFormation stack output or tag of the - In the outputs tab, it will show the version as the `ADFVersionNumber`. - In the tags on the CloudFormation stack, it is presented as - `serverlessrepo:semanticVersion`. + `ADFVersion` or `serverlessrepo:semanticVersion`. ### Latest ADF version that is available If you want to check which version is the latest one available, go to the -management account in `us-east-1`: - -1. Navigate to the AWS Deployment Framework Serverless Application Repository - _(SAR)_, it can be found - [here](https://console.aws.amazon.com/lambda/home?region=us-east-1#/create/app?applicationId=arn:aws:serverlessrepo:us-east-1:112893979820:applications/aws-deployment-framework). -2. You can find the latest version in the title of the page, like so: - `aws-deployment-framework — version x.y.z`. +[Open Source ADF Repository on GitHub to browse through its +releases](https://github.com/awslabs/aws-deployment-framework/releases). ## Updating Between Versions -Go to the management account in `us-east-1`: - -1. Navigate to the AWS Deployment Framework Serverless Application Repository - _(SAR)_, it can be found - [here](https://console.aws.amazon.com/lambda/home?region=us-east-1#/create/app?applicationId=arn:aws:serverlessrepo:us-east-1:112893979820:applications/aws-deployment-framework). -2. To ease maintaining your ADF installation, it is recommended to ensure that - the values specified reflect what is installed/in use at the moment. - To gather the values, you can either find them in the - `aws-deployment-framework-bootstrap` repository in the `adfconfig.yml` - file. Or by looking up the values that were specified the last time ADF got - installed/updated via the CloudFormation template parameters of the - `serverlessrepo-aws-deployment-framework` stack in `us-east-1`. -3. Tick the box at the bottom that states: _"I acknowledge that this app creates - custom IAM roles and resource policies."_ -4. Click the _Deploy_ button. - -This will take a few minutes to deploy and kick-off your SAR deployment using -CloudFormation. Leave the browser window open until it changes pages. +1. Follow the build and deploy steps, as documented in the [installation guide, + steps 1 to 3](./installation-guide.md). -Your `serverlessrepo-aws-deployment-framework` stack is updating -with new changes that were included in that release of ADF. +The `serverlessrepo-aws-deployment-framework` stack is updated through this +process with new changes that were included in that release of ADF. To check the progress in the management account in `us-east-1`, follow these steps: @@ -980,7 +959,7 @@ steps: in your AWS management account too. To ease this process, the AWS CloudFormation stack will run the - _InitialCommit_ Custom CloudFormation resource when updating via the SAR. + _InitialCommit_ Custom CloudFormation resource when updating ADF. This resource will open a pull request against the default branch (i.e. `main`) on the _bootstrap_ repository with a set of changes that you can optionally choose to merge. If those changes are merged into the default diff --git a/docs/installation-guide.md b/docs/installation-guide.md index 0ae5b640d..1431f574e 100644 --- a/docs/installation-guide.md +++ b/docs/installation-guide.md @@ -1,19 +1,22 @@ # Installation Guide -## Pre-Requisites +## Prerequisites - [awscli](https://aws.amazon.com/cli/). - [git](https://git-scm.com/) - [AWS CodeCommit Setup](https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-https-unixes.html) -- [AWS CloudTrail configured](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html) - in the `us-east-1` region within the AWS Organizations Management AWS Account. +- [docker](https://docs.docker.com/get-docker/) -## ADF-Compability with AWS Control Tower +The steps in this guide should be performed in the AWS Organizations Management +account. + +## Compatibility with AWS Control Tower ADF is fully compatible with [AWS Control Tower](https://aws.amazon.com/de/controltower/). ADF augments AWS Control Tower. A common operations model is defined as follows: -- AWS Control Tower is responsible for AWS Account creation and OU mapping. +- AWS Control Tower is responsible for AWS Account creation and Organization + Unit (OU) mapping. - ADF is responsible for deploying applications as defined in the ADF deployment maps. @@ -21,7 +24,8 @@ In the following, we assume that you install ADF without AWS Control Tower. However, if a specific installation step requires a "AWS Control Tower-specific action, we call those out explicitly. -It is okay to install ADF and AWS Control Tower in different regions. Example: +It is okay to install ADF and AWS Control Tower in different regions. +For example: - Install AWS Control Tower in eu-central-1. - Install ADF in us-east-1. @@ -29,63 +33,586 @@ It is okay to install ADF and AWS Control Tower in different regions. Example: **If you want to use ADF and AWS Control Tower, we recommend that you setup AWS Control Tower prior to installing ADF.** -## Installation Instructions - -1. Ensure you have setup [AWS CloudTrail](https://aws.amazon.com/cloudtrail/) - *(Not the default trail)* in your Management Account that spans **all - regions**, the trail itself can be created in any region. Events [triggered - via - CloudTrail](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_incident-response.html) - for AWS Organizations can only be acted upon in the US East (N. Virginia) - Region. - -2. In the AWS Console from your management account within `us-east-1`, head - over to the Serverless Application Repository *(SAR)*. From there, search - for `aws-deployment-framework` *(or "adf")* (ensure the checkbox - *"Show apps that create custom IAM roles or resource policies"* is checked). - - If you are deploying ADF for the first time, fill in the required parameters - for your specific use-case. For example, if you have no AWS Organization - or dedicated deployment account already created, you can enter an account - name and email address and ADF will create you an AWS Organization, the - deployment OU, along with an AWS Account that will be used to house - deployment pipelines throughout your Organization. - - If you already have an AWS Account you want to use as your deployment - account you can specify its Account ID in the parameter - `DeploymentAccountId` and leave the `DeploymentAccountName` plus - `DeploymentAccountEmail` empty. - - **AWS Control Tower-specific Note:** - If you use AWS Control Tower, we recommend to create the deployment AWS - Account via the account vending feature of AWS Control Tower. - - It is **MANDATORY**, that your designated deployment AWS Account resides in - the OU `deployment` (case-sensitive!). This can't be changed currently. - Otherwise, the ADF deployment will fail! - - Next, specify the `DeploymentAccountMainRegion` parameter as the region that - will host your deployment pipelines and would be considered your main AWS - region. - - In the `DeploymentAccountTargetRegions` section of the parameters - enter a list of AWS Regions that you might want to deploy your resources - or applications into via AWS CodePipeline *(this can be updated whenever)*. - Also specify a main notification endpoint *(email)* to receive updates - about the bootstrap process. - - **AWS Control Tower-specific Note:** - If you use AWS Control Tower, in the `CrossAccountAccessRoleName` section of - the parameters enter the string `AWSControlTowerExecution`. - Alternatively, leave empty for a default ADF setup. - - When you have entered all required information press **'Deploy'**. - -3. As the stack `serverlessrepo-aws-deployment-framework` completes you can now +--------------------------------- + +## 1. Enable CloudTrail + +Ensure you have setup [AWS CloudTrail](https://aws.amazon.com/cloudtrail/) +*(Not the default trail)* in your Management Account that spans **all +regions**, the trail itself can be created in any region. Events [triggered via +CloudTrail](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_incident-response.html) +for AWS Organizations can only be acted upon in the us-east-1 (North Virginia) +region. + +Please use the [AWS CloudTrail +instructions](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html) +to configure the CloudTrail in the `us-east-1` region within the AWS +Organizations Management AWS Account. + +## 2. Setup Your Build Environment + +### 2.1. Local Instructions + +For the following steps, we need to work on a Linux/Unix/macOS environment. +Additionally, you need to have AWS credentials configured to perform the +deployment actions in the AWS Organization Management account. + +Please note that building on *Windows* is not supported, please use the +[Cloud9 instructions](#22-cloud9-instructions) below instead. + +* [git](https://git-scm.com/) + * To test if it is available, run `git --version`. + This should return version 2.30 or later. +* [docker](https://docs.docker.com/get-docker/) + * To test if it is available, run: `docker ps -a`. + This should return a table that is possibly empty. + * Additionally, running `docker --version` should return version 19 or later. +* [make](https://www.gnu.org/software/make/) + * To test if it is available, run `make --version`. + This should return 4.3 or later. +* [python 3](https://www.python.org/downloads/) + * To test if it is available, run `python --version`. + This should return 3.11 or later. + +Install or update the above requirements as needed. +Alternatively, jump to the [Cloud9 instructions](#22-cloud9-instructions) instead +to use a cloud development machine to get started quickly. + +After your environment is setup, please follow the [Build and Deploy +instructions)](#3-build-and-deploy-instructions) to continue. + +### 2.2. Cloud9 Instructions + +These instructions help setup and configure your Cloud9 instance to build +and deploy ADF with in case your local environment did not meet the +requirements. If your local setup is configured and working, feel free to skip +this section and jump to the [Build and Deploy +instructions)](#3-build-and-deploy-instructions) instead. + +Read this [guide](https://docs.aws.amazon.com/cloud9/latest/user-guide/welcome.html) +to learn about AWS Cloud9 and how-to get started. + +Either deploy the Cloud9 in the management account itself, or configure AWS +credentials such that you can perform actions in the AWS Management account. + +Please pick an AWS Cloud9 compatible region to run the instance from. +During the deployment action of ADF, it will ask to perform the deployment +in the AWS Organizations control-plane region. Obviously, the credentials +available or configured in the Cloud9 instance should have access rights to +deploy ADF in the management account in that region. + +The Cloud9 instance is needed to build and deploy ADF. Feel free to delete +the Cloud9 environment after ADF is deployed successfully. + +#### 2.2.1. EC2 Instance + +Any instance type would work. + +The platform should be: **Amazon Linux 2023** or **Ubuntu Server 22.04 LTS** +or later. + +#### 2.2.2. Network Settings + +To build and deploy ADF, you will need to setup a Cloud9 environment that has +credentials setup. The recommended approach as documented +[here](https://docs.aws.amazon.com/cloud9/latest/user-guide/credentials.html) +is to use AWS Managed Temporary Credentials. In Cloud9, it will be able +to operate with your access rights (similarly to how AWS CloudShell works) if +you set it up: +* With SSM, no-ingress. Please note, use SSM as in AWS Systems Manager, *not* + SSH. +* Leave VPC to the default, or place the machine in a public subnet. + +#### 2.2.3. Free Disk Space + +Once the instance is created, you need at least 16 GB of free disk space. +Unfortunately, the default Cloud9 instance volume is too small. + +To change this, the quickest solution is to open the Cloud9 IDE and follow +the [resize the EBS volume guide in the Cloud9 +documentation](https://docs.aws.amazon.com/cloud9/latest/user-guide/move-environment.html#move-environment-resize). + +At the moment of writing, the volume Cloud9 creates is 10 GB, with around 2 GB +of free space. It is recommended to increase this to: **24 GB or larger**. + +Please confirm the free disk space is large enough by running: + +```bash +df -h $PWD +``` + +## 3. Build and Deploy Instructions + +### 3.1. Clone The Repository + +If this is the first-time you clone the repository in your build environment, +please run: + +```bash +git clone https://github.com/awslabs/aws-deployment-framework +``` + +If you cloned it before, you do not need to clone it again. + +### 3.2. Checkout Specific Version + +Before we continue, make sure you navigate into the cloned repository: + +```bash +cd aws-deployment-framework +``` + +By default, if you clone a repository, it checks out the main branch of the +repository. +It is strongly recommended to checkout the tag of the version you would like to +deploy. This makes it easier to investigate potential issues later on. + +First we need to update our local copy and fetch the available versions: + +```bash +git fetch --verbose --tags +``` + +Select the version you would like to deploy from the returned list of versions. +You can checkout a specific version by running: + +```bash +git checkout ${version_tag_goes_here} + +# For example: +git checkout v3.2.0 +``` + +### 3.3. Update Makefile + +Before we continue, we need to make sure we have the latest build script. + +Update the build script (Makefile) to the latest version by running: + +```bash +make update_makefile +``` + +If this throws an error, the specific version of ADF you are trying to use +might not support this command yet. In that this case, run: + +```bash +curl -fsSL \ + https://raw.githubusercontent.com/awslabs/aws-deployment-framework/make/latest/Makefile \ + -o ./Makefile.new +mv ./Makefile.new ./Makefile +``` + +Make sure to run `make clean` next. + +### 3.4. Build ADF + +Build the ADF version with this command next: + +```bash +make +``` + +If this fails, run `make clean` first and try again. + +### 3.5. Notes on a First-Time Deployment + +If you are deploying ADF for the first time, it is recommended to read this +section carefully and determine which parameters are required for your specific +use-case. + +For example, if you have no AWS Organization or dedicated deployment account +yet, you can enter an account name and email address and ADF will create you +an AWS Organization, the deployment organization unit (OU), along with an +AWS Account that will be used to house deployment pipelines throughout your +Organization. + +If you already have an AWS Account you want to use as your deployment +account you can specify its Account ID in the parameter +`DeploymentAccountId` and leave the `DeploymentAccountName` plus +`DeploymentAccountEmail` empty. + +**AWS Control Tower-specific Note:** +If you use AWS Control Tower, we recommend to create the deployment AWS +Account via the account vending feature of AWS Control Tower. + +It is **MANDATORY**, that your designated deployment AWS Account resides in +the `deployment` (case-sensitive!) OU. This cannot be changed currently. +Also make sure there is only one AWS Account in the deployment organization +unit. Without one, or with more than one the ADF deployment will fail! + +The `DeploymentAccountMainRegion` parameter asks for the region that +will host your deployment pipelines and would-be considered your main AWS +region. + +In the `DeploymentAccountTargetRegions` section of the parameters +enter a list of AWS Regions that you might want to deploy your resources +or applications into via AWS CodePipeline *(this can be updated later)*. + +When deploying ADF for the first time, part of the installation process will +automatically create an AWS CodeCommit repository in the management AWS Account +within the `us-east-1` region. It will also make the initial commit to the +default branch of this repository with a default set of examples that act as a +starting point to help define the AWS Account bootstrapping processes for your +Organization. + +Part of the questions that follow will end up in the initial commit into the +repository. These are passed directly the `adfconfig.yml` file prior to it +being committed. + +### 3.6. Notes on Updating ADF + +ADF releases follow the [semantic version number schems](https://semver.org/). +This implies that breaking changes would require a major version upgrade. +Hence if you upgrade from `2.x` to `3.x`, it is recommended to check the +[release notes](https://github.com/awslabs/aws-deployment-framework/releases) +for details and the breaking changes that it introduced. + +### 3.7. Deploy ADF + +Next, to deploy ADF, run: + +```bash +make deploy +``` + +This process will raise several questions, please use the documentation below +to determine what to answer for each. + +Once you deployed ADF this way, it will ask you to save the configuration. +It is recommended to do so and store the `samconfig.toml` file that it +generates in a Wiki or repository. The next time you need to perform an update +you can rely on that configuration file to use the same settings as configured +with the last deployment. + +If this is the first time you deploy using this method, but have an existing +ADF installation. It is recommended to ensure that the values specified reflect +what is installed/in use at the moment. +To gather the values, you can either find them in the +`aws-deployment-framework-bootstrap` repository in the `adfconfig.yml` +file. Or by looking up the values that were specified the last time ADF got +installed/updated via the CloudFormation template parameters of the +`serverlessrepo-aws-deployment-framework` stack in `us-east-1`. + +#### Stack Name + +Recommended value to use: `serverlessrepo-aws-deployment-framework` + +__Explanation:__ +This defines the name of the stack with which ADF is deployed +in AWS CloudFormation. In theory, you can customize the stack name. +However, in case of updates of existing installations, you should use the same +name as used before. Otherwise it will attempt to deploy a separate stack aside +of the original one, which will result in a failure. + +If you deployed ADF v3.2.0 or earlier and you are about to upgrade that, the +name that it used to deploy is the recommended value stated above. + +#### AWS Region + +Value to use depends on the AWS partition it is deployed to: + +* For the AWS partition (most common), use; `us-east-1` +* For the US-Gov partition, use: `us-gov-west-1` + +__Explanation:__ +ADF needs to be deployed in the region where the control plane of the +AWS Organizations service is hosted. Choose one of the above options based on +the partition you deploy ADF to. + +This does not need to be the same region as you are running the development +environment from with Cloud9 for example. Nor does this have to be the same +region as where all the pipelines will be created by ADF. This later region +is configured at a later stage. + +#### Parameter CrossAccountAccessRoleName + +Default value: `OrganizationAccountAccessRole` + +If your AWS Organization is managed via AWS Control Tower, specify +`AWSControlTowerExecution` instead. You can find more information on [using +AWS Control Tower and ADF here](#compatibility-with-aws-control-tower). + +__Explanation:__ +This role is used to deploy the resources for cross-account management by ADF +initially. + +The name of the IAM Role that ADF will use to access other AWS Accounts within +your Organization to create and update base CloudFormation stacks. + +This role must exist in all AWS accounts within your Organization that you +intend to use ADF with. When creating new AWS Accounts via AWS Organizations +you can define an initial role that is created on the account, that role name +should be standardized and can be used as this initial cross-account access +role. + +*This is not required when performing an update between versions of ADF.* + +Please note that changing this value does not change the configured role in an +existing ADF installation. Please update the role to use in the `adfconfig.yml` +file instead, as documented in the [adfconfig section in the Admin +Guide](./admin-guide.md#adfconfig). + +#### Parameter MainNotificationEndpoint + +Optional, default value: (empty) + +Example: `jane@example.com` + +__Explanation:__ +This allows you to configure the main notification endpoint that should be +informed in case the ADF bootstrapping pipeline failed on the management +account. + +*This is not required when performing an update between versions of ADF.* + +Please note that changing this value does not change the configured value in an +existing ADF installation. Please update the value to use in the `adfconfig.yml` +file instead, as documented in the [adfconfig section in the Admin +Guide](./admin-guide.md#adfconfig). + +#### Parameter DeploymentAccountName + +Optional, default value: (empty) + +Only required upon first-install, if you want ADF to create a new +deployment account for you. If you specify this, along with the +DeploymentAccountId, it will use the existing deployment account instead +and ignore this setting. + +Example: `deployment` + +__Explanation:__ +The Name of the centralized Deployment Account. +If you have an existing account you wish to use as the deployment account, +insert the name of the AWS account here. + +*This is not required when performing an update between versions of ADF.* +*Only supported when installing ADF for the first time.* + +Please note that changing this value does not change the account name in an +existing ADF installation. + +#### Parameter DeploymentAccountEmailAddress + +Optional, default value: (empty) + +Example: `jane@example.com` + +Only required upon first-install, if you want ADF to create a new +deployment account for you. If you specify this, along with the +DeploymentAccountId, it will use the existing deployment account instead +and ignore this setting. + +__Explanation:__ +The email address associated with the deployment account, +If you have an existing account you wish to use as the deployment account, +insert the email address of the AWS account here. + +*This is not required when performing an update between versions of ADF.* +*Only supported when installing ADF for the first time.* + +Please note that changing this value does not change the account email in an +existing ADF installation. + +#### Parameter DeploymentAccountAlias + +Optional, default value: (empty) + +Example: `deployment` + +__Explanation:__ +The Alias of the deployment account. The account alias is a globally unique +name for an account that enable things such as custom login URLs. Read more +[here](https://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html#AboutAccountAlias). + +*This is not required when performing an update between versions of ADF.* +*Only supported when installing ADF for the first time.* + +Please note that changing this value does not change the account email in an +existing ADF installation. + +#### Parameter DeploymentAccountId + +Optional, default value: (empty) + +Example: `123456789012` + +Only required upon updates or a first installation if you want ADF to use an +existing deployment account. + +__Explanation:__ +The AWS Account number of the **existing** deployment account, only required +if an existing account should be used. + +A deployment account will be created if this value is omitted. + +Please note that changing this value does not change the account to use in an +existing ADF installation. + +#### Parameter DeploymentAccountMainRegion + +Required, no default value. + +Example: `eu-west-1` + +__Explanation:__ +The AWS region that will centrally hold all AWS CodePipeline Pipelines. +Pipeline deployments can still span multiple regions. However, they are still +stored and viewed from a single region perspective. +This region is considered your default ADF AWS Region. + +Please note that changing this value does not change the configured region in +an existing ADF installation. You need to remove and reinstall ADF to change +the main region. + +#### Parameter DeploymentAccountTargetRegions + +Optional, default value: (empty) + +Example: `eu-central-1,us-east-1` + +__Explanation:__ +An optional comma-separated list of regions that you may want to deploy +resources *(Applications, CloudFormation etc)* into via AWS CodePipeline. +This can always be updated later via the `adfconfig.yml` file. + +You don't need to include the main region in this list. For example, if you +use the example values for the default region and target regions, it will allow +pipelines to deploy to `eu-west-1`, `eu-central-`, and `us-east-1`. + +*This is not required when performing an update between versions of ADF.* +*Only supported when installing ADF for the first time. + +Please note that changing this value does not change the configured regions in +an existing ADF installation. Please update the target regions to use in the +`adfconfig.yml` file instead, as documented in the [adfconfig section in the +Admin Guide](./admin-guide.md#adfconfig). + +#### Parameter ProtectedOUs + +Optional, default value: (empty) + +Example: `ou-123,ou-234` + +__Explanation:__ +An optional comma-separated list of Organization Unit identifiers that you may +want to protect against having bootstrap stacks applied. + +*This is not required when performing an update between versions of ADF.* +*Only supported when installing ADF for the first time. + +Please note that changing this value does not change the configured protected +OUs in an existing ADF installation. +Please update the configuration to use in the `adfconfig.yml` +file instead, as documented in the [adfconfig section in the Admin +Guide](./admin-guide.md#adfconfig). + +#### Parameter LogLevel + +Optional, default value: `INFO` + +Example: `DEBUG` +Valid options are: `DEBUG`, `INFO`, `WARN`, `ERROR`, and `CRITICAL`. + +__Explanation:__ +At what Log Level the ADF should operate, default is INFO. + +#### Confirm changes before deploy + +Recommended: `Yes` + +__Explanation:__ +It is recommended to answer with `Yes` here. +This allows you to check if the change set that is created does not introduce +any breaking changes before it is executed. + +#### Disable rollback + +Recommended: `No` + +__Explanation:__ +It is recommended to answer with `No` here. +Named resources like the AWS Lambda Layer cannot be updated if rollback is +disabled. However, disabling rollback could be useful if you are experiencing +issues and you want to investigate. + +#### Save arguments to configuration file + +Recommended: `Yes` + +__Explanation:__ +It is recommended to answer with `Yes` here. +This will save answers for these questions to the `samconfig.toml` +configuration file. +It is recommended to store the `samconfig.toml` file in a Wiki or repository. +The next time you need to perform an update you can rely on that configuration +file to use the same settings as configured with the last deployment. + +#### SAM configuration file + +Recommended and default value: `samconfig.toml` + +__Explanation:__ +You are allowed to change this to `samconfig.yml` or `samconfig.yaml`. +Any other name is not recommended, as the deployment script would need to know +where to find the file. If you insist on using a specific file, please rename +it after and before you use the build and deploy steps. + +#### SAM configuration environment + +Default value: `default` + +__Explanation:__ +You are allowed to change this, this is especially useful if you deploy ADF to +multiple environments. Keeping the configuration together in a single file. + +__Please note:__ This is the last question, once you provided the answer it +will continue to upload the assets and build the CloudFormation change-set. + +## What happens next? + +Once the **make deploy** command succeeds, the base AWS Deployment Framework +[CloudFormation stack](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks) +is created. + +This stack contains a few notable resources such as: + +- an [AWS CodeCommit + Repository](https://console.aws.amazon.com/codesuite/codecommit/repositories/aws-deployment-framework-bootstrap/browse?region=us-east-1) +- along with an associated [AWS CodeBuild + Project](https://console.aws.amazon.com/codesuite/codebuild/projects/aws-deployment-framework-base-templates/history?region=us-east-1) +- and an [AWS CodePipeline +Pipeline](https://console.aws.amazon.com/codesuite/codepipeline/pipelines/aws-deployment-framework-bootstrap-pipeline/view?region=us-east-1) + +The CodeCommit Repository will have its first commit made to it automatically +in the installation process. This is a starting point for using ADF and helps +define the bootstrap process for your AWS ecosystem. + +If you updated ADF, it will create a pull request to the main branch on the +same repository instead. + +Once the commits land in the main branch, CodePipeline will kick off the AWS +CodeBuild Project to sync the content of the repository with Amazon S3 and +bootstrap the AWS Accounts where required. + +It is recommended to go through [the steps how ADF handles AWS Account +provisioning as documented in the admin +guide](./admin-guide.md#account-provisioning). + +If you updated ADF, instead of the following steps, +follow the steps described in the [admin guide on updating +ADF](./admin-guide.md#updating-between-versions). + +For first-time deployments, the process that deploys ADF continues +automatically in the background, to follow its progress: + +1. Please navigate to the AWS Console in the AWS Management account. + As the stack `serverlessrepo-aws-deployment-framework` completes you can now open AWS CodePipeline from within the management account in `us-east-1` and - see that there is an initial pipeline execution that has been started. + see that there is an initial pipeline execution that started. - When ADF is deployed for the first time, it will make the initial commit + When ADF is deployed for the first-time, it will make the initial commit with the skeleton structure of the `aws-deployment-framework-bootstrap` CodeCommit repository. @@ -94,26 +621,27 @@ AWS Control Tower prior to installing ADF.** via AWS CloudFormation Templates, Service Control Policies or Tagging Policies. -4. As part of the AWS CodePipeline Execution from the previous step, the +2. As part of the AWS CodePipeline Execution from the previous step, the account provisioner component will run *(in CodeBuild)*. - OPTION 4.1: ONLY applies when requesting the creation of a NEW deployment + OPTION 2.1: ONLY applies when requesting the creation of a NEW deployment account AND when using ADF for vending AWS Accounts. - - If you let ADF create a new Deployment account for you - *(by not giving a pre-existing account id when deploying from SAR)*, + - If you let ADF create a new deployment account for you + *(by not giving a pre-existing account id for the deployment account + at the time of ADF deployment)*, then ADF will handle creating and moving this account automatically into the deployment OU. - OPTION 4.2: ONLY applies when reusing an pre-created deployment account - AND when using ADF for vending AWS Accounts + OPTION 2.2: ONLY applies when reusing a pre-created deployment account + AND when using ADF for vending AWS Accounts. - - If you are using a pre-existing deployment account, you will need to - move the account into the deployment OU from within the Organization - console, or add your deployment account into a `.yml` file within the - `adf-accounts` folder *(see docs)*. + - If you are using a pre-existing deployment account, ADF will + move the account into the deployment OU automatically. + It will also configure your deployment account into a `.yml` file + within the `adf-accounts` folder. - OPTION 4.3: ONLY applies when reusing a pre-existing deployment account + OPTION 2.3: ONLY applies when reusing a pre-existing deployment account AND when using AWS Control Tower for vending AWS Accounts - Ensure that the AWS Control Tower-created deployment AWS Account @@ -126,9 +654,9 @@ AWS Control Tower prior to installing ADF.** console for the step function `AccountBootstrappingStateMachine-` in the `us-east-1` region. -5. Once the Step Function has completed, switch roles over to the newly +3. Once the Step Function has completed, switch roles over to the newly bootstrapped deployment account in the region you defined as your main - region from step 2. + region at ADF deployment time. An AWS CodeCommit repository will have been created and will contain the initial skeleton structure committed which serves as a starting point for diff --git a/docs/serverless-application-repo.md b/docs/serverless-application-repo.md deleted file mode 100644 index c3a784657..000000000 --- a/docs/serverless-application-repo.md +++ /dev/null @@ -1,135 +0,0 @@ -# AWS Deployment Framework - -[![Build Status](https://github.com/awslabs/aws-deployment-framework/workflows/ADF%20CI/badge.svg?branch=master)](https://github.com/awslabs/aws-deployment-framework/actions?query=workflow%3AADF%20CI+branch%3Amaster) - -The [AWS Deployment Framework](https://github.com/awslabs/aws-deployment-framework) -*(ADF)* is an extensive and flexible framework to manage and deploy resources -across multiple AWS accounts and regions within an AWS Organization. This -application should be deployed via the SAR in the [Management AWS -account](admin-guide.md#management-account) of your AWS Organization within the -`us-east-1` region. For more information on setting up ADF please see the -[installation -guide](https://github.com/awslabs/aws-deployment-framework/tree/master/docs/installation-guide.md). - -- **Application Name:** The stack name of this application created via AWS - CloudFormation. By default, this should be `aws-deployment-framework`. - -- **CrossAccountAccessRoleName:** The name of the IAM Role that ADF will use to - access other AWS Accounts within your Organization to create and update base - CloudFormation stacks. This role must exist in all AWS accounts within your - Organization that you intend to use ADF with. When creating new AWS Accounts - via AWS Organizations you can define an initial role that is created on the - account, that role name should be standardized and can be used as this initial - cross-account access role. - - *This is not required when performing an update between versions of ADF.* - -- **DeploymentAccountEmailAddress:** The Email address associated with the - Deployment Account, If you have an existing account you wish to use as the - deployment account, insert the email address of the AWS account here. - - *This is not required when performing an update between versions of ADF.* - -- **DeploymentAccountId:** The AWS Account number of the **existing** Deployment - Account, only required if an existing account should be used. A deployment - account will be created if this value is omitted. Only required if using - pre-existing AWS Account as the Deployment Account. - - *This is not required when performing an update between versions of ADF.* - -- **DeploymentAccountMainRegion:** The AWS region that will centrally hold all - AWS CodePipeline Pipelines. Pipeline deployments can still span multiple - regions however they are still stored and viewed from a single region - perspective. This would be considered your default ADF AWS Region. - - *This is not required when performing an update between versions of ADF.* - -- **DeploymentAccountName:** The Name of the centralized Deployment Account. If - you have an existing account you wish to use as the deployment account, insert - the name of the AWS account here. *This is not required when performing an - update between versions of ADF.* - -## InitialCommit - -These are parameters that relate to the setting up the `adfconfig.yml` file and -its initial commit to the bootstrap repository. The `adfconfig.yml` file defines -base level settings for how ADF operates. - -When deploying ADF for the first time, part of the installation process will -automatically create an AWS CodeCommit repository on this AWS Account within the -`us-east-1` region. It will also make the initial commit to the default branch -of this repository with a default set of examples that act as a starting point -to help define the AWS Account bootstrapping processes for your Organization. - -When making this initial commit into the repository, these below settings are -passed directly the `adfconfig.yml` file prior to it being committed. - -- **DeploymentAccountAlias:** The Alias of the Deployment Account. The account - alias is a globally unique name for an account that enable things such as - custom login URLs. Read more - [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html#AboutAccountAlias) - -- **DeploymentAccountTargetRegions:** An optional comma separated list of - regions that you may want to deploy resources *(Applications, CloudFormation - etc)* into via AWS CodePipeline, this can always be updated later via the - `adfconfig.yml` file. **(e.g. `us-west-1`,`eu-west-1`)**. - - *This is not required when performing an update between versions of ADF.* - -- **MainNotificationEndpoint:** An optional Email Address that will receive - notifications in regards to the bootstrapping pipeline on the management - account. - - *This is not required when performing an update between versions of ADF.* - -- **ProtectedOUs:** An optional comma separated list of OU ids that you may want - to protect against having bootstrap stacks applied **(e.g. - `ou-123`,`ou-234`)**. - - *This is not required when performing an update between versions of ADF.* - -## What happens next? - -After hitting **Deploy** the base AWS Deployment Framework -[CloudFormation stack](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks) -will be created. - -This Stack contains a few notable resources such as: - -- an [AWS CodeCommit - Repository](https://console.aws.amazon.com/codesuite/codecommit/repositories/aws-deployment-framework-bootstrap/browse?region=us-east-1) -- along with an associated [AWS CodeBuild - Project](https://console.aws.amazon.com/codesuite/codebuild/projects/aws-deployment-framework-base-templates/history?region=us-east-1) -- and an [AWS CodePipeline -Pipeline](https://console.aws.amazon.com/codesuite/codepipeline/pipelines/aws-deployment-framework-bootstrap-pipeline/view?region=us-east-1) - -The CodeCommit Repository will have its first commit made to it automatically by -the ADF installation process. This is a starting point for using ADF and helps -define the bootstrap process for your AWS Ecosystem. - -Once the initial commit has been made, CodePipeline will run which kicks off AWS -CodeBuild to sync the content of the repository with Amazon S3. - -Once the content is synced, you are ready to bootstrap your Deployment account -by moving it into the deployment OU within AWS Organizations. - -Before bootstrapping an AWS Account its important to understand how ADF handles -AWS Account provisioning. Read more about [AWS Account -Provisioning](./admin-guide.md#account-provisioning) in the admin guide. - -## Upgrading between versions? - -Ensure the **CrossAccountAccessRole** and **Application Name** are the same -value you used for your initial deployment. Click **Deploy**. As part of the -installation phase, a custom CloudFormation resource will make a Pull Request -against the [AWS CodeCommit -Repository](https://console.aws.amazon.com/codesuite/codecommit/repositories/aws-deployment-framework-bootstrap/browse?region=us-east-1) -with the content from the latest release of ADF. - -Ensure none of your specific bootstrap templates are being overwritten or -affected in any way before merging. Once merged, the pipeline will run, this -completes the update process. - -Upgrading from `*2.x` to `3.x*`? See the -[3.0 release notes](https://github.com/awslabs/aws-deployment-framework/releases) -for details. diff --git a/requirements-dev.txt b/requirements-dev.txt index af21021c8..0155bdfe2 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -6,3 +6,4 @@ pytest~=7.4.4 pytest-cov==4.1.0 tox==4.11.4 yamllint==1.33.0 +yq==3.2.3 diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/initial_commit/initial_commit.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/initial_commit/initial_commit.py index 4ceaa6adb..a1db0e5ab 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/initial_commit/initial_commit.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/initial_commit/initial_commit.py @@ -40,7 +40,7 @@ https://github.com/awslabs/aws-deployment-framework/releases/tag/v{0} This PR was automatically created when you deployed version {0} of the -AWS Deployment Framework through the Serverless Application Repository. +AWS Deployment Framework. Review this PR to understand what changes will be made to your bootstrapping repository. If you also made changes to the repository yourself, diff --git a/src/lambda_codebase/initial_commit/initial_commit.py b/src/lambda_codebase/initial_commit/initial_commit.py index 994a890bb..3993c4ad0 100644 --- a/src/lambda_codebase/initial_commit/initial_commit.py +++ b/src/lambda_codebase/initial_commit/initial_commit.py @@ -55,7 +55,7 @@ https://github.com/awslabs/aws-deployment-framework/releases/tag/v{0} This PR was automatically created when you deployed version {0} of the -AWS Deployment Framework through the Serverless Application Repository. +AWS Deployment Framework. Review this PR to understand what changes will be made to your bootstrapping repository. If you also made changes to the repository yourself, diff --git a/src/template.yml b/src/template.yml index b372f9626..ca9ab75dd 100644 --- a/src/template.yml +++ b/src/template.yml @@ -14,7 +14,7 @@ Metadata: Author: AWS ADF Builders Team SpdxLicenseId: Apache-2.0 LicenseUrl: ../LICENSE.txt - ReadmeUrl: ../docs/serverless-application-repo.md + ReadmeUrl: ../docs/installation-guide.md Labels: ["adf", "aws-deployment-framework", "multi-account", "cicd", "devops"] HomePageUrl: https://github.com/awslabs/aws-deployment-framework diff --git a/tox.ini b/tox.ini index 7340d7866..b467c5737 100644 --- a/tox.ini +++ b/tox.ini @@ -49,5 +49,4 @@ deps = -r{toxinidir}/src/lambda_codebase/requirements.txt commands = - make test - make lint + make -f Makefile.tox