Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release v3.2.1 #729

Merged
merged 19 commits into from
May 27, 2024
Merged

Release v3.2.1 #729

merged 19 commits into from
May 27, 2024

Conversation

sbkok
Copy link
Collaborator

@sbkok sbkok commented May 17, 2024

v3.2.1

The upcoming v4 release will introduce breaking changes. As always, it is
recommended to thoroughly review and test the upgrade procedure in a
non-production environment.

Unfortunately, a few issues popped up in ADF v3.2.0 that block a clean
installation in a new environment. Making it harder to test the upgrade
process. This release resolves those issues and includes an updated installer
for ADF to simplify installation.

We hope this shortens the time required to prepare for the v4 upgrade.


Fixes 🐞


Installation enhancements 🚀

This release is the first release with the new installation process baked in.
Please read the Installation Guide
how to install ADF. In case you are upgrading, please follow the admin guide
on updating ADF

instead.

Changes baked into this release to support the new installation process:


By submitting this pull request, I confirm that you can use, modify, copy, and
redistribute this contribution, under the terms of your choice.

sujay0412 and others added 14 commits May 16, 2024 21:51
* add autoescape true

* Update initial_commit.py with space

* Update initial_commit.py

Add autoescape=True to avoid XSS
Co-authored-by: Christoffer Pozeus <christoffer.pozeus.2@consultant.volvo.com>
Co-authored-by: Stewart Wallace <stewwall@amazon.com>
**Why?**

To allow the deletion of the Default VPC if one were to manage the management
account via ADF's Account Management (adf-accounts).

**What?**

Added the missing permissions to delete and describe the default VPCs in the
management account.
…wslabs#700)

**Why?**

When ADF needs to tag or untag the cross-account access role, it fails to do
so.

**What?**

* Granting the required permissions to tag and untag the roles.
* Removal of the invalid `s3:listObjects` permission, the permissions required
  to list objects in a bucket are managed via the `s3:listBucket` permission.
  Which is already present.
**Why?**

With the CloudFormation deployment provider: It would fail to create the
pipeline if one were to set the `change_set_approval` property to `True`,
while the pipeline lives in another region than where it is asked to deploy to.

**What?**

* The approval action should be performed in the region that hosts the
  pipeline. Not the target region.
* New install process

**Why?**

With the upcoming release of ADF, CDK v2 support is added.
The dependencies that go with it, unfortunately increased the deployment size
beyond the limit that is supported by the Serverless Application Repository.

Hence a new installation mechanism had to be developed.

**What?**

This change proposes to use make as the deployment mechanism.
I chose make, as this eases support queries on deployments of old version and
future versions.

For example, by default, the `make` command will install the latest
`aws-sam-cli` and `yq` dependencies. However, if these appear to be
incompatible, we can instruct to use a specific version of the make install
script:

```bash
make update_makefile

make UPDATE_VERSION=make/2.0 update_makefile
```

Similarly, we can instruct using the AWS SAM CLI used at the time the ADF
version was developed by running:

```bash
make clean deps docker version_number git_ignore sam_build post_build
```

In case someone runs into an issue while building or deploying ADF, they can
generate a report of their environment with:

```bash
make version_report
```

This generates a report of all the versions of the build environment.

_Long story short:_
The proposed Makefile enables various paths to resolve and investigate issues.

**Changes**

* Updated the Makefile to act as the new deployment mechanism.
* Moved tox related tasks to test and lint the code to Makefile.tox.
  As these need to run in the virtual environment managed by Tox, not the
  one managed by the new Makefile.
* Updated the docs to support ADF administrators updating and installing
  using the new Makefile.
* Tested with v3.2.0 deployments and the upcoming release of ADF.
* Added support to generate version numbers on customized ADF installations.
* Updated the contribution docs to clarify how to deploy and test.
* Added support to update the Makefile in the future, such that a deployment
  issue in a specific ADF version can be fixed by updating the Makefile.
* Updated GitHub workflows accordingly, plus added new tests to ensure
  the dependencies installed by ADF are compatible with each other.
* Added a new step in the GitHub workflow to test a full build of ADF.
* Updated the Bug Issue template to include the version report and to allow
  the reporter to indicate that they are working on a fix already.

* Add macOS support to generate a version report

* Update contribution guide to include upstream tags fetch step

* Fix contributing guide typo
…wslabs#702)

**Why?**

When running the make / install process of ADF, we need to ensure it will not
continue if some of the tools are not installed yet.

For example, in the past, if you did not install jq, it would still continue
with the build process. Resulting in a broken template at deployment time.

As reported in awslabs#696, the Python version in the environment was set to v3.8,
which resulted in a broken installation process too.

**What?**

* Updated the documentation to indicate that `jq` needs to be available too.
* Added checks in the build process to exit and warn the user about missing
  dependencies.
* Updated the version number of the Makefile.
**Why?**

When you attempt to install ADF the first time, it will bootstrap the
deployment account via the account bootstrap state machine.
This, however, happens just before executing the bootstrap pipeline the first
time. Resulting in missing parameters in the deployment account at the time the
`regional` stack is deployed.

This stack requires the `extensions/terraform/enabled` parameter.
If that is not there yet, which is the case in the situation above, it will
fail to bootstrap the account. Hereby the bootstrap state machine and pipeline
both fail initially.

**What?**

When the bootstrap state machine determines the event details, it should try
to retrieve the parameter in the management account to see if Terraform is
enabled or not. If this is not configured yet, it should default to False.
…ootstrap

**Why?**

Issues: awslabs#659 and awslabs#594.

When installing ADF the first time, the global IAM bootstrap stack that gets
deployed is sourced from the `adf-bootstrap/global-iam.yml`.

The reason for this behaviour is the absence of the `global-iam.yml` file
in the deployment OU bootstrap folder
(`adf-bootstrap/deployment/global-iam.yml`).

It iterates to the parent directory until it finds a `global-iam.yml` to
deploy. Hence, when the `adf-bootstrap/global-iam.yml` gets deployed in the
deployment account, it was looking for the `deployment_account_id` SSM
parameter. That did not get deployed in the deployment account.

**What?**

* Add the creation of the `deployment_account_id` in the deployment account,
  so if the global IAM bootstrap stack failed to deploy before, it will work
  in the next release. This would be the case if the previous deployment failed
  but the same `aws-deployment-framework-bootstrap` repository is used in the
  upgrade.
* When installing the first time, it creates the bootstrap repository.
  At the time of creation, it will copy the
  `adf-bootstrap/deployment/example-global-iam.yml` to
  `adf-bootstrap/deployment/global-iam.yml`.
  The same logic as how ADF creates the initial `adf-bootstrap/global-iam.yml`.
**Why?**

At the moment, pytest failures were ignored due to a change in the Makefile
used to execute tests. The ADF CI GitHub Workflow would result in a
success, even when a test case failed.

**What?**

Fixed by exiting on the first failure using Makefile foreach instead.
awslabs#721)

* Bump jinja2 from 3.1.3 to 3.1.4 in /src/lambda_codebase/initial_commit

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.3 to 3.1.4.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](pallets/jinja@3.1.3...3.1.4)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Move PR 720 into 721

**Why?**

Both requirements need to be changed in synchrony.
Otherwise the pipeline will fail. Hence updating this PR to include the changes from awslabs#720.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Simon Kok <sbkok@users.noreply.github.com>
**Why?**

When you clone the ADF Github repository, it will not fetch the git tags.
The installation docs describe that you should run `git fetch --tags` to
retrieve those. However, if someone forgets to run that command, you run
into a weird error that does not describe what is going wrong in a resolvable
manner:

> Error: Failed to create changeset for the stack: aws-deployment-framework,
> Parameter validation failed:
> Invalid length for parameter Tags[0].Value, value: 0, valid min length: 1

Additionally, if you installed ADF with the updated installer before. Then
forget to fetch the latest git tags when you are about to deploy a new version
of ADF, it will return a version number that is based on the old version
instead of the new one. Making it hard to determine what version of ADF was
installed. Long story short, the version in the `src/template.yml` should never
be newer than the version determined via git tags.

**What?**

Added check to verify that the version number can be determined correctly.

* If it did not return the version number, but an empty string instead:
  * it will test whether there is a .git folder.
    * If not, the user should clone a repository instead.
    * If the .git directory exists, it will instruct the user to fetch the
      git tags.
* If a version number is returned, it will check whether the version number
  matches with the tag.
  * If it does, this means that it is building a released version of ADF.
  * If it does not, the user might not have switched to the right version tag
    yet. Hence, we warn the user to proceed with caution.

Additionally, this change will add a warning when ADF gets deployed that the
specific version and changes should be reviewed and tested in a non-production
environment.

Lastly, a check is added to ensure that the ADF version of the
`src/template.yml` file should not be newer than the version resolved via git
tags. If this is the case, it should instruct the user to fetch the latest tags
from the ADF repository.
@sbkok sbkok added this to the v3.2.1 milestone May 17, 2024
@sbkok sbkok requested review from javydekoning and StewartW May 17, 2024 11:43
javydekoning
javydekoning previously approved these changes May 17, 2024
sbkok and others added 2 commits May 23, 2024 15:55
**Why?**

In case one were to checkout a tagged release, then make changes, ADF will
receive the tagged release number. Instead of tracking the change that was
made in the release number as well.

By preventing these uncommitted changes, it makes it easier to detect what
code is running in a specific version of ADF. So in case someone were to report
an issue we can assume the number indicates which version of the code failed.

**What?**

* Added check for uncommitted changes, these include files that are tracked
  by the git repository.
* Added check for newly created files, these would not be tracked by git yet.
**Why?**

In the next v4 release, the parameters automatically add the required prefixes.
With v3.2, this was not the case yet. Since we are porting changes back into
a new release of v3.2 we need to make sure they are compatible.

**What?**

Fixing the lookup and put path references as required to install and bootstrap
ADF.
@sbkok sbkok force-pushed the fix/v3.2-first-install branch from ec43408 to 86a383f Compare May 23, 2024 13:56
sbkok added a commit to sbkok/aws-deployment-framework that referenced this pull request May 23, 2024
The upcoming v4 release will introduce breaking changes. As always, it is
recommended to thoroughly review and test the upgrade procedure in a
non-production environment.

Unfortunately, a few issues popped up in ADF v3.2.0 that block a clean
installation in a new environment. Making it harder to test the upgrade
process. This release resolves those issues and includes an updated installer
for ADF to simplify installation.

We hope this shortens the time required to prepare for the v4 upgrade.

---

**Fixes 🐞**

* Fix management account config alias through ADF account management (awslabs#596) by @sbkok.
* Fix CodeBuild stage naming bug (awslabs#628) by @pozeus.
* Fix Jinja2 template rendering with autoescape enabled (awslabs#690) by @sujay0412.
* Fix missing deployment_account_id and initial deployment global IAM bootstrap
  (awslabs#686) by @sbkok, resolves awslabs#594 and awslabs#659.
* Fix permissions to enable delete default VPC in management account (awslabs#699) by @sbkok.
* Fix tagging of Cross Account Access role in the management account (awslabs#700) by @sbkok.
* Fix CloudFormation cross-region changeset approval (awslabs#701) by @sbkok.
* Fix clean bootstrap of the deployment account (awslabs#703) by @sbkok, resolves awslabs#696.
* Bump Jinja2 from 3.1.3 to 3.1.4 (awslabs#720 and awslabs#721) by @dependabot.
* Fix account management lambdas in v3.2 (awslabs#729) by @sbkok.

---

**Installation enhancements 🚀**

This release is the first release with the new installation process baked in.
Please read the [Installation Guide](https://github.com/awslabs/aws-deployment-framework/blob/make/latest/docs/installation-guide.md)
how to install ADF. In case you are upgrading, please follow [the admin guide
on updating ADF](https://github.com/awslabs/aws-deployment-framework/blob/make/latest/docs/admin-guide.md#updating-between-versions)
instead.

Changes baked into this release to support the new installation process:

* New install process (awslabs#677) by @sbkok.
* Install: Add checks to ensure installer dependencies are available (awslabs#702) by
  @sbkok.
* Install: Add version checks and pre-deploy warnings (awslabs#726) by @sbkok.
* Ensure tox fails at first pytest failure (awslabs#686) by @sbkok.
javydekoning
javydekoning previously approved these changes May 23, 2024
sbkok added a commit to sbkok/aws-deployment-framework that referenced this pull request May 24, 2024
The upcoming v4 release will introduce breaking changes. As always, it is
recommended to thoroughly review and test the upgrade procedure in a
non-production environment.

Unfortunately, a few issues popped up in ADF v3.2.0 that block a clean
installation in a new environment. Making it harder to test the upgrade
process. This release resolves those issues and includes an updated installer
for ADF to simplify installation.

We hope this shortens the time required to prepare for the v4 upgrade.

---

**Fixes 🐞**

* Fix management account config alias through ADF account management (awslabs#596) by @sbkok.
* Fix CodeBuild stage naming bug (awslabs#628) by @pozeus.
* Fix Jinja2 template rendering with autoescape enabled (awslabs#690) by @sujay0412.
* Fix missing deployment_account_id and initial deployment global IAM bootstrap
  (awslabs#686) by @sbkok, resolves awslabs#594 and awslabs#659.
* Fix permissions to enable delete default VPC in management account (awslabs#699) by @sbkok.
* Fix tagging of Cross Account Access role in the management account (awslabs#700) by @sbkok.
* Fix CloudFormation cross-region changeset approval (awslabs#701) by @sbkok.
* Fix clean bootstrap of the deployment account (awslabs#703) by @sbkok, resolves awslabs#696.
* Bump Jinja2 from 3.1.3 to 3.1.4 (awslabs#720 and awslabs#721) by @dependabot.
* Fix account management lambdas in v3.2 (awslabs#729) by @sbkok.
* Fix management account missing required IAM Tag Role permission in v3.2
  (awslabs#729) by @sbkok.

---

**Installation enhancements 🚀**

This release is the first release with the new installation process baked in.
Please read the [Installation Guide](https://github.com/awslabs/aws-deployment-framework/blob/make/latest/docs/installation-guide.md)
how to install ADF. In case you are upgrading, please follow [the admin guide
on updating ADF](https://github.com/awslabs/aws-deployment-framework/blob/make/latest/docs/admin-guide.md#updating-between-versions)
instead.

Changes baked into this release to support the new installation process:

* New install process (awslabs#677) by @sbkok.
* Ensure tox fails at first pytest failure (awslabs#686) by @sbkok.
* Install: Add checks to ensure installer dependencies are available (awslabs#702) by
  @sbkok.
* Install: Add version checks and pre-deploy warnings (awslabs#726) by @sbkok.
* Install: Add uncommitted changes check (awslabs#733)
@sbkok sbkok force-pushed the fix/v3.2-first-install branch from 86a383f to ac77301 Compare May 24, 2024 18:45
src/template.yml Fixed Show fixed Hide fixed
sbkok added 2 commits May 24, 2024 20:53
**Why?**

Issue: awslabs#728

With the latest Python runtime update provided by AWS Lambda, the boto3
dependencies were upgraded. However, with the account management Lambda
functions, the `aws-xray-sdk` requires botocore v1.29.54. This version
is incompatible with the latest boto3 package that was added to the Lambda
function runtime. Hence, it returned an error as described in 728.

**What?**

To ensure compatible versions are installed, ADF's lambda functions should
set the specific version of boto3 in its requirements.txt file if they rely on
that.
**Why?**

When deploying ADF as a fresh installation, it requires `iam:TagRole`
permission to create the cross-account management role. This was not required
in the past, so existing IAM roles were not impacted.

**What?**

Added the required permission and limited it to the two roles it needs to
create. Additionally, removal of `iam:PassRole` as that was not used anywhere
in the management account.
The upcoming v4 release will introduce breaking changes. As always, it is
recommended to thoroughly review and test the upgrade procedure in a
non-production environment.

Unfortunately, a few issues popped up in ADF v3.2.0 that block a clean
installation in a new environment. Making it harder to test the upgrade
process. This release resolves those issues and includes an updated installer
for ADF to simplify installation.

We hope this shortens the time required to prepare for the v4 upgrade.

---

**Fixes 🐞**

* Fix management account config alias through ADF account management (awslabs#596) by @sbkok.
* Fix CodeBuild stage naming bug (awslabs#628) by @pozeus.
* Fix Jinja2 template rendering with autoescape enabled (awslabs#690) by @sujay0412.
* Fix missing deployment_account_id and initial deployment global IAM bootstrap
  (awslabs#686) by @sbkok, resolves awslabs#594 and awslabs#659.
* Fix permissions to enable delete default VPC in management account (awslabs#699) by @sbkok.
* Fix tagging of Cross Account Access role in the management account (awslabs#700) by @sbkok.
* Fix CloudFormation cross-region changeset approval (awslabs#701) by @sbkok.
* Fix clean bootstrap of the deployment account (awslabs#703) by @sbkok, resolves awslabs#696.
* Bump Jinja2 from 3.1.3 to 3.1.4 (awslabs#720 and awslabs#721) by @dependabot.
* Fix account management lambdas in v3.2 (awslabs#729) by @sbkok.
* Fix management account missing required IAM Tag Role permission in v3.2
  (awslabs#729) by @sbkok.

---

**Installation enhancements 🚀**

This release is the first release with the new installation process baked in.
Please read the [Installation Guide](https://github.com/awslabs/aws-deployment-framework/blob/make/latest/docs/installation-guide.md)
how to install ADF. In case you are upgrading, please follow [the admin guide
on updating ADF](https://github.com/awslabs/aws-deployment-framework/blob/make/latest/docs/admin-guide.md#updating-between-versions)
instead.

Changes baked into this release to support the new installation process:

* New install process (awslabs#677) by @sbkok.
* Ensure tox fails at first pytest failure (awslabs#686) by @sbkok.
* Install: Add checks to ensure installer dependencies are available (awslabs#702) by
  @sbkok.
* Install: Add version checks and pre-deploy warnings (awslabs#726) by @sbkok.
* Install: Add uncommitted changes check (awslabs#733)
@sbkok sbkok force-pushed the fix/v3.2-first-install branch from ac77301 to a70bb12 Compare May 24, 2024 18:56
@sbkok sbkok requested a review from javydekoning May 24, 2024 19:13
@sbkok sbkok merged commit c3f6955 into awslabs:release/v3.2.1 May 27, 2024
6 checks passed
@sbkok sbkok deleted the fix/v3.2-first-install branch May 27, 2024 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants