Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Deprecating infrastructure-pipelines architecture docs #1683

Merged
merged 36 commits into from
May 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
42a5eb7
WIP - feat: Deprecating `infrastructure-pipelines` architecture docs
yhakbar Apr 30, 2024
64fd12e
WIP - feat: Updating Pipelines `security` and `overview` docs
yhakbar May 1, 2024
af92767
WIP - feat: Updating `architecture`, `data-collection` and `maintain`…
yhakbar May 1, 2024
81e2e48
feat: Regenerating the `docs` directory
yhakbar May 1, 2024
f72af71
fix: Removing older sidebar entries
yhakbar May 1, 2024
3bae317
fix: Fixing link to `/pipelines/maintain/extending`
yhakbar May 1, 2024
374a5d6
fix: Setting the `infrastructure-pipelines` upgrade doc to WIP so tha…
yhakbar May 1, 2024
8255994
chore: make docs more time-independent
ZachGoldberg May 2, 2024
73ca568
fix: Cleaning up machine user documentation for clarity
yhakbar May 6, 2024
e67e2f1
feat: Adding migration guide
yhakbar May 6, 2024
20c8354
fix: Running local regeneration
yhakbar May 6, 2024
5b05704
fix: Cleaning up docs
yhakbar May 6, 2024
6c9f1a0
fix: Cleaning up deprecation docs
yhakbar May 7, 2024
c419504
fix: Fixing broken link
yhakbar May 7, 2024
2814ede
More cleanup
yhakbar May 7, 2024
cfa9388
fix: Cleaning up docs
yhakbar May 7, 2024
9662d1c
Update deprecation message
oredavids May 7, 2024
c558f63
Merge branch 'feat/deprecating-infrastructure-pipelines' of github.co…
oredavids May 7, 2024
197df0d
fix: Adding language to make customers feel safer
yhakbar May 7, 2024
4fae7cf
fix: Fixing broken architecture link
yhakbar May 7, 2024
965db00
fix: Adding sidebar for `infrastructure-pipelines`
yhakbar May 7, 2024
2b588fa
fix: Reverting change to `docs/guides/stay-up-to-date/index.md`
yhakbar May 7, 2024
e56d8c3
Update sidebars
oredavids May 7, 2024
f3c69dd
Update infrastructure-pipelines layout
oredavids May 7, 2024
0655931
Update upgrade guide
oredavids May 7, 2024
fd430b7
fix: Cleaning up language for org repo admin token
yhakbar May 7, 2024
dc4171c
fix: Cleaning up language for fine grained tokens
yhakbar May 7, 2024
a7ae65d
fix: Improving wording for organization secrets
yhakbar May 7, 2024
7fa16e3
fix: Improving syntax on warnings
yhakbar May 7, 2024
6d19671
fix: Adjusting permissions for the `ci-user`
yhakbar May 7, 2024
10d24bc
fix: Typo
yhakbar May 7, 2024
30037cf
Update initial setup docs
oredavids May 7, 2024
d27602f
Merge branch 'feat/deprecating-infrastructure-pipelines' of github.co…
oredavids May 7, 2024
dab8650
fix: Updating `initial-setup` docs
yhakbar May 7, 2024
04046bb
fix: Typo
yhakbar May 7, 2024
921205d
fix: Adjusting link in legacy docs
yhakbar May 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions _docs-sources/ecs-deploy-runner/how-it-works/index.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# How it works

:::info Newer Version Available
This documentation pertains to an old version of Gruntwork Pipelines which used the `infrastructure-pipelines` repository. [Click here](../../pipelines/overview/) to view documentation for the most recent version.
:::

![Gruntwork Pipelines Architecture](/img/guides/build-it-yourself/pipelines/tftg-pipeline-architecture.png)

## External CI Tool
Expand Down
4 changes: 4 additions & 0 deletions _docs-sources/ecs-deploy-runner/maintain/extending.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ import TabItem from '@theme/TabItem';

# Extending your ECS Deploy Runner

:::info Newer Version Available
This documentation pertains to an old version of Gruntwork Pipelines which used the `infrastructure-pipelines` repository. [Click here](../../pipelines/overview/) to view documentation for the most recent version.
:::

Pipelines can be extended in several ways:
- Adding repositories to supporting building Docker images for many applications
- Updating which branches can kick off which jobs
Expand Down
4 changes: 4 additions & 0 deletions _docs-sources/ecs-deploy-runner/maintain/updating.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ import TabItem from '@theme/TabItem';

# Updating Your ECS Deploy Runner

:::info Newer Version Available
This documentation pertains to an old version of Gruntwork Pipelines which used the `infrastructure-pipelines` repository. [Click here](../../pipelines/overview/) to view documentation for the most recent version.
:::

Pipelines is built using the [`terraform-aws-ci`](../../reference/modules/terraform-aws-ci/ecs-deploy-runner/) module. We recommend updating your pipeline whenever there’s a new release of the module.

By default, Pipelines cannot update it’s own infrastructure (ECS cluster, AWS Lambda function, etc), so you must run upgrades to Pipelines manually from your local machine. This safeguard is in place to prevent you from accidentally locking yourself out of the Pipeline when applying a change to permissions.
Expand Down
4 changes: 4 additions & 0 deletions _docs-sources/ecs-deploy-runner/multi-account/index.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Deploying Multi-Account Pipelines

:::info Newer Version Available
This documentation pertains to an old version of Gruntwork Pipelines which used the `infrastructure-pipelines` repository. [Click here](../../pipelines/overview/) to view documentation for the most recent version.
:::

Have you heard about AWS multi-account setups? It's like having a pack of dogs - each one with its own unique personality, strengths, and weaknesses, but all working together to accomplish a common goal.

Imagine you have a pack of dogs, each with their own special skills. You've got a fierce protector who guards the house, a speedy runner who chases down anything that moves, and a snuggly lap dog who just wants to cuddle all day. Each dog has its own needs, but they all rely on you as their owner to provide for them and keep them safe.
Expand Down
4 changes: 0 additions & 4 deletions _docs-sources/ecs-deploy-runner/overview/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
This documentation pertains to an old version of Gruntwork Pipelines which used the ECS Deploy Runner. [Click here](../../pipelines/overview/) to view documentation for the most recent version.
:::


Gruntwork Pipelines is a framework that enables you to use your preferred CI tool to
securely run an end-to-end pipeline for infrastructure code ([Terraform](https://www.terraform.io/)) and
app code ([Docker](https://www.docker.com/) or [Packer](https://www.packer.io/)). Rather than replace your existing CI/CD provider, Gruntwork Pipelines is designed to enhance the security
Expand All @@ -16,6 +15,3 @@ greater than they might otherwise need.
Gruntwork Pipelines allows a highly restricted set of permissions to be supplied to the CI/CD tool while
infrastructure related permissions reside safely within your own AWS account. This reduces the exposure of your
high value AWS secrets.



26 changes: 15 additions & 11 deletions _docs-sources/foundations/iac-foundations/initial-setup.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,31 @@
# Initial setup

:::info Recent Upgrade
This documentation relates to the latest version of Gruntwork Pipelines released in May 2024.

If you are using the older version of Gruntwork Pipelines that includes the `infrastructure-pipelines` repository, click [here](../../infrastructure-pipelines/iac-foundations/initial-setup.md) to get the documentation for that version.
:::

To set up IaC Foundations, we use three pre-configured git repository templates that include best practices and also allow for customization.

For each repository below, navigate to the template repository and select **Use this template** -> **Create a new Repository**. This will initiate repository creation. You should select your org as the owner, add a description if you like, make sure you are creating a **private** repo, and click **Create repository**.

The repository template will be created, and you can follow the instructions in the `README` to bootstrap your IaC Foundations. Gruntwork is available to assist with questions around other patterns as they arise.

### Infrastructure Live Template

_[https://github.com/gruntwork-io/infrastructure-live-template](https://github.com/gruntwork-io/infrastructure-live-template)_
## Infrastructure Live Root Template

This template creates an infrastructure-live repository with scaffolding for a best practices Terragrunt configuration, including patterns for module defaults, global variables, and account baselines. It also configures Gruntwork Pipelines, which is easy to remove if you don't want it.
[infrastructure-live-root-template](https://github.com/gruntwork-io/infrastructure-live-root-template)

### Infrastructure Modules Template
This template creates an infrastructure-live-root repository with scaffolding for a best practices Terragrunt configuration, including patterns for module defaults, global variables, and account baselines. It also configures Gruntwork Pipelines, which is easy to remove if you don't want it.

_[https://github.com/gruntwork-io/infrastructure-modules-template](https://github.com/gruntwork-io/infrastructure-modules-template)_
## Infrastructure Live Access Control Template

This template creates an empty infrastructure-modules repository that will be used to store Terraform/OpenTofu modules that your organization has authored and intends to use within your organization.

### Infrastructure Pipelines Template
[infrastructure-live-access-control-template](https://github.com/gruntwork-io/infrastructure-live-access-control-template)

_[https://github.com/gruntwork-io/infrastructure-pipelines-template](https://github.com/gruntwork-io/infrastructure-pipelines-template)_
This template is only necessary for Enterprise customers, but is recommended for all customers.

This template is only necessary if you plan on implementing [Gruntwork Pipelines](../pipelines).
## Infrastructure Modules Template

[infrastructure-modules-template](https://github.com/gruntwork-io/infrastructure-modules-template)

This template creates an empty infrastructure-modules repository that will be used to store Terraform/OpenTofu modules that your organization has authored and intends to use within your organization.
72 changes: 28 additions & 44 deletions _docs-sources/foundations/landing-zone/add-aws-account.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,44 +5,28 @@ import TabItem from '@theme/TabItem';

This document provides instructions for provisioning a new AWS account using Gruntwork Landing Zone. The described workflow gives you the flexibility to require approval for all new AWS account requests in accordance with the permissions configured in your git repository.

## Prerequisites
:::note

Before proceeding, ensure you have:
Before proceeding, ensure you have set up your infrastructure repositories by following the steps in the [IAC Foundations](/foundations/iac-foundations/initial-setup) guide.

1. An `infrastructure-live` repository created from the [Gruntwork infrastructure-live template](/foundations/iac-foundations/initial-setup#infrastructure-live-template).

If you are just now creating your `infrastructure-live` repo, be sure to follow the Bootstrap steps in the `infrastructure-live` Readme to complete the setup before proceeding.

1. An `infrastructure-pipelines` repository created from the [Gruntwork infrastructure-pipelines template](/foundations/iac-foundations/initial-setup#infrastructure-pipelines-template).

If you are just now creating your `infrastructure-pipelines` repo, be sure to follow the Bootstrap steps in the `infrastructure-pipelines` Readme to complete the setup before proceeding.

1. An installation of [Gruntwork Pipelines](/pipelines/overview)

1. Access tokens and secrets configured for both `infrastructure-live` and `infrastructure-pipelines` repos. Instructions for configuring access tokens and secrets can be found in the respective repo Readme files, as well as in [the Machine Users](/pipelines/security/machine-users) guide.

:::info

Your GitHub org MUST have fine-grained personal access tokens enabled before you can create the access tokens and secrets!

:::
:::

## 1. Create an AWS account request file

To initiate the process, you can use the GitHub Actions workflow in your `infrastructure-live` repo by following the below steps. Alternatively, you can choose to manually create the account request file and associated PR by following the steps in the **Account Request File (Manual)** dropdown below.
To initiate the process, you can use the GitHub Actions workflow in your `infrastructure-live` repo by following the below steps. Alternatively, you can choose to manually create the account request file and associated PR by following the steps in the **Account Request File (Manual)** dropdown below.

You will be required to fill in the following parameters:
You will be required to fill in the following parameters:

- `account_name`: The name of the account in AWS.
- `account_email`: The email address to associate with the account.
- `organizational_unit_name`: The name of the Organizational Unit(OU) where the new account will be added. Ensure this exactly matches an existing OU name in your organization, for example, “Security”.
- `aws_region`: The AWS region where the Gruntwork account baselines will be deployed. e.g., `us-east-1`.
- `org_name_prefix`: The prefix to use for some resources created by Gruntwork modules. For instance, S3 bucket files will be prefixed with this value
- `sso_user_first_name`: The first name of the user to create in AWS IAM Identity Center (Successor to AWS Single Sign-On(SSO)).
- `sso_user_last_name`: The last name of the user to create in AWS IAM Identity Center.
- `sso_user_email`: The email address of the user to create in AWS IAM Identity Center. This user will be able to login to the new account using AWS IAM Identity Center.
- `account_baseline_modules_version`: The version of the your account baseline modules to use for the new account. e.g., `v0.0.1`
- `requested_by`: The GitHub user ID or email address of the entity requesting the new account, for audit purposes.
- `account_name`: The name of the account in AWS.
- `account_email`: The email address to associate with the account.
- `organizational_unit_name`: The name of the Organizational Unit(OU) where the new account will be added. Ensure this exactly matches an existing OU name in your organization, for example, “Security”.
- `aws_region`: The AWS region where the Gruntwork account baselines will be deployed. e.g., `us-east-1`.
- `org_name_prefix`: The prefix to use for some resources created by Gruntwork modules. For instance, S3 bucket files will be prefixed with this value
- `sso_user_first_name`: The first name of the user to create in AWS IAM Identity Center (Successor to AWS Single Sign-On(SSO)).
- `sso_user_last_name`: The last name of the user to create in AWS IAM Identity Center.
- `sso_user_email`: The email address of the user to create in AWS IAM Identity Center. This user will be able to login to the new account using AWS IAM Identity Center.
- `account_baseline_modules_version`: The version of the your account baseline modules to use for the new account. e.g., `v0.0.1`
- `requested_by`: The GitHub user ID or email address of the entity requesting the new account, for audit purposes.

<Tabs groupId="request-aws-account">
<TabItem value="GitHub Action Workflow" label="GitHub Action Workflow" default>
Expand All @@ -62,22 +46,22 @@ The workflow will automatically create a PR to be reviewed.

1. To initiate the process, create an `account-<AWS-ACCOUNT-NAME>.yml` file in the `_new-account-requests` folder located in the root of your `infrastructure-live` repository. This file will be used to create a pull request and add the new account to your organization. The file should have the following format:

```yaml account-<AWS-ACCOUNT-NAME>.yml
account_name: <AWS-ACCOUNT_NAME>
account_email: <ROOT-EMAIL-FOR-ACCOUNT>
organizational_unit_name: <OU-TO-ADD-ACCOUNT-TO>
aws_region: <REGION-T0-DEPLOY-BASELINES>
org_name_prefix: <ORG-NAME-PREFIX-FOR-RESOURCES>
sso_user_first_name: <SSO-USER-FIRST-NAME>
sso_user_last_name: <SSO-USER-LAST-NAME>
sso_user_email: <SSO-USER-EMAIL>
account_baseline_modules_version: <ACCOUNT-BASELINE-MODULES-VERSION>
requested_by: <GITHUB_USER_ID_OR_EMAIL>
```
```yaml account-<AWS-ACCOUNT-NAME>.yml
account_name: <AWS-ACCOUNT_NAME>
account_email: <ROOT-EMAIL-FOR-ACCOUNT>
organizational_unit_name: <OU-TO-ADD-ACCOUNT-TO>
aws_region: <REGION-T0-DEPLOY-BASELINES>
org_name_prefix: <ORG-NAME-PREFIX-FOR-RESOURCES>
sso_user_first_name: <SSO-USER-FIRST-NAME>
sso_user_last_name: <SSO-USER-LAST-NAME>
sso_user_email: <SSO-USER-EMAIL>
account_baseline_modules_version: <ACCOUNT-BASELINE-MODULES-VERSION>
requested_by: <GITHUB_USER_ID_OR_EMAIL>
```

1. Create a pull request

Next, create a pull request containing the new account request file. This action will trigger the Gruntwork pipeline to `terragrunt plan` the new account and update the pull request with the plan output.
Next, create a pull request containing the new account request file. This action will trigger the Gruntwork pipeline to `terragrunt plan` the new account and update the pull request with the plan output.

</TabItem>
</Tabs>
Expand Down
39 changes: 39 additions & 0 deletions _docs-sources/infrastructure-pipelines/architecture/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Architecture

:::info Newer Version Available
This documentation pertains to an old version of Gruntwork Pipelines which used the `infrastructure-pipelines` repository. [Click here](../../pipelines/overview/) to view documentation for the most recent version.
:::

There are three components in pipelines - the orchestrator, the dispatcher, and the executor. The orchestrator determines the jobs that needs to be executed; the dispatcher calls the executor with each job and reports on its status; and the executor actually executes the jobs and makes any necessary updates in AWS.

## Orchestrator

The orchestrator identifies each infrastructure change in the pull request or git commit, classifies the type of infrastructure change it is (e.g. AccountAdded, ModuleChanged, EnvCommonChanged), and determines the right pipelines actions (e.g., `terragrunt plan`, `apply`, or `destroy`) to run based on that infrastructure change. For each infrastructure change, it calls the dispatcher.

## Dispatcher

The dispatcher takes an infrastructure change, a pipelines action, and parameters as input. Then, it calls the executor to run the action on the infrastructure change. Once complete, the dispatcher creates a pull request comment (for PR workflows) or a GitHub issue (for pushes to main) with a link to the GitHub logs.

## Executor

The executor takes a pipeline action and infra change as input then performs the action on the infra change. In some cases, like responding to AccountAdded events on merge, the executor will create a pull request in the `infrastructure-live` repository with additional IaC code to provision additional resources.

## Execution Flow

Pipelines starts with an event in GitHub - a pull request being created, synchronized, or re-opened, and a PR being merged to main (e.g., a push to main). From there, the orchestrator determines the infra-change set and appropriate action to run for each infra-change. The dispatcher then sends the information to the dispatcher, which calls the executor and waits for it to complete. Once complete, the status of the execution is reported to the calling pull request in a comment or as a GitHub issue if an apply or destroy fails.

![Gruntwork Pipelines Execution Flow](/img/pipelines/how-it-works/pipelines_execution_flow.png)

## Isolating IaC definitions and deployment

Due to the necessity of having administrative permissions in your CI system to deploy infrastructure, Gruntwork Pipelines separates where changes occur from where they are applied. Infrastructure is _defined_ in one GitHub repository (commonly named `infrastructure-live`) and the code that _deploys_ your infrastructure is stored in a different GitHub repository (commonly named `infrastructure-pipelines`). This is done to allow you to use the principle of least privilege to each repository. For more information on pipelines’ security model see [controls](../security/controls.md) and [repository access](../security/repository-access.md).

This allows your team to have many collaborators for your IaC in your `infrastructure-live` repository, while permitting a subset of administrators access to the workflows in `infrastructure-pipelines` that can access your AWS environments.

Workflows running in the `infrastructure-live` repository trigger workflows to run in `infrastructure-pipelines`, which runs the appropriate terragrunt command on the changed code to perform changes to your infrastructure. Pipelines uses OIDC to assume a role in your AWS account with a policy that limits access exclusively to the `main` branch of your `infrastructure-pipelines` repository. No other repository or branch can leverage this role to gain access to your AWS accounts.

![Gruntwork Pipelines Architecture](/img/pipelines/how-it-works/pipelines_architecture.png)

:::info Previous Version Available
You are reading documentation for Gruntwork Pipelines. The previous version of Gruntwork Pipelines is known as [ECS Deploy Runner](../../ecs-deploy-runner/overview/).
:::
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Usage Data

:::info Newer Version Available
This documentation pertains to an old version of Gruntwork Pipelines which used the `infrastructure-pipelines` repository. [Click here](../../pipelines/overview/) to view documentation for the most recent version.
:::

Gruntwork Pipelines captures usage data to better understand how our customers use our products. This information includes the duration of pipelines runs, the number of jobs you run, customer name, and application errors that occur during execution. If you would like to disable telemetry, please set the environment variable `PIPELINES_TELEMETRY_OPT_OUT=1` in your CI job.
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Allowing Pipelines Actions in GitHub Actions

:::info Newer Version Available
This documentation pertains to an old version of Gruntwork Pipelines which used the `infrastructure-pipelines` repository. [Click here](../../pipelines/overview/) to view documentation for the most recent version.
:::

Gruntwork Pipelines uses a set of Gruntwork-built reusable Github Actions, which are available in the GitHub Marketplace. Gruntwork is a Verified Creator of GitHub Actions.

### GitHub Enterprise
Expand Down Expand Up @@ -37,11 +41,3 @@ Navigate to each Gruntwork repository to retrieve the latest tagged release for

Currently GitHub Actions does not support selecting specific repos outside of your own GitHub organization for the team and pro tier. To use Gruntwork Pipelines you must select the **Allow all actions and reusable workflows** option in the Actions general settings at the Organization level.



<!-- ##DOCS-SOURCER-START
{
"sourcePlugin": "local-copier",
"hash": "e9c63a765fc90941ee2e214842d318ce"
}
##DOCS-SOURCER-END -->
Loading