feat: Deprecating `infrastructure-pipelines` architecture docs #1683

yhakbar · 2024-04-30T16:00:00Z

Deprecating the legacy documentation for Gruntwork Pipelines that leveraged the infrastructure-pipelines repository.

netlify · 2024-04-30T16:00:17Z

✅ Deploy Preview for pensive-meitner-faaeee ready!

Name	Link
🔨 Latest commit	`921205d`
🔍 Latest deploy log	https://app.netlify.com/sites/pensive-meitner-faaeee/deploys/663a9772db969a000838d780
😎 Deploy Preview	https://deploy-preview-1683--pensive-meitner-faaeee.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

… docs

…t this can be shared earlier

ZachGoldberg

Went through most of this, made one slight change. Really great work here Yousif! I think there's more we can clarify about how access control works, future Pipelines actions, highlighting the enterprise delegated-repo feature etc., but none of that IMO should be blocking for a day 1 launch of these docs.

…m:gruntwork-io/docs into feat/deprecating-infrastructure-pipelines

josh-padnick

Yousif, this is great! I left a bunch of comments, but to be clear, feel free to merge whenever you'd like, as we'll wind up iterating on these docs several times.

Overall, the docs do a great job of discovering how Pipelines works under the hood, though, there's very limited info on:

How to install it as a user or a new team (or whether you can install it outside of DF)
How to use and configure it as an end user
What the actual UX is like (it might be worth showing a screenshot so users can imagine the flow without having to set it all up)

For next steps, perhaps you could include what you can from the above, and in a follow-up docs update we can fill in the gaps around usability, configuration and UX.

josh-padnick · 2024-05-08T01:31:45Z

_docs-sources/pipelines/overview/index.md


-Gruntwork Pipelines runs as a binary and series of steps in GitHub Actions, determining what _actions_ need to be taken, in which _environments_, based on the [_infrastructure changes_](#infrastructure-change) that occurred.
+Gruntwork Pipelines runs as a combination of a [binary](https://github.com/gruntwork-io/pipelines-cli) and a [GitHub Workflow](https://github.com/gruntwork-io/pipelines-workflows) composed of [GitHub Actions](https://github.com/search?q=topic%3Aaction+topic%3Apipelines+org%3Agruntwork-io&type=repositories), determining what _actions_ need to be taken, in which _environments_, based on the [_infrastructure changes_](#infrastructure-change) that occurred.


Nit: The GitHub workflow link is public, but the docs site pops up a "this is a private" dialog box. I'm not sure if we have any way to annotate that a link is public.

josh-padnick · 2024-05-08T01:33:30Z

_docs-sources/pipelines/overview/index.md


 ### Infrastructure change set

 Sometimes users create a pull request that changes more than one file at a time. And sometimes a change to a single file affects how multiple other files will be "applied" to your live cloud account. For example, many users of Terragrunt use a pattern known as "envcommon" as a way to specify a default set of values for modules.

-When more than one infra-change is made at a time, we call this an _infrastructure-change set._ We sometimes call this an "infra-change set" for short.
+When more than one infra-change is made at a time, we call this an _infrastructure-change set._

 ### Pipelines actions



Gruntwork is responsible for adding support for a growing library of Pipelines Actions and we will continue to add more over time.

Should we clarify that Gruntwork will add some, but customers can add their own?

josh-padnick · 2024-05-08T01:36:20Z

_docs-sources/pipelines/architecture/index.md


 ## Executor

-The executor takes a pipeline action and infra change as input then performs the action on the infra change. In some cases, like responding to AccountAdded events on merge, the executor will create a pull request in the `infrastructure-live` repository with additional IaC code to provision additional resources.
+The executor takes a pipeline action and infra change as input then performs the action on the infra change. In some cases, like responding to `AccountsAdded` events on merge, the executor will create a subsequent pull request in the `infrastructure-live-root` repository with additional IaC code to baseline the account(s) that were just added.


Suggested change

The executor takes a pipeline action and infra change as input then performs the action on the infra change. In some cases, like responding to `AccountsAdded` events on merge, the executor will create a subsequent pull request in the `infrastructure-live-root` repository with additional IaC code to baseline the account(s) that were just added.

The executor takes a pipeline action and infra change as input, and then performs the action on the infra change. In some cases, like responding to `AccountsAdded` events on merge, the executor will create a subsequent pull request in the `infrastructure-live-root` repository with additional IaC code to baseline the account(s) that were just added.

josh-padnick · 2024-05-08T01:36:32Z

_docs-sources/pipelines/architecture/index.md


-The orchestrator identifies each infrastructure change in the pull request or git commit, classifies the type of infrastructure change it is (e.g. AccountAdded, ModuleChanged, EnvCommonChanged), and determines the right pipelines actions (e.g., `terragrunt plan`, `apply`, or `destroy`) to run based on that infrastructure change. For each infrastructure change, it calls the dispatcher.
+There are two major components in pipelines - the orchestrator and the executor. The orchestrator determines the jobs to be executed and the executor actually executes the jobs and makes any necessary updates in AWS.


Can we create a top-level heading like "Components" and then make "Orchestrator" and "Executor" sub-headings under that?

This way each of the following are on the same hierarchy level:

Components

Execution flow

Trust boundaries

josh-padnick · 2024-05-08T01:45:59Z

_docs-sources/pipelines/architecture/index.md


 ## Executor

-The executor takes a pipeline action and infra change as input then performs the action on the infra change. In some cases, like responding to AccountAdded events on merge, the executor will create a pull request in the `infrastructure-live` repository with additional IaC code to provision additional resources.
+The executor takes a pipeline action and infra change as input then performs the action on the infra change. In some cases, like responding to `AccountsAdded` events on merge, the executor will create a subsequent pull request in the `infrastructure-live-root` repository with additional IaC code to baseline the account(s) that were just added.


Suggested change

The executor takes a pipeline action and infra change as input then performs the action on the infra change. In some cases, like responding to `AccountsAdded` events on merge, the executor will create a subsequent pull request in the `infrastructure-live-root` repository with additional IaC code to baseline the account(s) that were just added.

The executor takes a pipeline action and infra change as input from the orchestrator, and then performs the specified pipelines actions on the specified infra change.

For example, the orchestrator might specify that there was a `ModuleChanged` infra-change on `/dev/us-west-2/database` and that the executor should execute a `terragrunt apply`. The executor will then run `terragrunt apply` on `/dev/us-west-2/database`.

In some cases, the executor will create a new pull request in a different git repo after the original pull request is merged. For example, when Pipelines detects an `AccountsAdded` infra-change, once the pull request is merged, Pipelines will open a new pull request in the `infrastructure-live-root` repository with additional IaC code to add a configuration baseline to the AWS account(s) that were just added.

josh-padnick · 2024-05-08T01:59:22Z

_docs-sources/pipelines/security/repository-access.md

@@ -1,9 +1,45 @@
 # Repository Access


This page is great, but it assumes prior knowledge of what each of the three repos is. I argued in an earlier comment that we should describe those repos in the architecture page. Perhaps it's worth including an info callout that points back to that here?

josh-padnick · 2024-05-08T02:04:03Z

_docs-sources/pipelines/security/repository-access.md


-The diagram below visually illustrates the above teams:
+Gruntwork recommends that you grant permissions to GitHub repositories by defining three [GitHub Teams](https://docs.github.com/en/organizations/organizing-members-into-teams/about-teams), which should map to three separate personas in your organization. Each team and its permissions are designed to apply the [_principle of least privilege_](https://en.wikipedia.org/wiki/Principle_of_least_privilege) to each individual (or machine user) in your organization for them to be able to perform changes to your infrastructure.


Suggested change

Gruntwork recommends that you grant permissions to GitHub repositories by defining three [GitHub Teams](https://docs.github.com/en/organizations/organizing-members-into-teams/about-teams), which should map to three separate personas in your organization. Each team and its permissions are designed to apply the [_principle of least privilege_](https://en.wikipedia.org/wiki/Principle_of_least_privilege) to each individual (or machine user) in your organization for them to be able to perform changes to your infrastructure.

## Recommended roles and GitHub teams

Gruntwork recommends that you grant permissions to GitHub repositories by defining three [GitHub Teams](https://docs.github.com/en/organizations/organizing-members-into-teams/about-teams), which should map to three separate personas in your organization. Each team and its permissions are designed to apply the [_principle of least privilege_](https://en.wikipedia.org/wiki/Principle_of_least_privilege) to each individual (or machine user) in your organization for them to be able to perform changes to your infrastructure.

I found the team names a little confusing and hard to follow. Perhaps we could identify and describe the roles and then recommend team names associated with those roles?

For example, I see the following roles based on your write-up:

Root

Infrastructure collaborators

Infrastructure observers

josh-padnick · 2024-05-08T02:06:36Z

_docs-sources/infrastructure-pipelines/security/branch-protection.md

+This documentation pertains to an old version of Gruntwork Pipelines which used the `infrastructure-pipelines` repository. [Click here](../../pipelines/overview/) to view documentation for the most recent version.
+:::
+
+Gruntwork Pipelines is designed to be used with a PR based workflow.


Suggested change

Gruntwork Pipelines is designed to be used with a PR based workflow.

Gruntwork Pipelines is designed to be used with a pull-request (PR) based workflow.

Some users may not be familiar with that acronym. That being said, I think we only need to define it once per page.

josh-padnick · 2024-05-08T02:08:53Z

_docs-sources/pipelines/security/machine-users.md

@@ -3,7 +3,13 @@ import TabItem from '@theme/TabItem';

 # Machine users


This content is excellent, but it seems like the closest thing we have to "installation" instructions. Perhaps we should create a new sidebar section called "Initial setup" and list this there? Also, could you clarify when a user would need to do this? Or perhaps we should clarify explicitly that Gruntwork Pipelines is only available with DF, but will be available for standalone installation soon?

josh-padnick · 2024-05-08T02:10:57Z

sidebars/pipelines.js

@@ -103,16 +81,27 @@ const sidebar = [
        collapsible: false,
        items: [
          {
-            label: "Upgrading from ECS Deploy Runner",
+            label: "Infrastructure Pipelines",


Hm, we're adopting the terms "Infrastructure Pipelines" and "Gruntwork Pipelines" but that seems awfully subtle. Not sure what else to use, though...

yhakbar · 2024-05-08T13:34:42Z

Yousif, this is great! I left a bunch of comments, but to be clear, feel free to merge whenever you'd like, as we'll wind up iterating on these docs several times.

Overall, the docs do a great job of discovering how Pipelines works under the hood, though, there's very limited info on:
* How to install it as a user or a new team (or whether you can install it outside of DF)

* How to use and configure it as an end user

* What the actual UX is like (it might be worth showing a screenshot so users can imagine the flow without having to set it all up)
For next steps, perhaps you could include what you can from the above, and in a follow-up docs update we can fill in the gaps around usability, configuration and UX.

Given that nothing seems to require stopping the show, I will merge this, then circle back to your feedback!

WIP - feat: Deprecating infrastructure-pipelines architecture docs

42a5eb7

yhakbar and others added 7 commits May 1, 2024 12:32

WIP - feat: Updating Pipelines security and overview docs

64fd12e

WIP - feat: Updating architecture, data-collection and maintain…

af92767

… docs

feat: Regenerating the docs directory

81e2e48

fix: Removing older sidebar entries

f72af71

fix: Fixing link to /pipelines/maintain/extending

3bae317

fix: Setting the infrastructure-pipelines upgrade doc to WIP so tha…

374a5d6

…t this can be shared earlier

chore: make docs more time-independent

8255994

ZachGoldberg previously approved these changes May 2, 2024

View reviewed changes

fix: Cleaning up machine user documentation for clarity

73ca568

yhakbar dismissed ZachGoldberg’s stale review via 73ca568 May 6, 2024 21:09

yhakbar and others added 13 commits May 6, 2024 19:49

feat: Adding migration guide

e67e2f1

fix: Running local regeneration

20c8354

fix: Cleaning up docs

5b05704

fix: Cleaning up deprecation docs

6c9f1a0

fix: Fixing broken link

c419504

More cleanup

2814ede

fix: Cleaning up docs

cfa9388

Update deprecation message

9662d1c

Merge branch 'feat/deprecating-infrastructure-pipelines' of github.co…

c558f63

…m:gruntwork-io/docs into feat/deprecating-infrastructure-pipelines

fix: Adding language to make customers feel safer

197df0d

fix: Fixing broken architecture link

4fae7cf

fix: Adding sidebar for infrastructure-pipelines

965db00

fix: Reverting change to docs/guides/stay-up-to-date/index.md

2b588fa

yhakbar changed the title ~~WIP - feat: Deprecating infrastructure-pipelines architecture docs~~ feat: Deprecating infrastructure-pipelines architecture docs May 7, 2024

yhakbar marked this pull request as ready for review May 7, 2024 16:34

yhakbar requested review from ebeneliason and oredavids as code owners May 7, 2024 16:34

Update sidebars

e56d8c3

oredavids and others added 13 commits May 7, 2024 11:04

Update infrastructure-pipelines layout

f3c69dd

Update upgrade guide

0655931

fix: Cleaning up language for org repo admin token

fd430b7

fix: Cleaning up language for fine grained tokens

dc4171c

fix: Improving wording for organization secrets

a7ae65d

fix: Improving syntax on warnings

7fa16e3

fix: Adjusting permissions for the ci-user

6d19671

fix: Typo

10d24bc

Update initial setup docs

30037cf

Merge branch 'feat/deprecating-infrastructure-pipelines' of github.co…

d27602f

…m:gruntwork-io/docs into feat/deprecating-infrastructure-pipelines

fix: Updating initial-setup docs

dab8650

fix: Typo

04046bb

fix: Adjusting link in legacy docs

921205d

josh-padnick approved these changes May 8, 2024

View reviewed changes

yhakbar merged commit 573dc7c into master May 8, 2024
8 checks passed

yhakbar deleted the feat/deprecating-infrastructure-pipelines branch May 8, 2024 13:34

yhakbar added a commit that referenced this pull request May 8, 2024

fix: WIP - Addressing feedback from #1683

9ce3465

yhakbar mentioned this pull request May 8, 2024

fix: WIP - Addressing feedback from #1683 #1699

Closed

yhakbar added a commit that referenced this pull request May 9, 2024

fix: WIP - Addressing more feedback from #1683

757776e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Deprecating `infrastructure-pipelines` architecture docs #1683

feat: Deprecating `infrastructure-pipelines` architecture docs #1683

yhakbar commented Apr 30, 2024

netlify bot commented Apr 30, 2024 •

edited

Loading

ZachGoldberg left a comment

josh-padnick left a comment •

edited

Loading

josh-padnick May 8, 2024

josh-padnick May 8, 2024

josh-padnick May 8, 2024

josh-padnick May 8, 2024

josh-padnick May 8, 2024

josh-padnick May 8, 2024

josh-padnick May 8, 2024

josh-padnick May 8, 2024

josh-padnick May 8, 2024

josh-padnick May 8, 2024

josh-padnick May 8, 2024

yhakbar commented May 8, 2024


		Gruntwork Pipelines runs as a binary and series of steps in GitHub Actions, determining what _actions_ need to be taken, in which _environments_, based on the [_infrastructure changes_](#infrastructure-change) that occurred.
		Gruntwork Pipelines runs as a combination of a [binary](https://github.com/gruntwork-io/pipelines-cli) and a [GitHub Workflow](https://github.com/gruntwork-io/pipelines-workflows) composed of [GitHub Actions](https://github.com/search?q=topic%3Aaction+topic%3Apipelines+org%3Agruntwork-io&type=repositories), determining what _actions_ need to be taken, in which _environments_, based on the [_infrastructure changes_](#infrastructure-change) that occurred.

	The executor takes a pipeline action and infra change as input then performs the action on the infra change. In some cases, like responding to `AccountsAdded` events on merge, the executor will create a subsequent pull request in the `infrastructure-live-root` repository with additional IaC code to baseline the account(s) that were just added.
	The executor takes a pipeline action and infra change as input, and then performs the action on the infra change. In some cases, like responding to `AccountsAdded` events on merge, the executor will create a subsequent pull request in the `infrastructure-live-root` repository with additional IaC code to baseline the account(s) that were just added.


		The orchestrator identifies each infrastructure change in the pull request or git commit, classifies the type of infrastructure change it is (e.g. AccountAdded, ModuleChanged, EnvCommonChanged), and determines the right pipelines actions (e.g., `terragrunt plan`, `apply`, or `destroy`) to run based on that infrastructure change. For each infrastructure change, it calls the dispatcher.
		There are two major components in pipelines - the orchestrator and the executor. The orchestrator determines the jobs to be executed and the executor actually executes the jobs and makes any necessary updates in AWS.

-The executor takes a pipeline action and infra change as input then performs the action on the infra change. In some cases, like responding to `AccountsAdded` events on merge, the executor will create a subsequent pull request in the `infrastructure-live-root` repository with additional IaC code to baseline the account(s) that were just added.
+The executor takes a pipeline action and infra change as input from the orchestrator, and then performs the specified pipelines actions on the specified infra change.
+For example, the orchestrator might specify that there was a `ModuleChanged` infra-change on `/dev/us-west-2/database` and that the executor should execute a `terragrunt apply`. The executor will then run `terragrunt apply` on `/dev/us-west-2/database`.
+In some cases, the executor will create a new pull request in a different git repo after the original pull request is merged. For example, when Pipelines detects an `AccountsAdded` infra-change, once the pull request is merged, Pipelines will open a new pull request in the `infrastructure-live-root` repository with additional IaC code to add a configuration baseline to the AWS account(s) that were just added.


		The diagram below visually illustrates the above teams:
		Gruntwork recommends that you grant permissions to GitHub repositories by defining three [GitHub Teams](https://docs.github.com/en/organizations/organizing-members-into-teams/about-teams), which should map to three separate personas in your organization. Each team and its permissions are designed to apply the [_principle of least privilege_](https://en.wikipedia.org/wiki/Principle_of_least_privilege) to each individual (or machine user) in your organization for them to be able to perform changes to your infrastructure.

	Gruntwork Pipelines is designed to be used with a PR based workflow.
	Gruntwork Pipelines is designed to be used with a pull-request (PR) based workflow.

		@@ -3,7 +3,13 @@ import TabItem from '@theme/TabItem';

		# Machine users

feat: Deprecating infrastructure-pipelines architecture docs #1683

feat: Deprecating infrastructure-pipelines architecture docs #1683

Conversation

yhakbar commented Apr 30, 2024

netlify bot commented Apr 30, 2024 • edited Loading

✅ Deploy Preview for pensive-meitner-faaeee ready!

ZachGoldberg left a comment

Choose a reason for hiding this comment

josh-padnick left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yhakbar commented May 8, 2024

feat: Deprecating `infrastructure-pipelines` architecture docs #1683

feat: Deprecating `infrastructure-pipelines` architecture docs #1683

netlify bot commented Apr 30, 2024 •

edited

Loading

josh-padnick left a comment •

edited

Loading