NAP-360

[chris-bridgett-nandos]

Overview:

• Google's upstream main merged into our release
• A lot of terraform moved from their experimental folder to setup. There is also a migration path for existing fourkeys projects that were provisioned using the previous setup.sh, but we were already using experimental/terraform. Changes in infrastructure have already been made to point at the correct cloudbuild.yaml.
• Added a variable for dashboard provider - default grafana, but our trigger passes in looker (already done in infrastructure).
• All tests passing... it would be nice to keep it that way. 😄
• The terraform plan looks okay to me (new plan further down). Things are mostly just updated in place due to some extra meta.
• There may be some tweaking needed. Everything looks sensible, but I had to unpick the Pager Duty verification stuff a bit, since we deviated too far from the original design.

Overviewed Continued 17/03/22:

• Tested in infra-dev, with both a copy of events_raw from the fourkeys project and generated data. ✅

• Added documentation (see the tail end of setup/README.md for provisioning fourkeys from scratch via Terraform).
• New terraform plan for review.

Tasks:

• Pre-Merge: Backup the events_raw Big Query dataset

[matthew-green-nandos]

Hey @chris-bridgett-nandos, IMHO nothing should change so I find Plan: 8 to add, 16 to change, 7 to destroy worrying or I am miss understanding. My concern is mainly about losing data. Also, I think it's confusing between the older terraform structure vs the newer setup structure. I had made the assumption that we would need to do the following:

• move our code to the same location as the upstream terraform
• use terraform mv to update the state file (back up first or do this in an infra-dev)

I think I would do a dry run in an infra-dev project to understand what is going on.

[matthew-green-nandos]

Also before we do this we need to back up the state file and the BigQuery data.

[chris-bridgett-nandos]

Hi @matthew-green-nandos,

So:

IMHO nothing should change so I find Plan: 8 to add, 16 to change, 7 to destroy worrying or I am miss understanding.

To dissect what is happening a little more:

• Update-in-place:
  • ✅ (16x) All the updates are either added functionality or meta.
• Replace (counts for +1 create and +1 destroy):
  • ⚠️ (5x) Upstream has renamed all the "workers" to "parsers" (i.e. argocd-worker -> argocd-parser). Since parsers are stateless (I'd hope...), it should be safe to let it recreate rather than fiddling with state.
  • ✅ Another replace is because of NAP-371 😒 (and can be safely ignored until that ticket is resolved).
• Create:
  • ✅ There is a new BQ routine for parsing timestamps.
  • ✅ IAM for the would-be relocated event-handler secret (re. NAP-371).
• Destroy:
  • ✅ IAM for the would-be relocated event-handler secret (re. NAP-371).

I believe these changes are okay?

move our code to the same location as the upstream terraform

Correct. Most of the Terraform code that was previously in experimental is now in setup (inline with upstream).

use terraform mv to update the state file (back up first or do this in an infra-dev)

I don't think terraform mv would resolve anything here, since no resources have been re-aliased. We could manipulate the state file to prevent some of the plan, but I'm not sure I see the benefit vs. letting apply do it.

---

Totally onboard re. not losing data. Happy to throw it in an infra-dev and/or backup our BQ dataset when the time comes to merge. In fact, will update the original PR comment with that now.

[matthew-green-nandos]

Hey @chris-bridgett-nandos,

Okay. With the secret value change?

[chris-bridgett-nandos]

Hey @chris-bridgett-nandos,

Okay. [Will](?) the secret value change?

I don't think so.

In fact, going to hop on NAP-371 now and resolve that, that should make things clearer.

[chris-bridgett-nandos]

@matthew-green-nandos

New Cloud Build log post-NAP-371.

The only part that I'm not 100% on is why it's recreating IAM for the secret, but it doesn't overly concern me.

Also raised an infrastructure PR here in order to deploy this stuff in an isolated environment.

[chris-bridgett-nandos]

@matthew-green-nandos ready for review (cannot re-request, since you didn't request changes before).

[matthew-green-nandos]

Some change to just tidy up the code and to make sure PagerDuty keeps working.
I find the code base confusing with Terraform being in experimental and setup. Also the mixing of languages. I am not sure I understand it works as it seems all over the place. IMHO there should be experimental branches rather than folders.

[chris-bridgett-nandos]

@matthew-green-nandos

Some change to just tidy up the code and to make sure PagerDuty keeps working.

Sorted. 👌

I find the code base confusing with Terraform being in experimental and setup.

As far as I'm aware, they represent different approaches. Nothing from experimental is imported when using their intended configuration (the setup script + terraform in the setup directory).

Also the mixing of languages. I am not sure I understand it works as it seems all over the place.

Mixing of languages? Sorry, I'm not sure I follow unless you're referring to HCL/bash/python? (HCL for infrastructure, bash for bootstrapping, python for the application?)

IMHO there should be experimental branches rather than folders.

I don't disagree! This is the maintainer's approach at the moment. I suppose they may originally have wanted to offer visibility of their preliminary terraform approach or other experimental features to start with, without hiding it away in a branch. 🤷

[matthew-green-nandos]

@chris-bridgett-nandos some more thoughts:

As far as I'm aware, they represent different approaches. Nothing from experimental is imported when using their intended configuration (the setup script + terraform in the setup directory).

Should all of our code be in the setup folder then (I might be missing the trees for the wood/forest of changes)?

Mixing of languages? Sorry, I'm not sure I follow unless you're referring to HCL/bash/python? (HCL for infrastructure, bash for bootstrapping, python for the application?)

Yes I was referring to HCL/bash/python

[matthew-green-nandos]

Let's go for it.

[chris-bridgett-nandos]

@chris-bridgett-nandos some more thoughts:

As far as I'm aware, they represent different approaches. Nothing from experimental is imported when using their intended configuration (the setup script + terraform in the setup directory).

Should all of our code be in the setup folder then (I might be missing the trees for the wood/forest of changes)?

Yep, and I believe it is. 🙂

Upstream still maintains some experimental things in the experimental directory, which are of no bearing to us (our triggers now point at the setup directory[1]).

[1] ...or at least they will once this PR is merged. Almost forgot.