Migrate from Terragrunt to Terraform

[nyarly]

Overview

Migrate Incubator from Terragrunt to Terraform.

Terragrunt was an important contribution to reusable Terraform at one point, but since its introduction Terraform has added features that make Terragrunt mostly irrelevant. In the meantime, it adds concepts and syntax to the infrastructure code that act as obstacles to newcomers (to ops in general, and Hack for LA in particular) to onboarding and becoming productive.

Action Items

• Investigate translating Terragrunt to Terraform automatically. (The terragrunt tool does this when it runs anyway, so if the intermediate Terraform isn't too bad, maybe we can use/start with that.)
• Migrate the existing Terragrunt code to Terraform.
• Document process for migrating projects to Terraform

This should be a straightforward change, despite touching much of the repo. It's going to be much better to complete this task all at once than to do incrementally. It's probably better to refactor Terraform later if there's a choice between a perfect, partial Terraform translation vs. a complete, imperfect one.

Resources/Instructions

https://terragrunt.gruntwork.io/docs/#reference
https://developer.hashicorp.com/terraform/language

[nyarly]

Translation/refactoring may result in renaming resources (e.g. as that move between modules.) I've recently had a lot of success with https://github.com/padok-team/tfautomv to generate moved blocks to handle this.

[mxajPrice]

@nyarly how is this process going? Are there any updates?

[mxajPrice]

This issue has been open since April and has created a blocker to our deployment process and getting the Access the Data test site up and running. Not deploying the test site runs a risk of putting the Access the Data project on hold as well as any other project that needs deployment.

I have reached out to Jason (@JasonEb) to set up a meeting with Judson (@nyarly) so that Judson can get the AWS credentials he needs to get this process up and running.

[merlinsmagic]

@nyarly Please give us an update on this issue.

Progress: “What is the current status of your project? What have you completed and what is left to do?”
Blockers: “Difficulties or errors encountered.”
Availability: “How much time will you have this week to work on this issue?”
ETA: “When do you expect this issue to be completed?”
Pictures: “Add any pictures of the visual changes made to the site so far.

[nyarly]

As of last week, I have the accesses I need to work with Terragrunt, and I've found a translation path for one of the projects there. I intend this week to clean up the automatic translation to Terraform and issue a pull request.

I also want to build out Access the Data on the Terraform code, as well as adding some functionality to support that in a mature DevOps way. Maybe that properly should be a separate Issue? Opinions solicited, @JasonEb @merlinsmagic @mxajPrice

I think I can get as far as delivering working Terraform for PeopleDepot, as well as documentation as to how that translation was accomplished. I would expect that to serve as a template for other projects, which will need to be followed up by folks who have access to the root configurations for them. Ops as a CoP will need to start driving that process

[merlinsmagic]

Hi @nyarly do we have any new updates on this?

Progress: “What is the current status of your project? What have you completed and what is left to do?”
Blockers: “Difficulties or errors encountered.”
Availability: “How much time will you have this week to work on this issue?”
ETA: “When do you expect this issue to be completed?”
Pictures: “Add any pictures of the visual changes made to the site so far.

[nyarly]

Situation here is that Terragrunt does not appeared to have stored a full state of the shared_resources modules to S3 - maybe that's because they don't all get used, or maybe because it stores them somewhere else. I'm going to reassess the Terragrunt process and see if I can discern anything else out of that. I suspect I'll have something useful on that front soon.

I want to elaborate a little on what "Done" is going to mean for this Issue:

I have access to the people-depot development config. I'm going to try to complete that deploy process, and then write up the process of translating from Terragrunt so that folks responsible for other projects can complete that process.

Once I'm done there, I'll want to pick up the Access The Data configuration, including a deployment that avoids distributing secrets to Ops personnel, and thus will be reliable and resilient to rotating volunteer staff.