Atlantis nodes in different accounts with one repository

[cullenmcdermott]

We have a repository that contains our live terraform definitions for multiple accounts. We currently have 4 accounts and plan to have an Atlantis node in each account. We've tossed around the idea of allowing atlantis to assume a role in each account but this won't work because of network partitions. It would be nice if there was some configuration flag that could be set on the server, or some line in the yaml that could be added to target a specific atlantis url. Currently we will have to have a script run before our plan/apply and check the name of the parent dir and exit 1 if it doesn't match the environment name.

I imagine this would have to be a configuration on the Atlantis server as it doesn't appear that there's any context around the intended target from a quick look through the github and gitlab webhook responses.

[mechastorm]

My team had a similar request for such a feature but we ended up just doing cross-account roles instead for Atlantis to assume. The main blocker is that you still have one repository and one webhook to Atlantis.

One idea we have been thinking about was to have the webhook go to a custom proxy

• The proxy runs a running a custom listener that reads the Github/Gitlab Hook event
• Based on the Github/Gitlab hook payload, redirect the request to the appropriate Atlantis server

[lkysow]

Hi Cullen, thanks for the issue. While I'd prefer to solve this issue by making it easier for users to run a single Atlantis instance, I understand that there will still be use cases where this doesn't make sense.

If #47 is implemented (server-side atlantis.yaml) then what if it let you do something like this:

repos:
- name: github.com/runatlantis/atlantis
  projects:
  - dir: account1
  - dir: account2
    ignore: true

Where if Atlantis got a hook that involved the account2/ dir, it would be ignored. Would that work?

One question, you said:

assume a role in each account but this won't work because of network partitions

Can you expand on that a bit? Because if you're just calling the AWS api then there's no network partition I'm aware of so you must be doing something else.

[cullenmcdermott]

Yes, I believe #47 would solve our problem as long as the dir doesn't need to point to the folder with the TF config in it.

We currently have a directory structure similar to this:

├── README.md
├── accounts
│   ├── account1
│   │   ├── my-app
│   │   │   ├── main.tf
│   │   │   └── terraform.tfvars
│   │   ├── some-database
│   │   │   ├── main.tf
│   │   │   └── terraform.tfvars
│   │   └── terraform.tfvars
│   └── account2
│       ├── my-app
│       │   ├── main.tf
│       │   └── terraform.tfvars
│       ├── some-database
│       │   ├── main.tf
│       │   └── terraform.tfvars
│       └── terraform.tfvars
├── atlantis.yaml

The example you posted would work great as long as it would track any changes within directories underneath the account1 folder.

As far as network partitions I was referring to TF modules that utilize provisioners that need direct access to an EC2 instance, RDS or something similar.

[lkysow]

As far as network partitions I was referring to TF modules that utilize provisioners that need direct access to an EC2 instance, RDS or something similar.

👍

Okay thanks, I'll see what I can do.

[osterman]

Hello from cloudposse. We're a very large maintainer of open source terraform modules (100+) and provide commercial support around terraform and kubernetes.

I'd like to propose an alternative solution that would address our use-case, but also because I think it would be the simplest to implement.

We're new to atlantis, but not new to CI/CD =)

use-case

Our objective is to run atlantis once per AWS account. Each account corresponds to a stage like prod, staging, qa, dev, security, identity, etc. One of our "best practices" is that AWS accounts share nothing, thus should not share atlantis. We define our "root module" invocations here. Versioning terraform is not sufficient. We need to version other dependencies as well for a particular stage (e.g. kubectl, helm and kops). Thus, for a particular stage, we package and ship that in a container. Now, we want to ship atlantis with this container as well and run it in a given account. =)

proposal

Currently, atlantis operates on one manifest: /atlantis.yaml. Every instance of atlantis will look for this one file, if enabled with --allow-repo-config. The complexity of parameterizing this atlantis.yaml file with conditional logic will quickly spiral out of control. This has been our experience with CircleCI which tries to do everything, which leads to very complex pipelines that are difficult to maintain.

We use codefresh extensively for CI/CD. Codefresh allows any number of pipelines per repo. This allows us to define a "prod" pipeline which can look materially different from an "integration testing" pipeline and keeps our manifests simple. It also allows us to configure those pipelines in different "Codefresh" accounts, so we can reduce the blast radius.

If we extend this principle to atlantis, then all we need to do is allow each instance of atlantis to look for a different pipeline. Of course, it can default it to /atlantis.yaml.

For example, I'd like to define atlantis/testing.yaml to run against our testing environment and atlantis/prod.yaml to run against our production environment.

Another benefit of this approach is using CODEOWNERS so that the the appropriate people signoff on changes to particular pipelines. e.g. developers can always update atlantis/testing.yaml, but changes to atlantis/prod.yaml will require signoff by a specific team or person.

Suggested interface

Introduce a new --repo-config=[config] parameter to the server command that can override the AtlantisYAMLFilename.

Then we can do the following for production...

atlantis server --allow-repo-config --repo-config=atlantis/prod.yaml

...and for testing

atlantis server --allow-repo-config --repo-config=atlantis/testing.yaml

[alebabai]

@osterman atlantis already had flag to path yaml-config, docs/server-configuration.html#yaml. Does it work for your case?

[osterman]

@alebabai, the server configuration could be used to define the repo config. The repo config defines how the pipeline works (like a travis.yml, or circle.yml). The server config defines how the server should operate. What we need is a server config option to override the default file name for the repo config file name.

[osterman]

For those interested, check this out! darrylb-github#2 =)

[mechastorm]

Another good reason why one would need multiple-instances of Atlantis

AWS China will not allow other IAM roles to assume it from outside of China.
Example - A non-China IAM Role is attempting to assume an IAM role in China
which I assume is not allowed.

[jekhokie]

Hello - is there any estimate of if/when this may be prioritized and worked on? Or if there is a solution/workaround to accommodate this "single repo, multiple atlantis servers targeting specific directories" functionality?

Thank you!

[lkysow]

It's not being worked on right now but if a community PR came in we would review it.
The only workaround right now is to run 2 atlantis servers, intercept the webhook with your own app and then forward the webhook to the right atlantis server.

[jekhokie]

@lkysow thanks for the update, we'll investigate the webhook interception for now and we'll keep an eye out for this!

[js-timbirkett]

When running 2 instances it would be nice to be able to override their names as a server config so that statuses would be updated like:

atlantis-dev/plan
atlantis-prd/plan
...

I see "atlantis" is hard-coded in: https://github.com/runatlantis/atlantis/blob/master/server/events/commit_status_updater.go

Currently plans and applies overwrite eachother's statuses.

UPDATE: I'll raise a PR for this and see if it's a desired feature.

[tsunamishaun]

Note that interception doesn't account for prs with changes across both accounts, results in servers having boltdb locks for runs they aren't responsible for. Also seeing that if using custom workflow names (by account name) that server configs need to have all workflows from the atlantis.yaml defined. Would be nice to have that atlantis.yaml filename changeable and the global check on workflow name removed. Any idea why #310 was scrapped @lkysow ? Would love some direction/advice if anyone is currently implementing.

[qingvincentyin]

This last few comments in this thread have reduced the problem to:

1. Run multiple installations/instances of Atlantis servers.
2. The Git repo maps one webhook to multiple Atlantis servers in (1). In order for the HTTP request to end up on the right Atlantis server, you need to install an HTTP proxy server and then write your own custom interception program/plug-in on that HTTP proxy server.

If my understanding above is correct, then why not reducing (2) to a much simpler design:

• Break down the all-in-one Git repo into multiple repos, one per account? i.e., instead of

├── README.md
├── accounts
│   ├── account1
│   │   ├── my-app
│   │   │   ├── main.tf
│   │   │   └── terraform.tfvars
│   │   ├── some-database
│   │   │   ├── main.tf
│   │   │   └── terraform.tfvars
│   │   └── terraform.tfvars
│   └── account2
│       ├── my-app
│       │   ├── main.tf
│       │   └── terraform.tfvars
│       ├── some-database
│       │   ├── main.tf
│       │   └── terraform.tfvars
│       └── terraform.tfvars
├── atlantis.yaml

Change that to:

Git repo 1:

  account1
    ├── README.md
    ├── atlantis.yaml
    ├── my-app
    │   ├── main.tf
    │   └── terraform.tfvars
    ├── some-database
    │   ├── main.tf
    │   └── terraform.tfvars
    └── terraform.tfvars

and Git repo 2:

  account2
    ├── README.md
    ├── atlantis.yaml
    ├── my-app
    │   ├── main.tf
    │   └── terraform.tfvars
    ├── some-database
    │   ├── main.tf
    │   └── terraform.tfvars
    └── terraform.tfvars

That way, you can have a straight-forward 1-to-1 mapping between a Git repo and an Atlantis installation/instance. No HTTP proxy needed.

[nitrocode]

In the upcoming v0.22.0 release, @krrrr38 has implemented #2805 to use a custom Atlantis comment name per each Atlantis instance and #2798 to customize the atlantis.yaml repo file location to use multiple files (without a hacky pre_workflow_hook).

See this page for more information

https://www.runatlantis.io/docs/server-side-repo-config.html#multiple-atlantis-servers-handle-the-same-repository

If you'd like to try it out now before the 0.22.0 release please try the pre release

https://github.com/runatlantis/atlantis/pkgs/container/atlantis/60556622?tag=v0.22.0-pre.20221226
https://github.com/runatlantis/atlantis/releases/tag/v0.22.0-pre.20221226

[Almenon]

Hi @mechastorm, do you have an example of how you configured Atlantis to assume the cross-account roles? I like your solution as less servers = less maintenance.