Multibranching strategy

[hossein-rasi]

Does atlantis support GitHub multi-branching strategy?
What we need is to use separated git branches which consists of different terraform configuration for different evironments (like dev,staging,production) and need to use atlantis to do terraform plan whenever a pull request opened to the specific branch. This means there is no common branch like master which carry all the configurations. I couldn't find any specific branching strategy for atlantis and would be great if you can let me know whether it is possible or not? Thanks!!

[Cellane]

I’d want to follow-up on the original idea with a similar issue, and I possible solution proposal (whose viability I didn’t have time to test 😭). I recently finished configuring CircleCI for a new project with a workflow that works something like this:

• when PR is opened against the develop branch, run terraform fmt -check, terraform validate and tflint, and once all three succeed, calculate Terraform plan against the development environment (using Terraform workspace, and different -var-file), save the plan for future applying, and notify the changes by commenting on the PR, as well as sending a message on Slack; on merge, apply the changes to the development environment only;
• similarly, when a PR is opened against the staging branch, perform all actions against the staging environment; master then affects the production environment; up to this point, I think my setup is somewhat similar to what @hossein-rasi is describing;
• the only difference, I suppose, is that each branch should really contain the same code, just in different versions at a time – as in, changes should “bubble” from develop through staging to production, and the only difference should be input variables and of course, state file.

After I spent two days fighting with CircleCI’s quirks, I never want to do that again 😅 (What kind of CI system allows you to set the project to only build on a commit which has a PR associated to it, but then doesn’t trigger a build once the PR is merged and closed? 🤯)

And that’s when I found Atlantis and realized that perhaps, I could’ve spent the past two days in a much better way. One thing that I’m not seeing in the documentation, however, is how I could achieve the “Git flow” setup that I described above… but what I see are a few pieces of the puzzle that when put together, they could perhaps help?

What I’m asking is… I’m sorry for the long introduction, but would the following setup have a chance to work for me or would I run into some other traps that it would be difficult to get out of?

1. I don’t think Atlantis workflows can be triggered based on the base branch (or did I miss that?). That leaves me with a workaround of using computed environmental variables and custom commands, like so:
2. In atlantis.yaml, define a single workflow for all three branches/environments.
3. The first step in the plan action would look something like this:

   - env:
       name: CUSTOM_WORKSPACE
       command: |
           if [ "${BASE_BRANCH_NAME}" == "master" ]; then
               echo "production"
           elif [ "${BASE_BRANCH_NAME}" == "staging" ]; then
               echo "staging"
           elif [ "${BASE_BRANCH_NAME}" == "develop" ]; then
               echo "development
           else
               # echo "Where are you merging into?"
               exit 1
           fi
   
   

4. The second step would then be init, the third step would switch the workspace to the value generated in the first step, and all of this would be followed by a rather simple adjustment of the standard plan step:

   - init
   - run: terraform workspace select $CUSTOM_WORKSPACE
   - plan:
       extra_args: ["-var-file", "${CUSTOM_WORKSPACE}.tfvars"]
   

5. Finally, similar adjustments would have to be done to the apply phase.

I’m sorry for asking for a consultation like this, but I’d be incredibly grateful to hear opinions/improvements/traps, and especially any warnings if I am proposing something impossible (perhaps variable interpolation in extra_args is not possible?)

[lkysow]

@hossein-rasi Atlantis actually doesn't care about the branches. It will operate on a PR opened to any branch and will actually read the atlantis.yaml from the pull request branch itself rather than the base branch. Have you tried it out? I think it should work for your use-case but if you have specific issues let me know.

@Cellane that approach looks good to me (caveat, I haven't tested it). FYI extra_args interpolation does work.

[hossein-rasi]

Thanks @Cellane for the detailed explanation. I will try out your approach and see whether it works or not. @lkysow Thanks for your response. The thing is each branch consists of different config for each env and so TF should at least know the base branch. This is mainly because of the reusable modules/resources and when we want to only plan one env and not the others even if we have changed in the other directories.

[Cellane]

So I just want to share my current progress. I’m not sure if I’m finished yet as I haven’t tested all possible scenarios, but I am at a stage where correct plans are being calculated. I think I encountered three gotchas:

Problem number one: various issues about escaping my multi-line env step’s `command.
Solution: remove all the lines 🤣

Problem number two: the default plan step switches Terraform workspace.
Solution: use another run step instead of the plan step, like so: - run: terraform plan -input=false -refresh -no-color -out $PLANFILE

Problem number three: Atlantis doesn’t update PATH to point to the project’s Terraform binary, so when you use the solution from the previous step, you will essentially disable Atlantis’ Terraform version switching.
Solution: I’m just going to share my entire config file at this point I think 😅

version: 3

projects:
  - dir: .
    terraform_version: v0.12.24
    workflow: custom

workflows:
  custom:
    plan:
      steps:
        - env: &define-custom-workspace
            name: CUSTOM_WORKSPACE
            command: 'if [ "${BASE_BRANCH_NAME}" == "master" ]; then echo "production"; elif [ "${BASE_BRANCH_NAME}" == "staging" ]; then echo "staging"; elif [ "${BASE_BRANCH_NAME}" == "develop" ]; then echo "development"; else exit 1; fi;'
        - env: &define-terraform-binary
            name: TERRAFORM_BINARY
            command: 'echo "/home/atlantis/.atlantis/bin/terraform${ATLANTIS_TERRAFORM_VERSION}"'
        - init
        - run: $TERRAFORM_BINARY workspace select $CUSTOM_WORKSPACE
        - run: $TERRAFORM_BINARY plan -input=false -refresh -no-color -out $PLANFILE

    apply:
      steps:
        - env: *define-custom-workspace
        - env: *define-terraform-binary
        - run: $TERRAFORM_BINARY workspace select $CUSTOM_WORKSPACE
        - run: $TERRAFORM_BINARY apply -no-color $PLANFILE

I haven’t yet properly tested the apply phase – from the documentation, I’m getting a hint that the workspace select step is not necessary, but I’m not sure about it…

And I kind of worry if I run into some problems regarding locks later on, because Atlantis for sure thinks that it’s working in the default workspace.

[dcatalano-figure]

I also have a similar question.

Where I work, if code is in develop branch it is in development environment. if code is in master/main it is in production environment.

We also use Terraform Workspaces, we define two; test & prod, and do not use default.

We'd like to use a custom workflow to do plan/apply against the correct workspace ONLY. Let me elaborate.

When develop is the base branch, the branch that's getting merged into, we want atlantis to plan/apply against the test terraform workspace. When master is the base branch, the branch that's getting merged into, we want atlantis to plan/apply against the prod terraform workspace.

One of the main motivations around using these different branch/workspace workflows is because a) we'd like to remove approval restrictions for plan/apply in development but keep them in production. The other reason is b) we do not want atlantis to plan for both workspaces/environments at the same time because, often, neither are ready at the same time IE. we iron out the kinks in development environment before it's ready for prod; a great usecase for terraform workspaces.

I understand atlantis does not care about which git branch but we do based on the workflow described above.

Worth noting, did come across the BASE_BRANCH_NAME environment variable. Just have not quite wrapped my brain around how to use it effectively. Any input would be appreciated. Cheers!

[dcatalano-figure]

based on what read in @Cellane post above is seems like I could do the following

if (wksp eq test) && (base branch eq develop)
  run plan
else if (wksp eq prod) && ((base branch eq master) || (base branch eq main))
  run plan
else if (wksp eq default)
  run plan
else
  exit

the last else if is because we do have some terraform runs that do not leverage workspaces

[dcatalano-figure]

So I just want to share my current progress. I’m not sure if I’m finished yet as I haven’t tested all possible scenarios, but I am at a stage where correct plans are being calculated. I think I encountered three gotchas:

Problem number one: various issues about escaping my multi-line env step’s `command.
Solution: remove all the lines 🤣

Problem number two: the default plan step switches Terraform workspace.
Solution: use another run step instead of the plan step, like so: - run: terraform plan -input=false -refresh -no-color -out $PLANFILE

Problem number three: Atlantis doesn’t update PATH to point to the project’s Terraform binary, so when you use the solution from the previous step, you will essentially disable Atlantis’ Terraform version switching.
Solution: I’m just going to share my entire config file at this point I think 😅

version: 3

projects:
  - dir: .
    terraform_version: v0.12.24
    workflow: custom

workflows:
  custom:
    plan:
      steps:
        - env: &define-custom-workspace
            name: CUSTOM_WORKSPACE
            command: 'if [ "${BASE_BRANCH_NAME}" == "master" ]; then echo "production"; elif [ "${BASE_BRANCH_NAME}" == "staging" ]; then echo "staging"; elif [ "${BASE_BRANCH_NAME}" == "develop" ]; then echo "development"; else exit 1; fi;'
        - env: &define-terraform-binary
            name: TERRAFORM_BINARY
            command: 'echo "/home/atlantis/.atlantis/bin/terraform${ATLANTIS_TERRAFORM_VERSION}"'
        - init
        - run: $TERRAFORM_BINARY workspace select $CUSTOM_WORKSPACE
        - run: $TERRAFORM_BINARY plan -input=false -refresh -no-color -out $PLANFILE

    apply:
      steps:
        - env: *define-custom-workspace
        - env: *define-terraform-binary
        - run: $TERRAFORM_BINARY workspace select $CUSTOM_WORKSPACE
        - run: $TERRAFORM_BINARY apply -no-color $PLANFILE

I haven’t yet properly tested the apply phase – from the documentation, I’m getting a hint that the workspace select step is not necessary, but I’m not sure about it…

And I kind of worry if I run into some problems regarding locks later on, because Atlantis for sure thinks that it’s working in the default workspace.

now I understand, why you did what you did 😆 ... thanks for the detailed example

[Matroxt]

Still not a perfect solution, but I found a way that is cleaner then the one proposed by @Cellane (In my opinion ofc)

First, I needed yq, so I updated my Atlantis helm chart values.yaml with this to add it with an initContainer:

extraVolumes:
- name: custom-tools
  emptyDir: {}
extraVolumeMounts:
- mountPath: /usr/local/bin/yq
  name: custom-tools
  subPath: yq
initContainers:
- name: download-tools
  image: alpine:3.14
  command: [sh, -c]
  args:
  - >-
    cd /tmp &&
    wget -O yq https://github.com/mikefarah/yq/releases/download/v4.14.2/yq_linux_amd64 &&
    chmod +x yq &&
    mv yq /custom-tools/
volumeMounts:
- mountPath: /custom-tools
  name: custom-tools

Then, I added a pre-workflow-hook to my server-side repos.yaml:

repos:
- id: /.*/
  branch: /.*/
  pre_workflow_hooks:
  - run: test -f ./generate-atlantis-yaml.sh && ./generate-atlantis-yaml.sh

This pre-workflow-hook checks if the generate-atlantis-yaml.sh script exists at the root of your Terraform Git repo, and if it does, it runs it.

Here's the content for mine, customize as you wish, you can now bind workspaces to branches
Make sure you've chmod +x generate-atlantis-yaml.sh before committing it in the repo, otherwise it will return exit 126.

# generate-atlantis-yaml.sh
#!/bin/bash

if [ "${BASE_BRANCH_NAME}" == "master" ] || [ "${BASE_BRANCH_NAME}" == "main" ]; then
    yq -i e '.projects[0].workspace = "prod"' atlantis.yaml
elif [ "${BASE_BRANCH_NAME}" == "dev" ]; then
    yq -i e '.projects[0].workspace = "dev"' atlantis.yaml
else 
    exit 1
fi

This makes it that my atlantis.yaml feels much cleaner.
The projects[0].workspace value is just a place holder and gets updated by the pre-workflow-hook

version: 3
projects:
  - dir: .
    terraform_version: v1.0.11
    workflow: custom
    workspace: to-be-updated-by-pre-workflow-hooks
workflows:
  custom:
    plan:
      steps:
        - init
        - plan:
            extra_args:
              - -var-file=${WORKSPACE}.tfvars
              - -input=false
              - -refresh
              - -no-color
              - -out=$PLANFILE
    apply:
      steps:
        - apply:
            extra_args:
              - -no-color
              - $PLANFILE

That way, we benefit from:

• Using the built-in plan/apply
• No need to specify the terraform binary in an env
• Same for the workspace
• The locking system will be handled properly

The better solution would be to be able to scope branches in the projects: part of atlantis.yaml but this is fine until then

[hasland]

I will just share a solution that worked for the environment that I'm testing:

• Same folder for development and production environments
• Different backends and .tfvars
• main branch used for production, development branch used for development

plan:
  steps:
    - env: 
         name: ENVIRONMENT
         command: 'if [ "${BASE_BRANCH_NAME}" == "main" ]; then echo "prd"; elif [ "${BASE_BRANCH_NAME}" == "development" ]; then echo "dev"; else exit 1; fi;'
    - init:
         extra_args:
           - -backend-config=${ENVIRONMENT}.azurerm.tfbackend
    - plan:
         extra_args:
           - -var-file=${ENVIRONMENT}.tfvars

Inside our project we have:

• dev.azurerm.tfbackend and prd.azurerm.tfbackend files (specifying the backend config without the access_key)
• dev.tfvars and prd.tfvars

Thanks to @Cellane and @Matroxt for sharing your codes, it helped me to achieve what I needed.