feat: add project to hooks

[Fabianoshz]

what

• Run pre and post workflow runs on non-default workspaces

why

• Take 'projects' into consideration on the pre/post workflow hooks as defined in this comment.
• This is a follow up of this discussion on slack.
• Try to fix the hooks behavior when using the new cloning to allow a more efficient locking mechanism as described on fix: add path to WorkingDir methods #2180.

references

• Closes Run plan/apply for multiple projects in parallel #260
• Relates to Plan Error: dir "dir-name-here" does not exist when using pre_workflow_hooks #2239
• Relates to fix: add path to WorkingDir methods #2180

[Fabianoshz]

Lot's of stuff broken here, still need to fix merges and small fixes. Focused more on learning that making it work.

What I'm trying to do:

• Run the hooks on the default workspace (as this always exist)
• Run the hooks on every necessary project

What I've found so far:

• It's fairly easy to look for projects being changed on the PR, so we can run the hooks on every changed project in the PR;
• Workspace don't seem to be a problem at all as I thought in the beginning;
• If someone is using something like https://github.com/transcend-io/terragrunt-atlantis-config this approach might increase execution time by a lot;

[nitrocode]

Thanks again for proposing another change!

Quite a lot of users use https://github.com/transcend-io/terragrunt-atlantis-config so as much as I'd like this change, we'd need to gate it behind a flag unless we can understand why the execution time cannot be lowered.

[Fabianoshz]

Hey @nitrocode, I believe we can reduce the time if we:

• Run the atlantis.yaml generator on the first hook
• Copy the generated atlantis.yaml from the first run on subsequent hooks

Either the users must run a script to check if the file already exists or we need differentiate the initial hook from the others.

[nitrocode]

The pre_workflow_hooks is not only used to generate an atlantis.yaml file. We also now have the ability to use custom atlantis.yaml files.

• What if the file is not generated but static? Will it still copy the file over?
• What if there are multiple atlantis.yaml files that are generated? Will it copy all of those over to the new directories?
• What about other generated files in the pre_workflow_hooks? How would you account for that?

I still do not quite understand why different directories would need to be used if you're solving for running pre_workflow_hooks for multiple workspaces.

Each workspace could use its own TF_DATA_DIR=.terraform-{workspace} and then re-use the same code directory without having to copy and paste any pre-generated files from one workflow to the next.

To me the root issue is that we create a new git clone and directory per workspace which is inefficient.

Consider this directory structure, single clone, single atlantis.yaml file

terraform/.terraform-default/  # generated by TF_DATA_DIR=.terraform-default terraform init
terraform/.terraform-ue1-prod/ # generated by TF_DATA_DIR=.terraform-ue1-prod terraform init
terraform/.terraform-ue2-prod/ # generated by TF_DATA_DIR=.terraform-ue2-prod terraform init
terraform/variables.tf
terraform/main.tf
atlantis.yaml                  # generated by a single pre-workflow-run

I think to make this process more efficient

• Use a single clone for each project
• Use a different TF_DATA_DIR for each workspace
• Create 2 types of pre-workflow-runs, a global one (existing), and one per workflow (new functionality)
  • Users can override the pre-workflow-hooks using allow_custom_workflows: true

# atlantis server's repos.yaml
repos:
  - id: /.*/
    allow_custom_workflows: true
    # existing functionality for global workflow hook
    pre_workflow_hooks:
      - run: |
          echo "parallel_plan: true" > atlantis.yaml

workflows:
  default:
    # potential new functionality could override above or append?
    pre_workflow_hooks:
      - run: |
          echo "parallel_plan: true\nparallel_apply: true" > atlantis.yaml
  
  custom_no_parallel_plans:
    # potential new functionality could override above or append?
    pre_workflow_hooks:
      - run: |
          echo "parallel_plan: false" > atlantis.yaml

# repo's atlantis.yaml
projects:

- name: terraform-ue1-prod
  branch: /main/
  dir: terraform
  workspace: ue1-prod
  workflow: default

- name: terraform-ue2-prod
  branch: /main/
  dir: terraform
  workspace: ue2-prod
  workflow: custom_no_parallel_plans

- name: terraform-ue1-dev
  branch: /main/
  dir: terraform
  workspace: ue1-dev
  workflow: default

workflows:
  default:
    # potential new functionality could override above or append?
    pre_workflow_hooks:
      - run: |
        echo "hello world"

[Fabianoshz]

• What if the file is not generated but static? Will it still copy the file over?

That's up to the user. I'm not saying that Atlantis itself should copy the files, I'm saying that we as users should copy the files to reduce the load if something that is CPU intensive is used, I should have stated that more clearly.

• What if there are multiple atlantis.yaml files that are generated? Will it copy all of those over to the new directories?
• What about other generated files in the pre_workflow_hooks? How would you account for that?

Both also should be up to the user too.

I still do not quite understand why different directories would need to be used if you're solving for running pre_workflow_hooks for multiple workspaces.

They don't, as I said, workspace don't seem to be a problem at all as I thought in the beginning, they are just part of the path where the hooks/plan runs. the project holds the workspace information. The problem is in the projects directory as stated here. Right now from what I can see each project checkout the entire repository for each plan:

# This is the default project (which is always initialized)
# Hooks will run here
'/home/fabiano/.atlantis/repos/Fabianoshz/atlantis-test/13/default/FY======': 
atlantis.yaml  README.md  standard  standard-2  standard-3  terragrunt.hcl

# This is project 'standard'
# Hooks will never run here
'/home/fabiano/.atlantis/repos/Fabianoshz/atlantis-test/13/default/ON2GC3TEMFZGI===':
atlantis.yaml  README.md  standard  standard-2  standard-3  terragrunt.hcl

# This is project 'standard-2'
# Hooks will never run here
/home/fabiano/.atlantis/repos/Fabianoshz/atlantis-test/13/default/ON2GC3TEMFZGILJS:
atlantis.yaml  README.md  standard  standard-2  standard-3  terragrunt.hcl

What I'm trying to do is ensure that the hooks run on each project (and consequently in the workspace of each project), right now it will only run on the default project, and that's why the issue #2239 was raised.

We definitely can make this process more efficient and with less requests to VCS, but that's another problem, that requires far more changes, in far more places.

[nitrocode]

The ticket comment you're linking to points to the PR #2180 (and reverted #2253) which moved to using base32 directories per plan instead of the normal directory which caused an issue of generating the atlantis.yaml file across workspaces. I understand it delves into the issue a bit more too.

I believe the root issue here seems to be the re-cloning of the directories. If we can reduce that to a single clone per PR, then the feature you're trying to implement will be much easier to maintain over time.

cc: @jamengual thoughts on this?

[jamengual]

@Fabianoshz and I had a chat in slack about this and I think he is aware than reusing the clone and copy is an option.

there can only be one Atlantis.yaml per workspace but not per project so copying the Atlantis.yaml from the default workspace to project workspace dir should work.

[Fabianoshz]

Ok, took a look at this and found out why we have one directory for each project, the reason is because the plan file is saved inside the directory of the project, so we can have a scenario where the same resource can be planned in two or more workspaces, which would lead to conflict if we just use the same directory but with a different workspace.

I've moved the Clone to RunAutoplanCommand and RunCommentCommand on this branch (compare here) and I believe I can change Clone to make a clean clone without any workspace in a safe directory and after that we can copy this clean workspace as needed. (The APIController would need some work though). Added copy from clean workspace.

Another thing we could do is change the plan file name to have a prefix with the workspace name, this could help us avoid conflicts, I'm not sure if this would be feasible because I didn't had enough time to dig the plan/apply code to see how it works.

[Fabianoshz]

Opened a PR trying to reduce the amount of clones here, if we go forward with that I can adjust this one later to use CreateWorkspaceDirectory instead of cloning.

[jamengual]

@Fabianoshz do you think you will have time to finish this?

@GenPage have been working on this to document the lock process #3345

[jamengual]

I will close this since updates have been stale for a long time.
If anyone is interested in working on this, comment on this PR or contact us in slack.