Symlinks in Terragrunt

[ArturChe]

Hi!

I have found that Terragrunt does not work with symlinks as expected. Or expected but just in a different way from Terraform.
For example, lets assume that we have such folders structure:

/aaa/bbb/ccc/cloudfront/development
/aaa/bbb/ccc/cloudfront/development/stage
/aaa/bbb/ccc/modules/cloudfront
~/cloudfront_development -> /aaa/bbb/ccc/cloudfront/development

Now, when I am trying to run ~/cloudfront_development$ terragrunt run-all plan it fails with an error:

ERRO[0000] Could not find any subfolders with Terragrunt configuration files
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

But commands ~/cloudfront_development$ terragrunt plan and ~/cloudfront_development/stage$ terragrunt plan separatelly works fine.

When I have navigated to /aaa/bbb/ccc/cloudfront/development directly and then run $ terragrunt run-all plan then it worked.

Also, I was trying to get rid of Terraform modules declaration and to declare scripts sources in the Terragrunt configuration terraform block:
replace modules.tf

module "stage" {
  source                          = "../../../modules/cloudfront"
}

with terragrunt.hcl

terraform {
  source = "../../..//modules/cloudfront"
}

But ~/cloudfront_development/stage$ terragrunt plan has also exited with error:

ERRO[0004] Hit multiple errors:
stat /home/modules: no such file or directory
ERRO[0004] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

As you can see, the Terraform is recognizing the source path correctly even when running its commands from the symlink location.

So is it an expected behavior with symlinks in Terragrunt?

[brikis98]

Ah, my guess is that the run-all code that scans the current directory for Terragrunt sub-folders does not take symlinks into account. A PR to fix that is welcome. Note that if we do support symlinks, we'll want to watch out for infinite cycles.

[mickael-ange]

Hi.

I'm new with Terraform and Terragrunt. I started playing with it for few days only.
I got similar issue with symlinks.

My first project directory structure was as follows.

# terraform$ tree
.
├── envs
│   ├── ec2-dev-v3
│   │   ├── security-groups
│   │   │   └── terragrunt.hcl 
│   │   ├── vars.yaml
│   │   └── vpc
│   │       └── terragrunt.hcl
│   ├── outposts-poc
│   │   ├── security-groups
│   │   │   └── terragrunt.hcl
│   │   ├── vars.yaml
│   │   └── vpc
│   │       └── terragrunt.hcl
│   ├── staging-v3
│   │   ├── security-groups
│   │   │   └── terragrunt.hcl
│   │   ├── vars.yaml
│   │   └── vpc
│   │       └── terragrunt.hcl
│   └── terragrunt.hcl
└── modules
    ├── security-groups
    │   ├── main.tf
    │   └── variables.tf
    └── vpc
        ├── main.tf
        ├── outputs.tf
        └── variables.tf

It works just fine but I had duplicated terragrunt.hcl in all my environments.
The modules are configured using vars.yaml in each environment which is loaded automatically by the root terragrunt.hcl

# cat envs/ec2-dev-v3/vars.yaml
---
# EC2-DEV V3

# Remote state config
terraform_remote_state_backend_bucket: ec2-dev-v2-devops-ap-northeast-1

env_prefix: ec2-dev-v3
aws_profile: sandbox
aws_region: ap-northeast-1
# AWS VPC Config
aws_vpc_cidr: 172.19.0.0/16
aws_vpc_azs:
  - ap-northeast-1a
  - ap-northeast-1c
  - ap-northeast-1d
# aws_vpc_public_subnets:
#   - 172.19.0.0/24
#   - 172.19.1.0/24
#   - 172.19.2.0/24
# aws_vpc_private_subnets:
#   - 172.19.16.0/20
#   - 172.19.32.0/20
#   - 172.19.48.0/20
aws_vpc_database_subnets:
  - 172.19.128.0/24
  - 172.19.129.0/24
  - 172.19.130.0/24

# cat envs/terragrunt.hcl
locals {
  # This allows to configure Terragrunt (remote_state, provider, etc.) dynamically
  # for each environment using 'vars.yaml' file located in each environment folder.
  vars = yamldecode(file(find_in_parent_folders("vars.yaml")))
}

# This populates the modules variables. Same values as 'locals'.
inputs = local.vars

# Configure backend for each module, we create a separated state file per environment and per module.
remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    # We manage all the state file of an environment in the same bucket.
    bucket         = local.vars.terraform_remote_state_backend_bucket
    # The state file is different for each environment and module in order to
    # segment the infrastructure to limit risk.
    # e.g. terraform/ec2-dev-v3/infra/vpc/terraform.tfstate
    key            = "terraform/${path_relative_to_include()}/terraform.tfstate"
    profile        = local.vars.aws_profile
    region         = local.vars.aws_region
    encrypt        = true
    # We manage all the state file locks of an environment in the same DynamoDB table.
    # The DynamoDB is created automatically if it does not exist.
    dynamodb_table = "${local.vars.env_prefix}-tfstates"
  }
}

# This generates the module's provider dynamically.
generate "provider" {
  path = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents = <<EOF
provider "aws" {
  profile = "${local.vars.aws_profile}"
  region  = "${local.vars.aws_region}"
}
EOF
}

What I wanted was to remove the duplicated terragrunt.hcl in all my environments.

So I created envs/shared/infra directory with its Terragrunt config for each modules. Then I created symlinks in all my environments, thus all environments can shared the same module Terragrunt config.

The project directory structure became:

# terraform$ tree
.
├── envs
│   ├── ec2-dev-v3
│   │   ├── infra -> ../shared/infra
│   │   └── vars.yaml
│   ├── outposts-poc
│   │   ├── infra -> ../shared/infra
│   │   └── vars.yaml
│   ├── shared
│   │   └── infra
│   │       ├── security-groups
│   │       │   └── terragrunt.hcl
│   │       └── vpc
│   │           └── terragrunt.hcl
│   ├── staging-v3
│   │   ├── infra -> ../shared/infra
│   │   └── vars.yaml
│   └── terragrunt.hcl
└── modules
    ├── security-groups
    │   ├── main.tf
    │   └── variables.tf
    └── vpc
        ├── main.tf
        ├── outputs.tf
        └── variables.tf

# cat envs/shared/infra/vpc/terragrunt.hcl 

include "root" {
  path = find_in_parent_folders()
}

terraform {
  source = "../../../../modules//vpc"
}

# cat envs/shared/infra/security-groups/terragrunt.hcl 
include "root" {
  path = find_in_parent_folders()
}

terraform {
  source = "../../../../modules//security-groups"
}

dependency "vpc" {
  config_path = "../vpc"
}

inputs = {
  vpc_id = dependency.vpc.outputs.vpc_id
}

dependency "vpc" {
  config_path = "../vpc"

  mock_outputs = {
    vpc_id = "(known after apply)"
  }

  mock_outputs_allowed_terraform_commands = ["validate", "init", "plan"]
}

It was too simple, when I ran terragrunt run-all validate, no terragrunt.hcl files were found because the underlying Golang library uses filepath.Walk() to scan the configuration files and this function does not traverse symlinks.

ec2-dev-v3$ terragrunt run-all validate
ERRO[0000] Could not find any subfolders with Terragrunt configuration files 
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

I thought I was toasted but eventually I found another library (facebookgo/symwalk) which supports Walk with symbolic links.

I decided to patch Terragrunt with the library.

diff --git a/config/config.go b/config/config.go
index 37dc9fa..bf10ea6 100644
--- a/config/config.go
+++ b/config/config.go
@@ -23,6 +23,8 @@ import (
        "github.com/gruntwork-io/terragrunt/options"
        "github.com/gruntwork-io/terragrunt/remote"
        "github.com/gruntwork-io/terragrunt/util"
+       
+       "github.com/facebookgo/symwalk"
 )
 
 const DefaultTerragruntConfigPath = "terragrunt.hcl"
@@ -517,7 +519,7 @@ func GetDefaultConfigPath(workingDir string) string {
 func FindConfigFilesInPath(rootPath string, terragruntOptions *options.TerragruntOptions) ([]string, error) {
        configFiles := []string{}
 
-       err := filepath.Walk(rootPath, func(path string, info os.FileInfo, err error) error {
+       err := symwalk.Walk(rootPath, func(path string, info os.FileInfo, err error) error {
                if err != nil {
                        return err
                }

Then I ran go build and copy the built terragrunt binary into my $PATH (e.g. ~/bin).

Et Voila!

Finally I can run terragrunt run-all validate for my entire environment using my project directory with symbolic links.

ec2-dev-v3$ terragrunt run-all validate
INFO[0000] The stack at /home/mickael/git/emq-devops/terraform/envs/ec2-dev-v3 will be processed in the following order for command validate:
Group 1
- Module /home/mickael/git/emq-devops/terraform/envs/ec2-dev-v3/infra/vpc

Group 2
- Module /home/mickael/git/emq-devops/terraform/envs/ec2-dev-v3/infra/security-groups
 
Success! The configuration is valid.

Success! The configuration is valid.

The only issue with that is that symwalk library is not stopping if an infinite loop is created with the symlinks.

I'm wondering whether it could be possible to patch Terragrunt as shown above and add a flag or a global Terragrunt config to allow following symlinks when running terragrunt run-all <command>?

For the time being I'm gonna use my Terragrunt patch/fork to go further in my exploring project with Terraform and Terragrunt.

I hope this message helped someone.

Regards
Mickael

Terragrunt: v0.36.3
Terraform: v1.1.7

[skaravad]

Hello all,

Any priority on this , I followed instructions provided by @mickael-ange and it really helped to address the issue, however with symlinks, the overall resource creation slowed down , and I think this is more of a .terragrunt-cache directory issue when traversing in the links.

The way I approached the config:
env.yaml (in a directory dev25)

env: "dev"
aws_region: "us-west-2"
aws_domain: "amazonaws.com"
env_type: dev
env_id: dev25

env.hcl

locals {
  env_vars = yamldecode(file("./env.yaml"))
  common_vars = yamldecode(file(find_in_parent_folders("common_env_vars.yaml")))
}

inputs = {
  region = local.env_vars.aws_region
  aws_region = local.env_vars.aws_region
  env = local.env_vars.env
  env_type = "dev"
  aws_account_id = get_aws_account_id()
  aws_domain = try(local.env_vars.aws_domain, "amazonaws.com")
  env_id = local.env_vars.env_id
  layer_runtime = "python3.8"
}
terraform {
    extra_arguments "plugin_dir" {
        commands = [
            "init",
            "plan",
            "apply",
            "destroy",
            "output"
        ]

        env_vars = {
            TF_PLUGIN_CACHE_DIR = "/tmp/plugins",
        }
    }
}
remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config = {
    bucket         = "${local.common_vars.terraform_s3_backend}"
    key            = "${local.env_vars.env_id}/${path_relative_to_include()}/terraform.tfstate"
    region         = "${local.common_vars.terraform_s3_backend_region}"
    encrypt        = true
    dynamodb_table = "${local.env_vars.env_id}-terraform-lock"
  }
}

And the wrapper script

export TERRAGRUNT_DOWNLOAD="/tmp/.`uuidgen |cut -d '-' -f1`"
echo "cache-dir: $TERRAGRUNT_DOWNLOAD"
terragrunt run-all apply -auto-approve --terragrunt-non-interactive --terragrunt-exclude-dir ../<module_name>

echo "cleaning up $TERRAGRUNT_DOWNLOAD"
rm -rf $TERRAGRUNT_DOWNLOAD

This way I just symlinked the terragrunt part , and it works without any issues,

[aslafy-z]

Here's a library that implements a walk that follow symbolic links but avoid visiting directories more than once.
https://github.com/edwardrf/symwalk
Any chance this pattern gets integrated into terragrunt?

[aslafy-z]

Please have a look to #3101

[denis256]

Fix included in https://github.com/gruntwork-io/terragrunt/releases/tag/v0.68.17

[pawcykca]

I still have problem with symlinks like below:

$ terragrunt --version
terragrunt version v0.71.1
$ ls -l
total 0
lrwxrwxrwx 1 user user 46 Dec 31 09:42 my_project -> /home/user/projects/my_project
$ cd my_project
$ terragrunt run-all validate
10:07:29.661 INFO   Terragrunt Cache server is listening on 127.0.0.1:41723
10:07:29.661 INFO   Start Terragrunt Cache server
10:07:29.663 INFO   Shutting down Terragrunt Cache server...
10:07:29.663 INFO   Terragrunt Cache server stopped
10:07:29.663 ERROR  could not find any subfolders with Terragrunt configuration files
10:07:29.664 ERROR  Unable to determine underlying exit code, so Terragrunt will exit with error code 1

When I go directly into /home/user/projects/my_project terragrunt run-all ... works correctly.
Should this be also fixed by #3562?

[mr-tko]

I can confirm symlinks are not working.

terragrunt --version
terragrunt version v0.71.1

ls -la child-folder-with-symlink
08:28 terragrunt.hcl -> ../../terragrunt.hcl

And finally running from a parent folder:

terragrunt run-all plan
08:30:27.953 ERROR  could not find any subfolders with Terragrunt configuration files
08:30:27.953 ERROR  Unable to determine underlying exit code, so Terragrunt will exit with error code 1

[yhakbar]

Note that symlinks have been downgraded to an experiment due to regressions caused by supporting it:
https://terragrunt.gruntwork.io/docs/reference/experiments/#symlinks

Make sure you're enabling the experiment if you'd like to use it.

[pawcykca]

Yes, it works with below export:

export TERRAGRUNT_EXPERIMENT=symlinks

Thanks for information.