Error "exit status 126"

[fsanzdev]

Hi,

When running a plan through the GW pipeline, I keep getting this error:

[ecs-deploy-runner][2023-10-26T10:49:21+0000] No changes. Your infrastructure matches the configuration.
[ecs-deploy-runner][2023-10-26T10:49:21+0000] Terraform has compared your real infrastructure against your configuration
[ecs-deploy-runner][2023-10-26T10:49:21+0000] and found no differences, so no changes are needed.
[ecs-deploy-runner][2023-10-26T10:49:21+0000] Releasing state lock. This may take a few moments...
[ecs-deploy-runner][2023-10-26T10:49:22+0000] time=2023-10-26T10:49:22Z level=error msg=3 errors occurred:
[ecs-deploy-runner][2023-10-26T10:49:22+0000] 	* [/tmp/tmpgzf8_cmq/br-prod/sa-east-1/br-prod/security/acms] exit status 126
[ecs-deploy-runner][2023-10-26T10:49:22+0000] 	* [/tmp/tmpgzf8_cmq/br-prod/sa-east-1/br-prod/networking/alb-internal] exit status 126
[ecs-deploy-runner][2023-10-26T10:49:22+0000] 	* [/tmp/tmpgzf8_cmq/br-prod/sa-east-1/br-prod/data-stores/s3-buckets/jobs-redirection] exit status 126
[ecs-deploy-runner][2023-10-26T10:49:22+0000] [INFO] [infrastructure-deploy-script] 2023-10-26 10:49:22  "terragrunt plan-all" exited with code 126
[ecs-deploy-runner][2023-10-26T10:49:22+0000] Traceback (most recent call last):
[ecs-deploy-runner][2023-10-26T10:49:22+0000]   File "/opt/ecs-deploy-runner/scripts/infrastructure-deploy-script", line 242, in <module>
[ecs-deploy-runner][2023-10-26T10:49:22+0000]     main(max_content_width=120, auto_envvar_prefix=ENVVAR_PREFIX)
[ecs-deploy-runner][2023-10-26T10:49:22+0000]   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1134, in __call__
[ecs-deploy-runner][2023-10-26T10:49:22+0000]     return self.main(*args, **kwargs)
[ecs-deploy-runner][2023-10-26T10:49:22+0000]   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1059, in main
[ecs-deploy-runner][2023-10-26T10:49:22+0000]     rv = self.invoke(ctx)
[ecs-deploy-runner][2023-10-26T10:49:22+0000]   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1401, in invoke
[ecs-deploy-runner][2023-10-26T10:49:22+0000]     return ctx.invoke(self.callback, **ctx.params)
[ecs-deploy-runner][2023-10-26T10:49:22+0000]   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 767, in invoke
[ecs-deploy-runner][2023-10-26T10:49:22+0000]     return __callback(*args, **kwargs)
[ecs-deploy-runner][2023-10-26T10:49:22+0000]   File "/opt/ecs-deploy-runner/scripts/infrastructure-deploy-script", line 226, in main
[ecs-deploy-runner][2023-10-26T10:49:22+0000]     raise exc
[ecs-deploy-runner][2023-10-26T10:49:22+0000]   File "/opt/ecs-deploy-runner/scripts/infrastructure-deploy-script", line 207, in main
[ecs-deploy-runner][2023-10-26T10:49:22+0000]     deploy.deploy(
[ecs-deploy-runner][2023-10-26T10:49:22+0000]   File "/usr/local/lib/python3.8/site-packages/infrastructure_deploy_script/deploy.py", line [156](https://github.com/justosbr/infra-tf-live/actions/runs/6653018428/job/18078139966?pr=1623#step:7:157), in deploy
[ecs-deploy-runner][2023-10-26T10:49:22+0000]     __run_deployment(options, deploy_dir)
[ecs-deploy-runner][2023-10-26T10:49:22+0000]   File "/usr/local/lib/python3.8/site-packages/infrastructure_deploy_script/deploy.py", line 205, in __run_deployment
[ecs-deploy-runner][2023-10-26T10:49:22+0000]     raise DeployError(

If I run the same command the deployer is trying to run from my local desktop everything runs as expected with No Changes:

terragrunt plan-all --terragrunt-non-interactive -input=false --terragrunt-modules-that-include=../_envcommon/data-stores/s3-redirection.hcl --terragrunt-modules-that-include=../_envcommon/networking/alb-internal.hcl --terragrunt-modules-that-include=../_envcommon/networking/alb-public.hcl --terragrunt-modules-that-include=../_envcommon/networking/nlb-public.hcl --terragrunt-modules-that-include=../_envcommon/security/acm.hcl

The weird thing is that this error doesn't happen all the time in the same place, it happens when the pipeline is running for 5m and processing the staging folder, and next time it happens when the pipeline is running for prod environment and after 10m. What we see is that it happens a lot when we have long-time running pipelines.

We set the log for the deployer to debug but we can't see anything related to the infra, just that Python error message.

Any idea what could be happening? Thanks!

---

Tracked in ticket #110556

[MoonMoon1919]

Hi @fsanzdev, thanks for creating this.

We've most often seen this issue when the container runs out of memory. Is your ECS deploy runner using the default amount of memory (4GB)?

You can test this locally by building a docker image with terragrunt installed, launching a container using that image locally (constraining the number of CPUs and the amount of memory to the same number you use in the task), then running your same terragrunt command.

From the base dir of your repo, you can run something like this (assuming you use the default values for mem and cpu for the deploy runner), after building your image.

docker run -v $(pwd):/home -m 4G --cpus 2 -it <name of your image> sh

Before running the terragrunt command in the container, I recommend having a separate shell window so you can run docker stats <container id for your container> so you can see memory usage, as well as a separate shell window to exec into the container so you can run top while terragrunt is running. If you're coming close to running out of memory locally, it is very likely that is the issue in ECS.

We've found that doubling the memory (to 8GB, if you're using the default to values) was generally sufficient, but you will want to test on your own modules to confirm.

[denis256]

Hi,
exit status 126 may also mean that something is wrong with the permissions

gruntwork-io/terragrunt#922 (comment)

[fsanzdev]

We already had 8GB in the provisioner task and although the plan is long, it's split into several pieces for each environment which uses a new Fargate instance each time, so it's an 8GB instance for some changes in just 4 folders.

Regarding permissions we don't know how that could be the issue since it's being run in the pipeline.

[MoonMoon1919]

@fsanzdev I would still recommend measuring memory locally to rule out memory. Even if the changes are localized to a small number of folders, if the plan is long, it could be consuming more memory than the container has available.

[fsanzdev]

@MoonMoon1919 I'm reopening the ticket because we run the docker test container and the consumption was always very low. We also increased the Fargate instances memory to 16GB to rule out the memory issue but we are still getting this "exit 126" error when trying to process planes with "envcommon" changes involved.

[denis256]

Hi,
on the cluster, are running any security scanners like like Crowdstrike?
I had cases when ecs runner got killed by security apps

[fsanzdev]

No, just the pipeline with the Ref Arch settings. On another note, it looks like this is only happening when changes in envcommon are involved.

[timrwilliams]

Was this issue resolved? We are seeing the same issue, also intermittent 126 errors when a change to envcommon is made.