Error "text file busy" : Random conflict on parallel stack execution which use the same provider

[caparisi]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

cdktf & Language Versions

cdktf = 0.15.5
Language = Python 3.10.10

Affected Resource(s)

Any stack resource.

Debug Output

Context : We have 2 stack (testtac-elastic and testtac-redis) which will both use the HELM provider to install some chart, and both will be executed in parallel with the same cdktf deploy testtac-elastic testtac-redis. We have randomly this kind of issue which happen (1 time on 100).

testtac-elastic  Initializing the backend...
testtac-redis  Initializing the backend...
testtac-redis  
               Successfully configured the backend "azurerm"! Terraform will automatically
               use this backend unless the backend configuration changes.
testtac-redis  Initializing provider plugins...
               - Finding hashicorp/helm versions matching "2.9.0"...
testtac-redis  - Finding hashicorp/kubernetes versions matching "2.19.0"...
testtac-redis  - Installing hashicorp/helm v2.9.0...
testtac-redis  - Installed hashicorp/helm v2.9.0 (signed by HashiCorp)
testtac-redis  - Installing hashicorp/kubernetes v2.19.0...
testtac-redis  - Installed hashicorp/kubernetes v2.19.0 (signed by HashiCorp)

               Terraform has created a lock file .terraform.lock.hcl to record the provider
               selections it made above. Include this file in your version control repository
               so that Terraform can guarantee to make the same selections by default when
               you run "terraform init" in the future.
testtac-redis  Terraform has been successfully initialized!
               
               You may now begin working with Terraform. Try running "terraform plan" to see
               any changes that are required for your infrastructure. All Terraform commands
               should now work.

               If you ever set or change modules or backend configuration for Terraform,
               rerun this command to reinitialize your working directory. If you forget, other
               commands will detect it and remind you to do so if necessary.
testtac-redis  - Fetching hashicorp/kubernetes 2.19.0 for linux_amd64...
testtac-redis  - Retrieved hashicorp/kubernetes 2.19.0 for linux_amd64 (signed by HashiCorp)
testtac-redis  - Fetching hashicorp/helm 2.9.0 for linux_amd64...
testtac-redis  - Retrieved hashicorp/helm 2.9.0 for linux_amd64 (signed by HashiCorp)
               - Obtained hashicorp/helm checksums for linux_amd64; All checksums for this platform were already tracked in the lock file
               - Obtained hashicorp/kubernetes checksums for linux_amd64; All checksums for this platform were already tracked in the lock file
testtac-redis  Success! Terraform has validated the lock file and found no need for changes.
testtac-elastic  
                 Successfully configured the backend "azurerm"! Terraform will automatically
                 use this backend unless the backend configuration changes.
testtac-elastic  Initializing provider plugins...
                 - Finding hashicorp/helm versions matching "2.9.0"...
testtac-elastic  - Finding hashicorp/kubernetes versions matching "2.19.0"...
testtac-elastic  - Finding hashicorp/random versions matching "3.4.3"...
testtac-elastic  - Installing hashicorp/helm v2.9.0...
testtac-redis  ╷
               │ Error: Failed to load plugin schemas
               │ 
               │ Error while loading schemas for plugin components: Failed to obtain provider
               │ schema: Could not load the schema for provider
               │ registry.terraform.io/hashicorp/helm: failed to instantiate provider
               │ "registry.terraform.io/hashicorp/helm" to obtain schema: fork/exec
               │ .terraform/providers/registry.terraform.io/hashicorp/helm/2.9.0/linux_amd64/terraform-provider-helm_v2.9.0_x5:
               │ text file busy..
               ╵
/usr/lib/node_modules/cdktf-cli/node_modules/yoga-layout-prebuilt/yoga-layout/build/Release/nbind.js:53
        throw ex;
        ^

[UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "Invoking Terraform CLI failed with exit code 1".] {
  code: 'ERR_UNHANDLED_REJECTION'
}

Expected Behavior

No fail during the execution, cdktf should prevent the situation where 2 stack completely independent in their behaviour but using the same provider impact each other.

Actual Behavior

Random fail 1% of the time when 1 stack do it init step and the other one try to use it provider. We give this example but this happen in any other configuration where we have many stack execution in parallel which use the same provider

Steps to Reproduce

Create 2 stack which use the same provider and run them in same time with cdktf deploy stack1 stack2

References

This look like to be a common issue with terraform parallel execution:

• https://dev.to/mxglt/how-to-improve-terragrunt-pipeline--gk0
• Race condition when executing terraform binary uber/astro#17

[labmonkey42]

Similar issue except that in my case I'm developing in vscode and when vscode-jest runs (via watch) my unit tests, if I so much as think about running cdktf or terraform the plan and validate tests fail and I get errors about the docker or aws provider "text file busy".

It seems like multiple CDKTF stack concurrency has race conditions with Terraform 1.4 due to the Dependency Lock file.

[froblesmartin]

Happening the same to me in GitHub Actions using https://github.com/marketplace/actions/terraform-cdk-action with multiple stacks deployment at once. When running it locally I do not face any issues 🤔.

I think the issue is the same as hashicorp/terraform#31964

[ZacharyATanenbaum]

This issues seems to scale with the number of stacks being deployed at once.

For context I'm deploying using Arch Linux locally and if I deploy to 3 stacks I have a low probability of hitting this error (<5% after ten attempts). As I deploy to more stacks at once (7 and 9 stacks) I tend to have a greater than 90% chance of hitting this error.

Versions (All combinations tested):
Terraform v1.4.5 on linux_amd64
CDKTF: 0.15.4 / 0.15.5 / 0.16.0
cdktf_cdktf_provider_aws: 12.0.12 / 12.0.13 / 13.0.0

[timello]

Hi there, any workaround?

[ZacharyATanenbaum]

Sadly, the work-around is deploying to the least amount of stacks possible. I've started to deploy to four stack at a time and using the --ignore-missing-stack-dependencies flag which has been tedious but sufficient.

[ansgarm]

Hi 👋

Alternatively the --parallelism 1 flag should do the trick, too.

[caparisi]

Thank for your fix @DanielMSchmidt ❤️

[github-actions]

I'm going to lock this issue because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If you've found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.