Expose parallelism semaphore in the cli ui as -parallelism=#

[knuckolls]

📣 This is a WIP. Feedback appreciated. 📣

Why
There are a couple of use cases I can came up with below but in general this semaphore was available within Terraform core but wasn't exposed up to the UI. I think fundamentally that if there's a configurable that it should be exposed to the end user.

Use Cases

Rate Limiting
For example, if you're hitting the AWS rate limit you will be able to limit the parallelism to perhaps avoid hitting the limit. There are other proposals for putting limits on individual providers but for now this will allow you to change from the default parallelism of 10 things at a time.

Safely Rolling Stateful Services
Consider a "clustered by default" stateful datastore that has a replication factor (i.e hdfs, cassandra, riak, elasticsearch). If you have a need to roll the whole cluster then create before destroy isn't enough. Whereas the nodes may have come up, the data may not be entirely replicated before the destroys begin. Doing a terraform apply -parallelism=1 gets us half of the way there by slowing down the process so that replication can occur properly.

The other part is a blocking script in a pre-destroy provisioner that checks to see if blocks have been sufficiently replicated before allowing the next destroy to occur. That will be my next PR and @phinze and I talked through the design as far as what can be done with the current internals.

TODO

• terraform apply -parallelism=n
• terraform destroy -parallelism=n
• terraform plan -parallelism=n
• terraform refresh -parallelism=n
• apply test written
• destroy test written
• plan test written
• refresh test written

[knuckolls]

Manually confirmed that destroy works as well.

[knuckolls]

UI code exposed to plan and refresh. Those are tricker to test by hand. I'll probably have to create a test resource that pings a localhost http endpoint that sleeps for a second and then time how long the plan and refresh take due to parallelism. I'll dig into the existing tests to see if there's any code I can lean on. Any pointers would be appreciated.

[knuckolls]

Tests written and passing for terraform apply -parallelism=1 and terraform apply -parallelism=2.

[knuckolls]

I duplicated the existing testCtxConfig function and added in the shell provisioner so I could determine from execution time whether or not the parallelism flag is functioning. Seems like I could DRY this up a bit, but this works for now.

[josephholsten]

@knuckolls how's this coming? looks like it needs a rebase, anything else needed to wrap this up?

[knuckolls]

@josephholsten Got pulled off to other work and never finished the tests as they weren't trivial to write at the time. A rebase is needed and any help finishing out the tests would be appreciated. I'm not likely going to be finishing up this PR in the short term.

All that said, this branch did function properly the last time I touched it and can likely be dropped in after a rebase to provide this functionality to the command line.

[josephholsten]

rebased at #3365

[phinze]

Closing in favor of #3365

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.