Pass ephemeral variables to terraform apply

[DanielMSchmidt]

This PR improves how we can pass values to ephemeral variables in the apply command, now CLI flags, environment variables, and vars files are supported. We will deal with interactive inputs in a separate PR to keep this one smaller.

Jira: TF-21848

Target Release

1.10.x

Draft CHANGELOG entry

ENHANCEMENTS

• Allow apply-time ephemeral variables

[jbardin]

Just a couple things inline, but overall this looks good.

[radeksimko]

LGTM

[github-actions]

Reminder for the merging maintainer: if this is a user-visible change, please update the changelog on the appropriate release branch.

[roman-zilist]

This change broke a lot of our pipelines with the following error:
The variable tags cannot be set using the -var and -var-file options when │ applying a saved plan file, because a saved plan includes the variable │ values that were set when it was created. The saved plan specifies ................................................................................................................................... │ as the value whereas during apply the value object with 9 attributes was │ set by an environment variable. To declare an ephemeral variable which is │ not saved in the plan file, use ephemeral = true.

It would be great if new TF versions were backward compatible!

[jbardin]

@roman-zilist, the regression was a bug which was fixed by #36121. Are you still seeing this with the current 1.10.1 release?

[roman-zilist]

@jbardin Yes, we're still experiencing the same issue. I assume we're using the latest version because of
required_version = ">= 1.7.5"

[jbardin]

@roman-zilist, if you can confirm that you are using v1.10.1 please file a new issue. #36106 was closed with the v1.10.1 release, and comments on an old PR are not going to be triaged.

Thanks!

[roman-zilist]

@jbardin We've decided to rollback to the most recent stable version 1.9.8 and pin all our terraform templates to use it. This is neither the first, nor the second time we get burned by dealing with newest terraform/terraform provider versions which introduce bugs that break our pipelines and derail prod deployments. We use Terraform to build and deploy many of our products, and we're extremely disappointed with reliability of Hashicorp Terraform platform

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.