Provider errors out on GCP SA key creation with "new value: Root resource was present, but now absent."

[skorobogatydmitry]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.0.11
on linux_amd64

Google provider:

provider "registry.terraform.io/hashicorp/google" {
  version     = "3.73.0"
  constraints = "3.73.0"
...

Affected Resource(s)

• google_service_account_key

Terraform Configuration Files

Module is simple:

resource "random_string" "sa_name" {
  length  = 23 #
  lower   = true
  number  = true
  special = false
  upper   = false
  lifecycle {
    ignore_changes = [
      special,
      override_special
    ]
  }
}

resource "google_service_account" "sa" {
  account_id   = random_string.sa_name.result
  display_name = "<NAME>"
}

resource "google_service_account_key" "key" {
  service_account_id = google_service_account.sa.id
}

Debug Output

There is no good way to dump it - issue occurs sporadically in automation.

Panic Output

google_service_account.sa: Creation complete after 0s [id=projects/XXX/serviceAccounts/YYY@XXX.iam.gserviceaccount.com]
google_service_account_key.key: Creating...
Error: Provider produced inconsistent result after apply
When applying changes to google_service_account_key.key, provider
"provider[\"registry.terraform.io/hashicorp/google\"]" produced an unexpected
new value: Root resource was present, but now absent.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Expected Behavior

is SA and key to be created successfully.

Actual Behavior

terraform apply errors out with error

Steps to Reproduce

It occurs sporadically in automation, so there it no good reproducer.
I checked GCP logs, doesn't report any errors accessing SA or its key. There are 2 operations: SA creation, Key creation.

Important Factoids

• SA and key creation happen almost one after another, without any notable delay. Maybe it causes GCP to return empty list at some point ?

References

• ticket Error: Provider produced inconsistent result after apply at GCP project + SA creation #6377 looks very similar, but provider version is way older.

[edwardmedia]

@skorobogatydmitry could you share the longer debug log which have more api requests and responses?

[skorobogatydmitry]

@edwardmedia , there are 8 occurrences of this issues for ~400 runs. So I don't have a reliable way to reproduce it with TF_LOG=trace. But I'll try to run it a couple times and post trace log here in case it gets reproduced.

[edwardmedia]

@skorobogatydmitry thanks. Waiting for the log.

[skorobogatydmitry]

I ran the module ~50 times and issue didn't reproduced. So it looks that I'm unable to provide logs.

[edwardmedia]

@skorobogatydmitry thank you for the efforts. I am closing the issue now. Feel free to reopen if the issue resurfaces.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.