Terraform fails to delete default root cloudsql user

[madamkiwi]

Terraform Version

terraform 0.11.3
provider 1.6.0

Affected Resource(s)

Please list the resources as a list, for example:

• google_sql_database_instance

Terraform Configuration Files

https://gist.github.com/madamkiwi/e0aa06d6b4479cdaf8b1cbbcb4eeeb83#file-cloud-sql-tf

Debug Output

google cloud api logs: https://gist.github.com/madamkiwi/e0aa06d6b4479cdaf8b1cbbcb4eeeb83#file-google-cloud-api-log

terraform cli logs for run 1: https://gist.github.com/madamkiwi/e0aa06d6b4479cdaf8b1cbbcb4eeeb83#file-terraform-cli-error-1-log

terraform cli logs for run 2: https://gist.github.com/madamkiwi/e0aa06d6b4479cdaf8b1cbbcb4eeeb83#file-terraform-cli-error-2-log

Unfortunately we did not run this with debug log level turned on.

Panic Output

Expected Behavior

The DB instance stands up correctly without errors.

Actual Behavior

Two terraform applys were triggered at the same time to create brand new resources and as a result it failed to bring up the db instance on both. The errors are in the links above.

There may be a race condition between the two but we dont really think that possible given both invocations were creating totally different cloudsql instances.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. run two terraform applies with templates that have a cloud sql db
2. both will fail with an error

Important Factoids

none

References

none

cc @kkallday

[nat-henderson]

I'll try to repro this. Did you try this more than once, and did it happen every time? Did both terraform apply tasks use the same project?

[nat-henderson]

Hm, your example config won't load, because:

* random_id.db-name: resource repeated multiple times
* google_sql_database_instance.master: resource repeated multiple times

I'm going to continue trying to reproduce, but could you provide some more details about what steps reproduce this error?

[nat-henderson]

When I ran this config (modified so that terraform apply would run) I successfully got 2 instances with no trouble. The trouble you had might have been a fluke, but please reopen if you have a consistent repro case or more details about the error.

[kkallday]

The issue was with running two applys at the same time. I'll confirm this next week and touch base.

Thanks for your help!

[ikarlashov]

Confirming bug. Going to create 1 SQL instance and getting the same issue. Affected version: 0.11.3.
Version 0.10.8 works well.

[nat-henderson]

Okay, it sounds like there is a bug here, but I'm still not clear on how I can repro it. Does it happen to you every time with the config as you've got it, @klausitto? Can you share that config?

[ikarlashov]

@ndmckinley, yep.

resource "google_sql_database_instance" "master" {
  name = "master-instance"
  region = "europe-west1"

  settings {
    tier = "db-f1-micro"
  }
}

resource "google_sql_database" "eval" {
  name      = "eval-db"
  instance  = "${google_sql_database_instance.master.name}"
}

resource "google_sql_user" "eval" {
  name     = "db-user"
  instance = "${google_sql_database_instance.master.name}"
  host     = "cloudsqlproxy~%"
  password = "jDWDm6HptkpzxVps"
}

So, each time I ran this config - I got an error mentioned by OP.
I switched to 0.10.8 - issue has hone.

[nat-henderson]

All righty, I'll take a look. Thanks!

[nat-henderson]

Sorry! Still no repro with:

Terraform v0.11.1-dev
+ provider.google (unversioned)

(The provider's unversioned because it's built from master)

...
google_sql_database_instance.master: Still creating... (7m0s elapsed)          
google_sql_database_instance.master: Still creating... (7m10s elapsed)         
google_sql_database_instance.master: Creation complete after 7m17s (ID: master-instance)  
...
google_sql_user.eval: Still creating... (10s elapsed)
google_sql_database.eval: Still creating... (10s elapsed)
google_sql_user.eval: Creation complete after 15s (ID: master-instance/db-user)
google_sql_database.eval: Still creating... (20s elapsed)
google_sql_database.eval: Creation complete after 26s (ID: master-instance:eval-db)

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

I copied that config exactly as posted. You could try attaching debug logs, and maybe I can work from there?

[nat-henderson]

I suppose I can try to update to terraform 0.11.3, in case the bug is introduced in the two patch versions between yours and mine, but terraform does semver, so I certainly hope not. :)

[ikarlashov]

@ndmckinley
I'm building "deployer" using docker. So It's easy for me to use any terraform version I wish. You may try using docker image as well with the latest version. I suspect a bug in the latest version. Probably your version doesn't have this bug.

[nat-henderson]

That would certainly be interesting! I'm testing that theory now.

[nat-henderson]

Sorry! Still no repro.

google_sql_database.eval: Creation complete after 10s (ID: aaaaaa:eval-db)
google_sql_user.eval: Still creating... (10s elapsed)
google_sql_user.eval: Still creating... (20s elapsed)
google_sql_user.eval: Creation complete after 26s (ID: aaaaaa/db-user)

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Can you post debug logs? Maybe we can see what's going wrong from there.

[ikarlashov]

I don't know how... but I switched back to 0.11.3 and everything works fine. But I swear... I tried multiple times (not concurrently) to spin up cloudsql - and it failed. So my user wasn't created and even error in CloudSQL log appeared like "Unexpected error".

[nat-henderson]

I'm going to chalk it up to intermittent server-side errors and close the issue.

[kkallday]

Yes I think this is due to server-issues. My team has a CI pipeline that runs two terraform apply at the same time in the same project. Each one is creating a CloudSQL instances, a few logical DBs, and users. They are name-spaced so that they won't collide. We've received various errors. Here is another type of error (in addition to the original post above):

* google_sql_user.some-user-1: Error reading SQL User "username" in instance "cloud-sql-id-of-instance": googleapi: Error 503: Service temporarily unavailable., serverException

Just as FYI @madamkiwi and I are on the same team.

[nat-henderson]

I'm sorry to hear you're having these issues! I've asked the Cloud SQL team to look into it, but I am going to leave this issue closed as it's not an issue with Terraform.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!