Switching a policy for individual members causes a race condition

[Stono]

Hola!
We were migrating away from google_project_iam_policy in favour of a bunch of google_project_iam_member resources.

The terraform apply tried to simultaneously delete the existing policy whilst creating the members, resulting in:

* google_project_iam_policy.artefacts_projects: Error applying IAM policy to project: Error applying IAM policy for project "at-artefacts". Policy is &cloudresourcemanager.Policy{AuditConfigs:[]*cloudresourcemanager.AuditConfig(nil), Bindings:[]*cloudresourcemanager.Binding(nil), Etag:"BwVuCwTaqFA=", Version:1, ServerResponse:googleapi.ServerResponse{HTTPStatusCode:200, Header:http.Header{"Server":[]string{"ESF"}, "Cache-Control":[]string{"private"}, "X-Frame-Options":[]string{"SAMEORIGIN"}, "X-Content-Type-Options":[]string{"nosniff"}, "Vary":[]string{"Origin", "X-Origin", "Referer"}, "Date":[]string{"Thu, 07 Jun 2018 17:18:30 GMT"}, "X-Xss-Protection":[]string{"1; mode=block"}, "Alt-Svc":[]string{"quic=\":443\"; ma=2592000; v=\"43,42,41,39,35\""}, "Content-Type":[]string{"application/json; charset=UTF-8"}}}, ForceSendFields:[]string(nil), NullFields:[]string(nil)}, error is googleapi: Error 409: There were concurrent policy changes. Please retry the whole read-modify-write with exponential backoff., aborted

Might be worth a lock across policy, and member resources?

[rosbo]

We already have a lock... The mutex key is iam-project-[project-id]. There should only be one IAM call happening simultaneously for a given project:
https://github.com/terraform-providers/terraform-provider-google/blob/master/google/iam_project.go#L68

It looks like the bug was caused by an etag mismatch. We have code handling 409 (Conflict) and it should retry. However, it seems from the error message that the error was wrapped twice. The retry logic probably failed to retry because the double wraping hid the fact that the underlying error was a 409.

[danawillow]

Ah, we have a mutex for the _member and _binding resources but not for _policy. @Stono, do you have a config you could share that reproduces the error? That would help me test a fix.

[Stono]

ahh @danawillow I don't now i'm afraid, but long story short we had a google_project_iam_policy.

and in one commit, we removed google_project_iam_policy and introduced a bunch of google_project_iam_member, so one plan was both simultaneously deleting the project policy and trying to set members on it. Should be easy enough to reproduce

[danawillow]

Cool, got it. Just sent out #1645.

Note though that although this fixes the error in question, there will still be a potential race condition left depending on which resource acquires the lock first. If the iam_policy resource does, then we're all good- it'll restore the policy to what it was before the resource was ever created, and then the iam_member will apply to recreate the policy. However, if the iam_member resource acquires the lock first, then it'll apply the policy that already exists, and then the iam_policy resource will remove it. Re-running terraform would fix it in that case so it's not the end of the world, but just be aware of that.

[stevemsmith]

I'm in the same situation. Tried to build the provider based on #1645 and it's not working. It's entirely possible I don't know what I'm doing, but here's what I did...

1. Followed the instructions to build.
2. Copied provider to ~/.terraform.d/plugins/darwin_amd64
3. Ran "init"
4. Ran "plan"
5. Ran "apply"

And I got the same error:

• google_project_iam_policy.project-policy: Error applying IAM policy to project: Error applying IAM policy for project .......

Ran it twice, just to be sure. Any help greatly appreciated.

[danawillow]

@biff2005 can you run TF_LOG=DEBUG terraform apply and share the output as a gist?

[stevemsmith]

Here you go...
https://gist.github.com/biff2005/511e00e23cdd9ead820c9cbabaa55a5a

Thanks for the assistance!

[danawillow]

Ah, ok. That's a different error message- this one is a 400 invalid_argument. Is that the entire debug output? The error message Error applying IAM policy for project only occurs in our codebase after a setIamPolicy call, but I don't see one in the logs.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!