google_project resource exits early leading to "Error: googleapi: Error 403: The project to be billed is associated with an absent billing account., accountDisabled"

[ghost]

This issue was originally opened by @c-dingwell as hashicorp/terraform#24079. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Terraform Version

Terraform v0.12.20

• provider.google v3.8.0
• provider.random v2.2.1

Affected Resource(s)

google_project

Terraform Configuration Files

locals {
    project_id = "cuyler-demo-project-${random_id.id.dec}"
}

provider "google" {
    project = local.project_id
}

resource "random_id" "id" {
  byte_length = 3
}

resource "google_project" "demo_project" {
    name = "demo project"
    project_id = local.project_id
    org_id = var.organization
    billing_account = var.billing_account
    # provisioner "local-exec" {
    #     command = "sleep 10"
    # }
}

resource "google_storage_bucket" "demo_bucket" {
    name     = google_project.demo_project.project_id
    force_destroy = true
    depends_on = [google_project.demo_project]
}

Expected Behavior

Project creation and assigning of the billing account should be fully done before further resources proceed to be created.

Actual Behavior

Encounter sporadically over last few days. Just ran in a loop ~40% of the time it works just fine. The remaining ~60% I received an error:

Error: googleapi: Error 403: The project to be billed is associated with an absent billing account., accountDisabled
on main.tf line 25, in resource "google_storage_bucket" "demo_bucket":
25: resource "google_storage_bucket" "demo_bucket" {

Steps to Reproduce

terraform apply

Important Factoids

Uncommenting the local exec and forcing a 10 second sleep causes this to work all the time. This isn't an elegant solution.

[venkykuberan]

@c-dingwell I tried few times and i didn't get the error message. I am able to create project every time in around 9 secs. I used the following config just to ensure i have the billing id from the data_source before i create the project. Could you please try this config and let me know how it goes ?

data "google_billing_account" "acct" {
  display_name = "xxx_xxx_billing"
  open         = true
}

resource "google_project" "demo_project" {
    name = "testing project"
    project_id = "testing-project-v3"
    org_id = "xxxxxx"
    billing_account = data.google_billing_account.acct.id
    # provisioner "local-exec" {
    #     command = "sleep 10"
    # }
}

If you get the same error even with the above config please attach the debug logs (export TF_LOG=DEBUG) to get a better understanding on what's going on ?

[c-dingwell]

I am unable to reproduce it using the "google_billing_account" data structure. If I change it from that to a variable that is passed ("billing_account = var.billing_account" in the google_project resource) then I can get a regular reoccurrence of the issue.

[venkykuberan]

Its an eventual consistency issue. An explicit delay is required to handle this issue which you already have in place.

Its a timing issue between 2 different APIs. Terraform/provider gets the confirmation from project API that billing account is enabled. But the update isn't propagated to storage API while creating the bucket resource thus it throws an error.

I am tagging @emilymye @rileykarson to evaluate any retry mechanism is needed for storage api calls.

[rileykarson]

@emilymye: You're most familiar with eventual consistency problems- how do you think I should solve this?

I'm leaning towards a retry predicate that checks for the specific The project to be billed is associated with an absent billing account on a 403, but also considered the lazy approach of building in a 10s sleep in project.

[emilymye]

yeah, i don't really see a good way to use PollAsync for this, since it's happening on resources that are children of the project. There might be another way we can read a project/resourcemanager-specific state that means it's actually ready, but this wouldn't hurt for now

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!