google_storage_transfer_job timeout while waiting for state to become 'success'

[AFTownsend]

4.47

Affected Resource(s)

google_storage_transfer_job

Terraform Configuration Files

resource "google_storage_transfer_job" "nightly_backup" {
  description = "${google_storage_bucket.bucket.name} backup"
  project     = var.project

  transfer_spec {
    gcs_data_source {
      bucket_name = google_storage_bucket.bucket.name
      path        = ""
    }
    gcs_data_sink {
      bucket_name = google_storage_bucket.backup_bucket.name
      path        = ""
    }
  }

  schedule {
    schedule_start_date {
      year  = 1979
      month = 1
      day   = 1
    }
  }

  depends_on = [google_storage_bucket_iam_member.bucket_admin_role, google_storage_bucket_iam_member.backup_bucket_admin_role]
}

Debug Output

Never used gist before. I ran terraform apply and debug and got many of these errors

"error": {
"code": 429,
"message": "Quota exceeded for quota metric 'Create requests' and limit 'Create requests per day' of service 'storagetransfer.googleapis.com' for consumer 'project_number:XXXXXXX'.",
"errors": [
{
"message": "Quota exceeded for quota metric 'Create requests' and limit 'Create requests per day' of service 'storagetransfer.googleapis.com' for consumer 'project_number:XXXXXX'.",
"domain": "global",
"reason": "rateLimitExceeded"
}
],

I checked my quotas and it said that resource hadn't even been used today.

Panic Output

Expected Behavior

The resource should be created

Actual Behavior

The resource is not created and times out after 1 minutes with this error

╷
│ Error: timeout while waiting for state to become 'success' (timeout: 1m0s)
│ 
│   with module.pipeline.module.pipeline_root_bucket_and_backup.google_storage_transfer_job.nightly_backup,
│   on ../modules/bucket_and_backup/main.tf line 67, in resource "google_storage_transfer_job" "nightly_backup":
│   67: resource "google_storage_transfer_job" "nightly_backup" {

It has tested in two different projects and the same behaviour has been observed.
I have created the resources manually in the console and it was successful.

Steps to Reproduce

1. Create resource that uses the transfer job
2. terraform apply
3. It will timeout

[edwardmedia]

@AFTownsend can you share the debug log that have all the api requests & responses? For the error of 409, Quota exceeded seems clear. I guess you need to take a close look.

[AFTownsend]

@AFTownsend can you share the debug log that have all the api requests & responses?

Hi @edwardmedia, yes sure. I will have to check with my security officer first as I don't know if anything confidential can be leaked.

[AFTownsend]

Hi @edwardmedia, also I did check our quotas and we have not exceeded them and we actually have not used any of the quota of that resource today.

[edwardmedia]

@AFTownsend does the error showing up when you apply this single resource?

Keep in mind, this error came from api. Likely there is not much we can do at the provider level.

[AFTownsend]

@edwardmedia I tried today and it has worked. I realise this looks like I have just waited a day and my quota has been reset but I swear I checked yesterday and the quota had not been used at all 😆 You'll just have to believe me.

Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.