fix(deploy): ensure build_sa iams are set before it can be used (#394)

* fix(deploy): ensure build_sa iams are set before it can be used Fixes possible race condition in cloud functions deployment where the function build started before the build_sa had all its IAMs bound. * fix: Updates following further investigation/testing --------- Co-authored-by: henrybell <henrybell@users.noreply.github.com>
cloudspannerecosystem · Oct 9, 2024 · aa48048 · aa48048
1 parent 0551694
commit aa48048
Show file tree

Hide file tree

Showing 16 changed files with 146 additions and 96 deletions.
diff --git a/terraform/cloud-functions/distributed/README.md b/terraform/cloud-functions/distributed/README.md
@@ -210,35 +210,6 @@ Autoscaler infrastructure, with the exception of Cloud Scheduler, lives.
  [known issue][provider-issue] in the Terraform Google provider, please retry
  with -parallelism=1.
 
- If you are using a recently created project, or one that does not have permissions
- automatically assigned to default service accounts, you may see an error message
- similar to the following:
-
- ```sh
- Error: Error waiting for Creating CloudFunctions Function: Error code 3, message: Build failed: failed to Fetch: failed to download archive gs://gcf-sources-[PROJECT_NUMBER]-us-central1/tf-poller-function-[UID]/version-1/function-source.zip:
- Access to bucket gcf-sources-[PROJECT_NUMBER]-us-central1 denied. You must grant Storage Object Viewer permission to [PROJECT_NUMBER]-compute@developer.gserviceaccount.com.
- ```
-
- A workaround for this issue is to run the following commands, which assign additional
- permissions to the default compute service account, which is used by Cloud Build:
-
- ```sh
- PROJECT_ID=$(gcloud config get-value project)
- PROJECT_NUMBER=$(gcloud projects list --filter="$(gcloud config get-value project)" --format="value(PROJECT_NUMBER)")
- gcloud projects add-iam-policy-binding "${PROJECT_ID}" --member="serviceAccount:${PROJECT_NUMBER}"-compute@developer.gserviceaccount.com --role='roles/storage.objectViewer'
- gcloud projects add-iam-policy-binding "${PROJECT_ID}" --member="serviceAccount:${PROJECT_NUMBER}"-compute@developer.gserviceaccount.com --role='roles/logging.logWriter'
- gcloud projects add-iam-policy-binding "${PROJECT_ID}" --member="serviceAccount:${PROJECT_NUMBER}"-compute@developer.gserviceaccount.com --role='roles/artifactregistry.writer'
- ```
-
- You can then re-run the `terraform apply` command listed above.
-
- Please note that the updates to the service account may take several
- minutes to propagate. The requirement for this workaround will be removed
- in an upcoming release.
-
- We recommend that you remove any unneeded permissions from the default compute
- service account when your use of the Spanner Autoscaler is complete.
-
 ## Preparing the Application Project
 
 In this section you prepare the deployment of the Cloud Scheduler, Forwarder
@@ -417,18 +388,6 @@ topic and function in the project where the Spanner instances live.
  [known issue][provider-issue] in the Terraform Google provider, please retry
  with -parallelism=1
 
- If you are using a recently created project, or one that does not have permissions
- automatically assigned to default service accounts, you may see an error message
- similar to the following:
-
- ```sh
- Error: Error waiting for Creating CloudFunctions Function: Error code 3, message: Build failed: failed to Fetch: failed to download archive gs://gcf-sources-[PROJECT_NUMBER]-us-central1/tf-poller-function-[UID]/version-1/function-source.zip:
- Access to bucket gcf-sources-[PROJECT_NUMBER]-us-central1 denied. You must grant Storage Object Viewer permission to [PROJECT_NUMBER]-compute@developer.gserviceaccount.com.
- ```
-
- If you see this issue, please follow the instructions in the section
- [Deploying the Autoscaler](#deploying-the-autoscaler) above.
-
 ### Authorize the Forwarder function to publish to the Poller topic
 
 1. Switch back to the Autoscaler project and ensure that Terraform variables

diff --git a/terraform/cloud-functions/distributed/app-project/.terraform.lock.hcl b/terraform/cloud-functions/distributed/app-project/.terraform.lock.hcl
diff --git a/terraform/cloud-functions/distributed/app-project/main.tf b/terraform/cloud-functions/distributed/app-project/main.tf
@@ -18,7 +18,7 @@ terraform {
  required_providers {
  google = {
  source = "hashicorp/google"
- version = ">= 6.3.0"
+ version = ">= 6.1.0"
  }
  }
 }

diff --git a/terraform/cloud-functions/distributed/autoscaler-project/.terraform.lock.hcl b/terraform/cloud-functions/distributed/autoscaler-project/.terraform.lock.hcl
diff --git a/terraform/cloud-functions/distributed/autoscaler-project/main.tf b/terraform/cloud-functions/distributed/autoscaler-project/main.tf
@@ -18,7 +18,7 @@ terraform {
  required_providers {
  google = {
  source = "hashicorp/google"
- version = ">= 6.3.0"
+ version = ">= 6.1.0"
  }
  }
 }

diff --git a/terraform/cloud-functions/per-project/.terraform.lock.hcl b/terraform/cloud-functions/per-project/.terraform.lock.hcl
diff --git a/terraform/cloud-functions/per-project/README.md b/terraform/cloud-functions/per-project/README.md
@@ -294,35 +294,6 @@ In this section you prepare your project for deployment.
  [known issue][provider-issue] in the Terraform Google provider, please retry
  the command above and include the flag `-parallelism=1`.
 
- If you are using a recently created project, or one that does not have permissions
- automatically assigned to default service accounts, you may see an error message
- similar to the following:
-
- ```sh
- Error: Error waiting for Creating CloudFunctions Function: Error code 3, message: Build failed: failed to Fetch: failed to download archive gs://gcf-sources-[PROJECT_NUMBER]-us-central1/tf-poller-function-[UID]/version-1/function-source.zip:
- Access to bucket gcf-sources-[PROJECT_NUMBER]-us-central1 denied. You must grant Storage Object Viewer permission to [PROJECT_NUMBER]-compute@developer.gserviceaccount.com.
- ```
-
- A workaround for this issue is to run the following commands, which assign additional
- permissions to the default compute service account, which is used by Cloud Build:
-
- ```sh
- PROJECT_ID=$(gcloud config get-value project)
- PROJECT_NUMBER=$(gcloud projects list --filter="$(gcloud config get-value project)" --format="value(PROJECT_NUMBER)")
- gcloud projects add-iam-policy-binding "${PROJECT_ID}" --member="serviceAccount:${PROJECT_NUMBER}"-compute@developer.gserviceaccount.com --role='roles/storage.objectViewer'
- gcloud projects add-iam-policy-binding "${PROJECT_ID}" --member="serviceAccount:${PROJECT_NUMBER}"-compute@developer.gserviceaccount.com --role='roles/logging.logWriter'
- gcloud projects add-iam-policy-binding "${PROJECT_ID}" --member="serviceAccount:${PROJECT_NUMBER}"-compute@developer.gserviceaccount.com --role='roles/artifactregistry.writer'
- ```
-
- You can then re-run the `terraform apply` command listed above.
-
- Please note that the updates to the service account may take several
- minutes to propagate. The requirement for this workaround will be removed
- in an upcoming release.
-
- We recommend that you remove any unneeded permissions from the default compute
- service account when your use of the Spanner Autoscaler is complete.
-
 ## Importing your Spanner instances
 
 If you have existing Spanner instances that you want to

diff --git a/terraform/cloud-functions/per-project/main.tf b/terraform/cloud-functions/per-project/main.tf
@@ -18,7 +18,7 @@ terraform {
  required_providers {
  google = {
  source = "hashicorp/google"
- version = ">= 6.3.0"
+ version = ">= 6.1.0"
  }
  }
 }

diff --git a/terraform/gke/decoupled/.terraform.lock.hcl b/terraform/gke/decoupled/.terraform.lock.hcl
diff --git a/terraform/gke/decoupled/main.tf b/terraform/gke/decoupled/main.tf
@@ -18,7 +18,7 @@ terraform {
  required_providers {
  google = {
  source = "hashicorp/google"
- version = ">= 6.3.0"
+ version = ">= 6.1.0"
  }
  kubernetes = {
  source = "hashicorp/kubernetes"

diff --git a/terraform/gke/unified/.terraform.lock.hcl b/terraform/gke/unified/.terraform.lock.hcl
diff --git a/terraform/gke/unified/main.tf b/terraform/gke/unified/main.tf
@@ -18,7 +18,7 @@ terraform {
  required_providers {
  google = {
  source = "hashicorp/google"
- version = ">= 6.3.0"
+ version = ">= 6.1.0"
  }
  kubernetes = {
  source = "hashicorp/kubernetes"

diff --git a/terraform/modules/autoscaler-base/main.tf b/terraform/modules/autoscaler-base/main.tf
@@ -68,9 +68,18 @@ resource "google_service_account" "build_sa" {
  display_name = "Autoscaler - Cloud Build Builder Service Account"
 }
 
-resource "google_project_iam_binding" "build_iam" {
- for_each = toset(["roles/storage.objectViewer", "roles/logging.logWriter", "roles/artifactregistry.writer"])
- project = var.project_id
- role = each.value
- members = ["serviceAccount:${google_service_account.build_sa.email}"]
+resource "google_project_iam_member" "build_iam" {
+ for_each = toset([
+ "roles/artifactregistry.writer",
+ "roles/logging.logWriter",
+ "roles/storage.objectViewer",
+ ])
+ project = var.project_id
+ role = each.value
+ member = "serviceAccount:${google_service_account.build_sa.email}"
+}
+
+resource "time_sleep" "wait_for_iam" {
+ depends_on = [google_project_iam_member.build_iam]
+ create_duration = "90s"
 }
diff --git a/terraform/modules/autoscaler-base/outputs.tf b/terraform/modules/autoscaler-base/outputs.tf
@@ -17,10 +17,11 @@
 output "build_sa_id" {
  value = google_service_account.build_sa.id
  description = "Service account ID for Builder SA"
- depends_on = [google_project_iam_binding.build_iam]
+ depends_on = [time_sleep.wait_for_iam]
 }
+
 output "build_sa_email" {
  value = google_service_account.build_sa.email
  description = "Service account email for Builder SA"
- depends_on = [google_project_iam_binding.build_iam]
+ depends_on = [time_sleep.wait_for_iam]
 }
diff --git a/terraform/modules/forwarder/main.tf b/terraform/modules/forwarder/main.tf
@@ -22,13 +22,21 @@ resource "google_service_account" "build_sa" {
  display_name = "Autoscaler - Cloud Build Builder Service Account"
 }
 
-resource "google_project_iam_binding" "build_iam" {
- for_each = toset(["roles/storage.objectViewer", "roles/logging.logWriter", "roles/artifactregistry.writer"])
- project = var.project_id
- role = each.value
- members = ["serviceAccount:${google_service_account.build_sa.email}"]
+resource "google_project_iam_member" "build_iam" {
+ for_each = toset([
+ "roles/artifactregistry.writer",
+ "roles/logging.logWriter",
+ "roles/storage.objectViewer",
+ ])
+ project = var.project_id
+ role = each.value
+ member = "serviceAccount:${google_service_account.build_sa.email}"
 }
 
+resource "time_sleep" "wait_for_iam" {
+ depends_on = [google_project_iam_member.build_iam]
+ create_duration = "90s"
+}
 
 resource "google_service_account" "forwarder_sa" {
  account_id = "forwarder-sa"
@@ -98,7 +106,7 @@ resource "google_cloudfunctions_function" "forwarder_function" {
  service_account_email = google_service_account.forwarder_sa.email
  build_service_account = google_service_account.build_sa.id
  depends_on = [
- google_project_iam_binding.build_iam,
+ time_sleep.wait_for_iam,
  google_pubsub_topic_iam_member.forwader_pubsub_sub_binding
  ]
 }