Assured workloads do not work as documented

[pgoldtho]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.13.7

• provider registry.terraform.io/hashicorp/google v3.76.0
• provider registry.terraform.io/hashicorp/google-beta v3.76.0

Affected Resource(s)

• google_assured_workloads_workload

Terraform Configuration Files

/***********************************
 Variables
************************************/
variable "org_id" {
	description = "The orgainization ID."
	type = string
}

variable "billing_account" {
  description = "The ID of the billing account to associate with assured workloads"
  type        = string
}

variable "location" {
	description = "The region in which to store any resources deployed under the Assured Workloads environment"
	type = string
    default = "us-central1"
}

variable "compliance_regime" {
	description = "Location and personnel control level"
	type = string
    default = "FEDRAMP_MODERATE"
}

variable "kms_key_rotation_period" {
    description = "Immutable. The period when the Key Management Service automatically rotates a key."
    type = string
    default = "864000s"
}

variable "aw_projects" {
    description = "GCP project IDs in a map to allow multiple assured workloads to be created in a folder"
    type = map(string)
    default = {
        "devops"   = "aw-dev-devops"
        "network"  = "aw-dev-network"
    }
}


/***********************************
 Main body
************************************/
locals {
    # Set the next key rotation time to current time + rotation period
    kms_rotation_time = timeadd(timestamp(), var.kms_key_rotation_period)
}

# Top level folder in the org identified by org_id
resource "google_folder" "parent-folder" {
    provider = google
    display_name = "Government Projects"
    parent       =  "organizations/${var.org_id}"
}

# Provision an assured workload for every project in the aw_projects map
resource "google_assured_workloads_workload" "workload" {
    provider = google-beta

    for_each = var.aw_projects

    display_name      = each.value
    billing_account   = "billingAccounts/${var.billing_account}"
    compliance_regime = var.compliance_regime
    organization      = var.org_id
    location          = var.location
    kms_settings {
       next_rotation_time = local.kms_rotation_time
       rotation_period    = var.kms_key_rotation_period
    }
    provisioned_resources_parent = google_folder.parent-folder.name
    resource_settings {
       resource_id   = each.value
       resource_type = "CONSUMER_PROJECT"
    }
}

/***********************************
 Outputs
************************************/
output "folder" {
  value = google_folder.parent-folder
}

output "workloads" {
    value = tomap(google_assured_workloads_workload.workload)
}

Debug Output

https://gist.github.com/pgoldtho/ea6e73d04ea5c2ce7a308eb48ff1a94c

Panic Output

n/a

Expected Behavior

The following structure should be created:

Government projects (folder)
  ├── aw-dev-devops (assured workload project)
  └── aw-dev-network (assured workload project)

Actual Behavior

Error: Error creating Workload: googleapi: Error 400: Request contains an invalid argument.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "assuredworkloads.googleapis.com",
    "reason": "ERROR_CODE_AW_PROJECTS_NOT_SUPPORTED"
  }
]

Was able to workaround the 400 error by commenting out the resource settings:

    resource_settings {
       resource_id   = each.value
       resource_type = "CONSUMER_PROJECT"
    }

Which produced the following structure:

Government projects (folder)
  ├── aw-dev-devops (folder)
   |       └── cmek-aw-dev-devops757943b8-d4a (project)
  └── aw-dev-network (folder)
           └── cmek-aw-dev-networked4f107a-29 (project)

terraform output for these look like this:

"resources" = [
  {
    "resource_id" = 801313193678
    "resource_type" = "ENCRYPTION_KEYS_PROJECT"
  },
  {
    "resource_id" = 1034520693847
    "resource_type" = "CONSUMER_FOLDER"
  },
]

Note:

• Two "encryption key projects" have been created to implement kms key rotation
• The assured workload was created as a folder
• No new Assured Workloads are visible on the Compliance screen in the GCP Console.
• Terraform destroy failed with

Error: Error deleting Workload: failed to delete Workload: googleapi: Error 400: Workload 00-a417bf57-483a-44b7-825f-e37 contains projects or other resources that are not deleted.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "assuredworkloads.googleapis.com",
    "reason": "ERROR_CODE_WORKLOAD_NOT_EMPTY"
  }
]

I was able to workaround some of these issues by:

1. Manually deleting the resources.
2. Changing the provider from google-beta to google
3. Commenting out the kms_settings

This provisioned the desired structure:

Government projects (folder)
  ├── aw-dev-devops (assured workload project)
  └── aw-dev-network (assured workload project)

But:

• The Project ID was system generated example: aw-4c467535-1853-4145-b015-dec
• It does not appear in the terraform output. The Project Number is returned instead. This is an issue because we cannot pass the number to terraform modules that require a Project ID as input.

Steps to Reproduce

1. terraform apply

See above

Important Factoids

us-central1 was pre-registered as an assured workload location for this org_id using the Assured Folder Creation form

Attempting to create a workload in a different location fails with:

Error: Error creating Workload: googleapi: Error 400: Assured Workloads request validation failed: The specified endpoint cannot serve requests for the location 'us-east1'
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "assuredworkloads.googleapis.com",
    "reason": "ERROR_CODE_INVALID_ENDPOINT"
  }
]

References

#8673

[edwardmedia]

@pgoldtho are you able to create the desired structure using different method? If yes, how and show the result.

[pgoldtho]

This is what I ended up using:

variable "org_id" {
	description = "The orgainization ID."
	type = string
}

variable "billing_account" {
  description = "The ID of the billing account to associate with assured workloads"
  type        = string
}

variable "location" {
	description = "The region in which to store any resources deployed under the Assured Workloads environment"
	type = string
    default = "us-central1"
}

variable "compliance_regime" {
	description = "Location and personnel control level"
	type = string
    default = "FEDRAMP_MODERATE"
}

variable "aw_projects" {
    description = "GCP project IDs in a map to allow multiple assured workloads to be created in a folder"
    type = map(string)
    default = {
        "devops"   = "aw-dev-devops"
        "network"  = "aw-dev-network"
    }
}

resource "google_folder" "parent-folder" {
    provider = google
    display_name = "Government Projects"
    parent       =  "organizations/${var.org_id}"
}

resource "google_assured_workloads_workload" "workload" {
    provider = google

    for_each = var.aw_projects

    display_name      = each.value
    billing_account   = "billingAccounts/${var.billing_account}"
    compliance_regime = var.compliance_regime
    organization      = var.org_id
    location          = var.location
    provisioned_resources_parent = google_folder.parent-folder.name
}

output "folder" {
  value = google_folder.parent-folder
}

output "workloads" {
    value = tomap(google_assured_workloads_workload.workload)
}

This created

Government projects (folder)
  ├── aw-dev-devops (assured workload project)
  └── aw-dev-network (assured workload project)

Which is what I wanted. The only issue was the project id was not available in the outputs.

[pgoldtho]

Based on the documentation. I would expect the Project IDs to be available as resource_id's in workloads.devops.resource_settings and workloads.network.resource_settings but these were populated with internal project numbers instead of IDs.

I suspect the ERROR_CODE_AW_PROJECTS_NOT_SUPPORTED error is related to this. The documentation shows the desired Project ID being passed as input. The API appears to be expecting an internal project number.

[pgoldtho]

The other issue was the way kms rotation periods were handled. I was not expecting a dedicated "ENCRYPTION_KEYS_PROJECT" for each assured workload but I'm still learning about assured workloads so could be wrong about that.

[edwardmedia]

@slevenick what do you see this issue?

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.