[ISSUE] The account_id attribute is sometimes not populated when updating databricks_mws_workspaces

[tonybaron]

Hi there,

It appears that the account_id attribute is sometimes not being populated when PATCHing databricks_mws_workspaces. The API returns MALFORMED_REQUEST: Invalid AccountId: because the account_id is an empty string on the patch request.

In my case this was related to adding customer managed key configurations.

Performing the below request manually with the REST API, but populating the account_id correctly resulted in the API returning a 200.

Terraform Version

0.13.6

Provider Version

0.3.4

Affected Resource(s)

• databricks_mws_workspaces

Debug Output

databricks_mws_workspaces.this: Modifying... [id=<redacted>/<redacted>]
...
PATCH /accounts/<redacted>/workspaces/<redacted> {
  "account_id": "",
  "aws_region": "<redacted>",
  "credentials_id": "<redacted>",
  "deployment_name": "",
  "managed_services_customer_managed_key_id": "<redacted>",
  "network_id": "<redacted>",
  "storage_configuration_id": "<redacted>",
  "storage_customer_managed_key_id": "<redacted>",
  "workspace_name": ""
}: timestamp=2021-05-13T15:03:36.845-0400
...
2021-05-13T15:03:37.568-0400 [DEBUG] plugin.terraform-provider-databricks_v0.3.4: 400 Bad Request: timestamp=2021-05-13T15:03:37.568-0400
2021-05-13T15:03:37.568-0400 [WARN]  plugin.terraform-provider-databricks_v0.3.4: /api/2.0/accounts/<redacted>/workspaces/<redacted>:400 - MALFORMED_REQUEST: Invalid AccountId:: timestamp=2021-05-13T15:03:37.568-0400
2021-05-13T15:03:37.568-0400 [WARN]  plugin.terraform-provider-databricks_v0.3.4: /api/2.0/accounts/<redacted>/workspaces/<redacted>:400 - MALFORMED_REQUEST: Invalid AccountId:: timestamp=2021-05-13T15:03:37.568-0400
...
Error: MALFORMED_REQUEST: Invalid AccountId:

Expected Behavior

The workspace should be updated with new attributes.

Actual Behavior

The API returns a 400 and fails to update AND the terraform state thinks that things are successfully applied. Subsequent requests do not attempt to modify this resource.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. Modify an attribute of databricks_mws_workspaces. In my case it was customer managed keys related, but I'm not sure if it is specific to that.
2. terraform apply

[nfx]

@tonybaron please provide more context, including cmk & workspace resource configs. and cmk request traffic. it might be the case that some resources didn't return proper content and this happened.

[tonybaron]

Hi @nfx. Sorry for the delay.

CMK resource config:

resource "databricks_mws_customer_managed_keys" "this" {
  provider   = databricks.mws
  account_id = var.databricks_account_id
  aws_key_info {
    key_arn   = aws_kms_key.control_plane_key.arn
    key_alias = aws_kms_alias.control_plane_key_alias.name
  }
  use_cases = ["MANAGED_SERVICES", "STORAGE"]
}

Workspace resource config:

resource "databricks_mws_workspaces" "this" {
  provider        = databricks.mws
  account_id      = var.databricks_account_id
  aws_region      = var.region
  workspace_name  = var.workspace_name
  deployment_name = var.workspace_name

  credentials_id           = databricks_mws_credentials.this.credentials_id
  storage_configuration_id = databricks_mws_storage_configurations.this.storage_configuration_id
  network_id               = databricks_mws_networks.this.network_id
  storage_customer_managed_key_id          = databricks_mws_customer_managed_keys.this.customer_managed_key_id
}

And some more logs:

2021-05-17T12:29:16.858-0400 [DEBUG] plugin.terraform-provider-databricks_v0.3.4: POST /accounts/<redacted>/customer-managed-keys {
  "account_id": "<redacted>",
  "aws_key_info": {
    "key_alias": "<redacted>",
    "key_arn": "<redacted>"
  },
  "use_cases": [
    "MANAGED_SERVICES",
    "STORAGE"
  ]
}: timestamp=2021-05-17T12:29:16.858-0400
2021-05-17T12:29:19.252-0400 [DEBUG] plugin.terraform-provider-databricks_v0.3.4: 201 Created {
  "account_id": "<redacted>",
  "aws_key_info": {
    "key_alias": "<redacted>",
    "key_arn": "<redacted>",
    "key_region": "us-east-1",
    "reuse_key_for_cluster_volumes": true
  },
  "creation_time": 1621268959313,
  "customer_managed_key_id": "<redacted>",
  "updated_time": 1621268959313,
  "use_cases": [
    "MANAGED_SERVICES",
    "STORAGE"
  ]
} <- POST /accounts/<redacted>/customer-managed-keys: timestamp=2021-05-17T12:29:19.252-0400
2021-05-17T12:29:19.252-0400 [DEBUG] plugin.terraform-provider-databricks_v0.3.4: GET /accounts/<redacted>/customer-managed-keys/<redacted>: timestamp=2021-05-17T12:29:19.252-0400
2021-05-17T12:29:19.451-0400 [DEBUG] plugin.terraform-provider-databricks_v0.3.4: 200 OK {
  "account_id": "<redacted>",
  "aws_key_info": {
    "key_alias": "<redacted>",
    "key_arn": "<redacted>",
    "key_region": "us-east-1",
    "reuse_key_for_cluster_volumes": true
  },
  "creation_time": 1621268959313,
  "customer_managed_key_id": "<redacted>",
  "updated_time": 1621268959313,
  "use_cases": [
    "MANAGED_SERVICES",
    "STORAGE"
  ]
} <- GET /accounts/<redacted>/customer-managed-keys/<redacted>: timestamp=2021-05-17T12:29:19.451-0400
...
databricks_mws_customer_managed_keys.this: Creation complete after 2s [id=<redacted>/<redacted>]
...
...
2021-05-17T12:43:54.852-0400 [DEBUG] plugin.terraform-provider-databricks_v0.3.4: PATCH /accounts/<redacted>/workspaces/<redacted> {
  "account_id": "",
  "aws_region": "us-east-1",
  "credentials_id": "<redacted>",
  "deployment_name": "",
  "network_id": "<redacted>",
  "storage_configuration_id": "<redacted>",
  "storage_customer_managed_key_id": "<redacted>",
  "workspace_name": ""
}: timestamp=2021-05-17T12:43:54.852-0400
...
2021-05-17T12:43:55.544-0400 [DEBUG] plugin.terraform-provider-databricks_v0.3.4: 400 Bad Request: timestamp=2021-05-17T12:43:55.543-0400
2021-05-17T12:43:55.544-0400 [WARN]  plugin.terraform-provider-databricks_v0.3.4: /api/2.0/accounts/<redacted>/workspaces/<redacted>:400 - MALFORMED_REQUEST: Invalid AccountId:: timestamp=2021-05-17T12:43:55.544-0400
2021-05-17T12:43:55.544-0400 [WARN]  plugin.terraform-provider-databricks_v0.3.4: /api/2.0/accounts/<redacted>/workspaces/<redacted>:400 - MALFORMED_REQUEST: Invalid AccountId:: timestamp=2021-05-17T12:43:55.544-0400
2021/05/17 12:43:55 [DEBUG] databricks_mws_workspaces.this: apply errored, but we're indicating that via the Error pointer rather than returning it: MALFORMED_REQUEST: Invalid AccountId:
...
ERRO[0045] Hit multiple errors:
Hit multiple errors:
exit status 1

Edited comment to reflect that I'm trying to apply the storage CMK, not the managed services CMK as I don't believe the latter can be added to a workspace.

[tonybaron]

@nfx As an aside, related to my previous comment, I see that the accounts REST API is returning a 200 when I try to add a managed services CMK. According to the documentation, that is only available when a workspace is created. This isn't reflected in the Terraform documentation at all, and led to a lot of confusion until I dug into the documentation further.

Could you route that issue to the correct party? I would expect that to return some 4xx with a clear error message.

[nfx]

@tonybaron Can you send a PR with a suggested change to docs/resources/customer_managed_key.md?..

[mbarrien]

My org is running into this issue updating a database_mws_workspaces object. I suspect a bug at https://github.com/databrickslabs/terraform-provider-databricks/blob/master/mws/resource_workspace.go#L110 since I don't see AccountID being populated in the workspace object being passed to the client.Patch method.

[nfx]

@mbarrien Good catch! We'll look into this soon. You can accelerate this by submitting a PR