[ISSUE] Error: cannot create group: unexpected error handling request: invalid character 'h' looking for beginning of value

[mike-decaro]

Description
Attempting to create a workspace group through terraform; intent of the group is to manage workspace permissions for any service principal associated to the account

Provider access key being used is that of an account-level admin.

Apply works successfully without below databricks_group resource

Reproduction

Provider:

provider "databricks" {
  host  = data.aws_ssm_parameter.databricks_workspace_url.value
  token = data.aws_ssm_parameter.databricks_workspace_token.value
}

Resource:

resource "databricks_group" "service_principals" {
  display_name = "Service Principals - ${local.workspace_name}"
}

Expected behavior
Group is created in workspace

Is it a regression?
No

Debug Logs
Request log:

│ POST /api/2.0/preview/scim/v2/Groups
│ > * Host: 
│ > * Accept: application/json
│ > * Authorization: REDACTED
│ > * Content-Type: application/scim+json; charset=utf-8
│ > * Traceparent: 00-03f3a8b365c15c5f7ba5441774ecae42-5471b186fc92ad05-01
│ > * User-Agent: databricks-tf-provider/1.50.0 databricks-sdk-go/0.44.0 go/1.22.6 os/darwin terraform/1.2.2 resource/group auth/pat
│ > {
│ >   "displayName": "Service Principals - playground",
│ >   "entitlements": [
│ >     {}
│ >   ],
│ >   "schemas": [
│ >     "urn:ietf:params:scim:schemas:core:2.0:Group"
│ >   ]
│ > }
│ < HTTP/2.0 403 Forbidden
│ < * Content-Type: application/json
│ < * Date: Wed, 21 Aug 2024 00:31:39 GMT
│ < * Server: databricks
│ < * Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
│ < * Vary: Accept-Encoding
│ < * X-Content-Type-Options: nosniff
│ < * X-Databricks-Org-Id: 1936515022650552
│ < * X-Databricks-Reason-Phrase: https://dbc-2afba066-fe0a.cloud.databricks.com/api/2.0/preview/scim/v2/Groups is only accessible by admins.
│ < https://dbc-2afba066-fe0a.cloud.databricks.com/api/2.0/preview/scim/v2/Groups is only accessible by admins.
│ 
│   with databricks_group.service_principals,
│   on groups.tf line 13, in resource "databricks_group" "service_principals":
│   13: resource "databricks_group" "service_principals" {
│ 

Other Information

• Terraform version 1.2.2
• Databricks provider 1.5.0

Additional context

Group is being created after change from basic auth to OAuth so that account level service principal's token can be used to apply workspace infrastructure.

[mike-decaro]

It looks like this resulted from an account-level issue we had. A service principal created in the account was not correctly propagated to all of our workspaces; in the ones that it did, we were able to assign admin permissions to the service principal and the terraform was successfully applied. In the others, the error remains.

The message buried in the error that "Groups is only accessible by admins." is correct here, because the service principal is not admin (as it is not found at all in the workspace), but I do feel it is a bit hidden in the response, and a message that "service principal could not be found in workspace XXXX" would have been more clear.

[dhawal25]

Hi @mike-decaro Even I am facing this issue with databricks_entitlement resource in TF. The SP used has admin level access for said DBX workspace. Anything specific I need to check?

[mike-decaro]

@dhawal25 here are the things I have added for the SP

Account

• Service Principal > Roles > Account Admin
• Service Principal > Permissions > Service Principal: Manager & Service Principal: User

Workspace

• workspace settings > Identity and Access > Groups > admins
  • I imagine you've done this already, but worth mentioning just as "stuff I have done"
• I also made a specific "Service Principals - " group that contains only this SP, and has Workspace Access and SQL Access entitlements. It feels like I don't need this, but things are too fragile right now for me to want to remove things.
• Make sure that the Permissions from the account and federated down to the workspace too, so in the workspace you see Service Principal: Manager & Service Principal: User on the SP's permission page for the account principal.

Caveat: Ran into this the same time I was converting to use OAuth instead of basic auth. In order to get the token updated in it's storage for the workspace to use (our account and workspace terraform are split into multiple backends), I had to hand-assign the SP's permissions in the workspace to apply the account Terraform, or the client ID & Secret being used by the account had no permissions to apply the workspace. It felt very circular, but that may well be just how our set up took. It was a previous issue we faced prior to this "group requires admin" though so I'm hoping you don't have to deal with this.

[dhawal25]

Hey @mike-decaro Thanks for the details. Granting admin rights to SP as mentioned worked. Thank you again.