[ISSUE] Unable to Create Databricks Repo with v0.3.8

[troyinsight]

Howdy,

I'm trying to set up a new Repo within Databricks, pointing to a private GitHub repository. I have manually created a GitHub PAT and set up my Git Integration to use this. I can manually create the Repo using the Databricks UI. Trying to create the repo via Terraform fails with an 'Invalid access token' error.

Configuration

provider "databricks" {
  host  = module.databricks_workspace.workspace_url
}

Expected Behavior

Creates a new Repo linked to my GitHub repository.

Actual Behavior

I receive an error stating: Error: Invalid access token

Steps to Reproduce

1. Create PAT token within GitHub
2. Update Git Integration to use this token
3. terraform apply

Terraform and provider versions

Terraform v1.0.8
on darwin_amd64

• provider registry.terraform.io/databrickslabs/databricks v0.3.8
• provider registry.terraform.io/hashicorp/azuread v2.6.0
• provider registry.terraform.io/hashicorp/azurerm v2.79.1
• provider registry.terraform.io/hashicorp/random v3.1.0
• provider registry.terraform.io/integrations/github v4.16.0
• provider registry.terraform.io/microsoft/azuredevops v0.1.7

[alexott]

@troyinsight please attach debug output. It looks like you can’t authenticate to DB itself

[troyinsight]

Thanks @alexott. I think I've just discovered the issue. You can see from the debug output below that it's using the DEFAULT profile from my .databrickscfg file. Why would it be using this file? I don't have any issues creating clusters, pools, secret scopes etc, just my new repo.

2021-10-08T18:37:49.950+1030 [INFO] provider.terraform-provider-databricks_v0.3.8: Using DEFAULT profile from /Users/troy/.databrickscfg: timestamp=2021-10-08T18:37:49.950+1030
2021-10-08T18:37:49.950+1030 [INFO] provider.terraform-provider-databricks_v0.3.8: Using Bearer authentication from ~/.databrickscfg: timestamp=2021-10-08T18:37:49.950+1030
2021-10-08T18:37:49.950+1030 [DEBUG] provider.terraform-provider-databricks_v0.3.8: POST /repos {
"url": "https://github.com/troy/databricks.git"
2021-10-08T18:37:50.236+1030 [DEBUG] provider.terraform-provider-databricks_v0.3.8: 403 Forbidden: timestamp=2021-10-08T18:37:50.236+1030
2021-10-08T18:37:50.236+1030 [WARN] provider.terraform-provider-databricks_v0.3.8: /api/2.0/repos:403 - Invalid access token: timestamp=2021-10-08T18:37:50.236+1030
2021-10-08T18:37:50.236+1030 [WARN] provider.terraform-provider-databricks_v0.3.8: /api/2.0/repos:403 - Invalid access token: timestamp=2021-10-08T18:37:50.236+1030

[alexott]

Repos functionality uses the same authentication as everything else. In your configuration you provided only host, but you need to provide token as well if you want to point to another workspace.

[troyinsight]

Sorry, I should have mentioned that I'm using Azure CLI Authentication.

The workspace is created using the AzureRM provider. I then reference this straight away and start creating creating clusters, pools, secret scopes etc. These all work fine. As a test I just upgraded one of my clusters to DBR 9.1. It seems to be just the Repo functionality.

[alexott]

Can you include bigger chunk of debug log? There was a refactoring for auth pieces, maybe it’s related…

[nfx]

@troyinsight most likely you have to add your GitHub PAT to Databricks workspace manually. At least for now, until the API for that is released soon.

[nfx]

@troyinsight so with missing GitHub PAT token, you'd see Missing Git provider credentials. Go to User Settings > Git Integration to add your personal access token. error. Invalid access token is just strange. You can also open a support ticket with more details and debug logs.

[troyinsight]

Thanks guys. I'm just doing some more testing this morning. I totally understand that at this stage we need to manually add our GitHub PAT token into the workspace, which I have done. As mentioned, I can create the repo within the Databricks UI and it works fine. As a test I've also tried using Terraform to create a Repo pointing to a different (and Public) GitHub repository. This didn't work either, so it doesn't seem to matter if the GitHub repo is public or private. I'll post a Gist soon.

[troyinsight]

Okay, here's a link to my debug output. I've totally refactored my Terraform module so it doesn't include anything but Databricks (to keep the output smaller). My module now creates a single Databricks workspace, creates a cluster and tries to create the repo. Obviously, after the Workspace is initially created I enter my GitHub PAT token.

https://gist.github.com/troyinsight/f970a933a413eb9f69a822e55f4b30df

[alexott]

Hmmmm, it’s strange - for cluster I see azcli auth, but not for repos. Can you post your terraform code to alexey.ott at databricks.com

[nfx]

@alexott aren't you on vacation? 🥸🤫

[nfx]

@troyinsight It looks to me that you didn't initialize Databricks provider in module.troymod - e.g. no host is supplied and it looks in .databrickscfg as the last resort.

Please make sure you pass host to a module and init it there as well.

[nfx]

@troyinsight can you confirm that provider has no initialization in the module with a repo declaration? And it works, once you explicitly pass the host of the workspace?

I've reopened #821 to make auth messages less confusing.

[troyinsight]

Ahh, thanks guys, it was user error. I am initialising the Databricks provider within my module, the issue is that I have two workspaces (one for ML). I wasn't using the provider alias within my databricks_repo resource. Sorry for the hassle.

It would be aweome if we could get Git Integration working through Terraform eventually :-)

[nfx]

API for adding GH PAT should be there in more than few months. Keep escalating it to your databricks representatives so it's pushed up in priority.

Closing this issue.