Metastore assignment appears incorrect with multiple metastores present

[james-servco]

What happened?

We have a databricks environment with multiple metastores (one that will soon be deprecated, one that we're moving to)
When I assign a new workspace to the new metastore, it appears to successfully assign it the first time it's created, but on subsequent pulumi up's, it believes that the metastore assignment metastoreId is the old metastore (which is also has never been assigned to). If i proceed with this change, it appears to remove the assignment off the workspace completely.

Example

az_databricks_workspace = az_databricks.Workspace("dbw-default",
    resource_group_name=resource_group.name,
    location=azure_location,
    sku=az_databricks.SkuArgs(
        name='Premium',
    ),
    managed_resource_group_id=f"/subscriptions/{sub_id}/resourceGroups/rg-dataprojects-databricks",
    workspace_name=databricksworkspacename,
    opts=pulumi.ResourceOptions(parent=resource_group)
)

# Assign the existing Metastore to the new Databricks workspace
metastore_assignment = databricks.MetastoreAssignment("dma-default",
    workspace_id=az_databricks_workspace.workspace_id,
    metastore_id=metastore_id,
    default_catalog_name="main",
    opts=pulumi.ResourceOptions(
        parent=az_databricks_workspace
    )
)

Output of pulumi about

CLI
Version 3.107.0
Go Version go1.22.0
Go Compiler gc

Plugins
NAME VERSION
azure-native 2.29.0
databricks 1.32.2
python unknown
random 4.15.1

Host
OS darwin
Version 14.2.1
Arch arm64

This project is written in python: executable='/opt/homebrew/bin/python3' version='3.11.7'

Backend
Name pulumi.com
URL https://app.pulumi.com/jw-servco
User jw-servco
Organizations jw-servco, servco
Token type personal

Dependencies:
NAME VERSION
databricks 0.2
pip 24.0
pulumi_azure_native 2.29.0
pulumi_databricks 1.32.2
pulumi_random 4.15.1
setuptools 69.1.0
wheel 0.42.0

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[iwahbe]

Hi @james-servco. I'm sorry you're hitting this issue. Just to clarify, the first time you run pulumi up it works as expected, but then subsequent pulumi ups produce invalid results. Is that right?

Could you give us a self-contained repro that we can run locally.

---

Looks like this may be related to databricks/terraform-provider-databricks#2489.

[james-servco]

Hey! Yep that is correct. The initial pulumi up works as expected, but subsequent ones (once the resources are created) report that the metastoreId points to the other metastore that's never been assigned, not the one that is currently assigned to it.

I'll attempt to create a repro - but i suspect the issue mentioned appears to be in line with what's happening!