[ISSUE] failed library in cluster errors terraform refresh

[tomasz-kuzemko]

Terraform Version

terraform -v   
Terraform v0.13.5
+ provider registry.terraform.io/databrickslabs/databricks v0.3.4

Affected Resource(s)

• databricks_cluster

Environment variable names

To get relevant environment variable names please copypaste the output of the following command:

ARM_CLIENT_ID
ARM_CLIENT_SECRET
ARM_TENANT_ID

Debug Output

2021-06-30T15:38:31.741+0200 [DEBUG] plugin.terraform-provider-databricks_v0.3.4.exe: 200 OK {
  "cluster_id": "XXX",
  "library_statuses": [
[...]
        ... (885 more bytes) <- GET /libraries/cluster-status?cluster_id=XXX: timestamp=2021-06-30T15:38:31.740+0200
2021/06/30 15:38:31 [ERROR] eval: *terraform.EvalRefresh, err: library_maven[com.microsoft.sqlserver:mssql-jdbc:9.3.1.jre15-preview] failed: com/microsoft/sqlserver/mssql-jdbc#9.2.1.jre15.jar has been evicted due to: [com/microsoft/sqlserver/mssql-jdbc#9.3.1.jre15-preview.jar]., com/microsoft/sqlserver/mssql-jdbc#6.4.0.jre8.jar has been evicted due to: [com/microsoft/sqlserver/mssql-jdbc#9.3.1.jre15-preview.jar].
2021/06/30 15:38:31 [ERROR] eval: *terraform.EvalSequence, err: library_maven[com.microsoft.sqlserver:mssql-jdbc:9.3.1.jre15-preview] failed: com/microsoft/sqlserver/mssql-jdbc#9.2.1.jre15.jar has been evicted due to: [com/microsoft/sqlserver/mssql-jdbc#9.3.1.jre15-preview.jar]., com/microsoft/sqlserver/mssql-jdbc#6.4.0.jre8.jar has been evicted due to: [com/microsoft/sqlserver/mssql-jdbc#9.3.1.jre15-preview.jar].

curl --netrc https://adb-XXX.azuredatabricks.net/api/2.0/libraries/cluster-status?cluster_id=XXX | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  
Current
                                 Dload  Upload   Total   Spent    Left  
Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--100  1447    0  1447    0     0   1513      0 --:--:-- --:--:-- --:--:--100  1447    0  1447    0     0   1513      0 --:--:-- --:--:-- --:--:--  1513
{
  "cluster_id": "XXX",
  "library_statuses": [
[...]
    {
      "library": {
        "maven": {
          "coordinates": "com.microsoft.sqlserver:mssql-jdbc:9.3.1.jre15-preview"
        }
      },
      "status": "FAILED",
      "messages": [
        "com/microsoft/sqlserver/mssql-jdbc#9.2.1.jre15.jar has been evicted due to: [com/microsoft/sqlserver/mssql-jdbc#9.3.1.jre15-preview.jar].",
        "com/microsoft/sqlserver/mssql-jdbc#6.4.0.jre8.jar has been evicted due to: [com/microsoft/sqlserver/mssql-jdbc#9.3.1.jre15-preview.jar]."
      ],
      "is_library_for_all_clusters": false
    }
  ]
}

Expected Behavior

Running terraform refresh should succeed without error, even with a library with "FAILED" status in a cluster managed by Terraform.

Actual Behavior

Running terraform refresh results in error if a Databricks cluster managed via this Terraform configuration happens to have a failed library as returned by the 2.0/libraries/cluster-status API:

PS> terraform refresh
[...]
Error: library_maven[com.microsoft.sqlserver:mssql-jdbc:9.3.1.jre15-preview] failed: com/microsoft/sqlserver/mssql-jdbc#9.2.1.jre15.jar has been evicted due to: [com/microsoft/sqlserver/mssql-jdbc#9.3.1.jre15-preview.jar]., com/microsoft/sqlserver/mssql-jdbc#6.4.0.jre8.jar has been evicted due to: [com/microsoft/sqlserver/mssql-jdbc#9.3.1.jre15-preview.jar].

After stopping the affected cluster in Databricks (e.g. via the UI), terraform refresh is able to succeed.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. Create cluster with databricks_cluster resource
2. Install a library that is likely to result in "FAILED" state
3. terraform refresh will fail when the cluster is running

[nfx]

This is a duplicate of #599, let's track it there.