Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ISSUE] Provider issue - cannot ignore library block completely #820

Closed
vfortierdatabricks opened this issue Sep 3, 2021 · 3 comments
Closed

[ISSUE] Provider issue - cannot ignore library block completely #820

vfortierdatabricks opened this issue Sep 3, 2021 · 3 comments

Comments

@vfortierdatabricks
Copy link

vfortierdatabricks commented Sep 3, 2021 •
edited
Loading

Hello,

opening this issue for a customer. When you create a cluster ressource, if you want to enable end users to add libraries to the cluster, if you use a lifecycle block to ignore changes to library, if you modify the cluster definition and reapply the terraform, it will remove all libraries from the cluster.

Configuration

data "databricks_node_type" "smallest" {
  local_disk = true
}

data "databricks_spark_version" "latest_lts" {
  long_term_support = true
}

resource "databricks_cluster" "shared_autoscaling" {
  cluster_name            = "Shared Autoscaling"
  spark_version           = data.databricks_spark_version.latest_lts.id
  node_type_id            = data.databricks_node_type.smallest.id
  autotermination_minutes = 20
  autoscale {
    min_workers = 2
    max_workers = 10
  }


lifecycle {
      ignore_changes = [library]
}

spark_conf = {
    "spark.databricks.service.server.enabled" = "true"
  }

}

Expected Behavior

If the cluster was running with actual libraries on it, it would have change the cluster but left the libraries alone.

Actual Behavior

Cluster modified and removed all libraries

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. terraform apply

Terraform and provider versions

Terraform v1.0.5
on darwin_amd64

  • provider registry.terraform.io/databrickslabs/databricks v0.3.7

Please paste the output of terraform version. If version of databricks provider is not the latest (https://github.com/databrickslabs/terraform-provider-databricks/releases), please make sure to use the latest one.

Debug Output

Please add turn on logging, e.g. TF_LOG=DEBUG terraform apply and run command again, paste it to gist & provide the link to gist. If you're still willing to paste in log output, make sure you provide only relevant log lines with requests.

It would make it more readable, if you pipe the log through | grep databricks | sed -E 's/^.* plugin[^:]+: (.*)$/\1/', e.g.:

TF_LOG=DEBUG terraform plan | grep databricks | sed -E 's/^.* plugin[^:]+: (.*)$/\1/'

2021-09-03T11:22:13.372-0400 [DEBUG] Adding temp file log sink: /var/folders/n_/z7nz8hxn4n95nyhyvzb5jt4h0000gp/T/terraform-log498155007
2021-09-03T11:22:13.372-0400 [INFO] Terraform version: 1.0.5
2021-09-03T11:22:13.372-0400 [INFO] Go runtime version: go1.16.4
2021-09-03T11:22:13.372-0400 [INFO] CLI args: []string{"/usr/local/bin/terraform", "plan"}
2021-09-03T11:22:13.372-0400 [DEBUG] Attempting to open CLI config file: /Users/vincent.fortier/.terraformrc
2021-09-03T11:22:13.372-0400 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2021-09-03T11:22:13.374-0400 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2021-09-03T11:22:13.374-0400 [DEBUG] ignoring non-existing provider search directory /Users/vincent.fortier/.terraform.d/plugins
2021-09-03T11:22:13.374-0400 [DEBUG] ignoring non-existing provider search directory /Users/vincent.fortier/Library/Application Support/io.terraform/plugins
2021-09-03T11:22:13.374-0400 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2021-09-03T11:22:13.374-0400 [INFO] CLI command args: []string{"plan"}
2021-09-03T11:22:13.376-0400 [DEBUG] New state was assigned lineage "9c70ce27-7f83-f5d1-8a70-fe756e8b80be"
2021-09-03T11:22:13.460-0400 [DEBUG] checking for provisioner in "."
2021-09-03T11:22:13.460-0400 [DEBUG] checking for provisioner in "/usr/local/bin"
2021-09-03T11:22:13.461-0400 [INFO] Failed to read plugin lock file .terraform/plugins/darwin_amd64/lock.json: open .terraform/plugins/darwin_amd64/lock.json: no such file or directory
2021-09-03T11:22:13.461-0400 [INFO] backend/local: starting Plan operation
2021-09-03T11:22:13.463-0400 [DEBUG] created provider logger: level=debug
2021-09-03T11:22:13.463-0400 [INFO] provider: configuring client automatic mTLS
2021-09-03T11:22:13.497-0400 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 args=[.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7]
2021-09-03T11:22:13.501-0400 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 pid=50604
2021-09-03T11:22:13.501-0400 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7
2021-09-03T11:22:13.520-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Databricks Terraform Provider (experimental)
2021-09-03T11:22:13.520-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:22:13.520-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Version 0.3.7
2021-09-03T11:22:13.520-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:22:13.520-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs
2021-09-03T11:22:13.520-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:22:13.521-0400 [INFO] provider.terraform-provider-databricks_v0.3.7: configuring server automatic mTLS: timestamp=2021-09-03T11:22:13.521-0400
2021-09-03T11:22:13.561-0400 [DEBUG] provider: using plugin: version=5
2021-09-03T11:22:13.561-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: plugin address: network=unix address=/var/folders/n_/z7nz8hxn4n95nyhyvzb5jt4h0000gp/T/plugin642139827 timestamp=2021-09-03T11:22:13.560-0400
2021-09-03T11:22:13.612-0400 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-09-03T11:22:13.613-0400 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 pid=50604
2021-09-03T11:22:13.613-0400 [DEBUG] provider: plugin exited
2021-09-03T11:22:13.613-0400 [INFO] terraform: building graph: GraphTypeValidate
2021-09-03T11:22:13.614-0400 [DEBUG] ProviderTransformer: "databricks_cluster.shared_autoscaling" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/databrickslabs/databricks"]
2021-09-03T11:22:13.614-0400 [DEBUG] ProviderTransformer: "data.databricks_node_type.smallest" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/databrickslabs/databricks"]
2021-09-03T11:22:13.614-0400 [DEBUG] ProviderTransformer: "data.databricks_spark_version.latest_lts" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/databrickslabs/databricks"]
2021-09-03T11:22:13.614-0400 [DEBUG] ReferenceTransformer: "databricks_cluster.shared_autoscaling" references: [data.databricks_node_type.smallest data.databricks_spark_version.latest_lts]
2021-09-03T11:22:13.614-0400 [DEBUG] ReferenceTransformer: "data.databricks_node_type.smallest" references: []
2021-09-03T11:22:13.614-0400 [DEBUG] ReferenceTransformer: "data.databricks_spark_version.latest_lts" references: []
2021-09-03T11:22:13.614-0400 [DEBUG] ReferenceTransformer: "provider["registry.terraform.io/databrickslabs/databricks"]" references: []
2021-09-03T11:22:13.615-0400 [DEBUG] Starting graph walk: walkValidate
2021-09-03T11:22:13.615-0400 [DEBUG] created provider logger: level=debug
2021-09-03T11:22:13.615-0400 [INFO] provider: configuring client automatic mTLS
2021-09-03T11:22:13.651-0400 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 args=[.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7]
2021-09-03T11:22:13.655-0400 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 pid=50605
2021-09-03T11:22:13.655-0400 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7
2021-09-03T11:22:13.675-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Databricks Terraform Provider (experimental)
2021-09-03T11:22:13.675-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:22:13.675-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Version 0.3.7
2021-09-03T11:22:13.675-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:22:13.675-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs
2021-09-03T11:22:13.675-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:22:13.677-0400 [INFO] provider.terraform-provider-databricks_v0.3.7: configuring server automatic mTLS: timestamp=2021-09-03T11:22:13.677-0400
2021-09-03T11:22:13.716-0400 [DEBUG] provider: using plugin: version=5
2021-09-03T11:22:13.716-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: plugin address: address=/var/folders/n_/z7nz8hxn4n95nyhyvzb5jt4h0000gp/T/plugin593816272 network=unix timestamp=2021-09-03T11:22:13.716-0400
2021-09-03T11:22:13.777-0400 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-09-03T11:22:13.778-0400 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 pid=50605
2021-09-03T11:22:13.778-0400 [DEBUG] provider: plugin exited
2021-09-03T11:22:13.778-0400 [INFO] backend/local: plan calling Plan
2021-09-03T11:22:13.778-0400 [INFO] terraform: building graph: GraphTypePlan
2021-09-03T11:22:13.779-0400 [DEBUG] ProviderTransformer: "databricks_cluster.shared_autoscaling (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/databrickslabs/databricks"]
2021-09-03T11:22:13.779-0400 [DEBUG] ProviderTransformer: "data.databricks_node_type.smallest (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/databrickslabs/databricks"]
2021-09-03T11:22:13.779-0400 [DEBUG] ProviderTransformer: "data.databricks_spark_version.latest_lts (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/databrickslabs/databricks"]
2021-09-03T11:22:13.779-0400 [DEBUG] ReferenceTransformer: "provider["registry.terraform.io/databrickslabs/databricks"]" references: []
2021-09-03T11:22:13.779-0400 [DEBUG] ReferenceTransformer: "databricks_cluster.shared_autoscaling (expand)" references: [data.databricks_node_type.smallest (expand) data.databricks_spark_version.latest_lts (expand)]
2021-09-03T11:22:13.779-0400 [DEBUG] ReferenceTransformer: "data.databricks_node_type.smallest (expand)" references: []
2021-09-03T11:22:13.779-0400 [DEBUG] ReferenceTransformer: "data.databricks_spark_version.latest_lts (expand)" references: []
2021-09-03T11:22:13.780-0400 [DEBUG] Starting graph walk: walkPlan
2021-09-03T11:22:13.781-0400 [DEBUG] created provider logger: level=debug
2021-09-03T11:22:13.781-0400 [INFO] provider: configuring client automatic mTLS
2021-09-03T11:22:13.817-0400 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 args=[.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7]
2021-09-03T11:22:13.821-0400 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 pid=50606
2021-09-03T11:22:13.821-0400 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7
2021-09-03T11:22:13.842-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Databricks Terraform Provider (experimental)
2021-09-03T11:22:13.842-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:22:13.842-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Version 0.3.7
2021-09-03T11:22:13.842-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:22:13.842-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs
2021-09-03T11:22:13.842-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:22:13.844-0400 [INFO] provider.terraform-provider-databricks_v0.3.7: configuring server automatic mTLS: timestamp=2021-09-03T11:22:13.844-0400
2021-09-03T11:22:13.884-0400 [DEBUG] provider: using plugin: version=5
2021-09-03T11:22:13.884-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: plugin address: address=/var/folders/n_/z7nz8hxn4n95nyhyvzb5jt4h0000gp/T/plugin951928013 network=unix timestamp=2021-09-03T11:22:13.884-0400
2021-09-03T11:22:13.940-0400 [WARN] ValidateProviderConfig from "provider["registry.terraform.io/databrickslabs/databricks"]" changed the config value, but that value is unused
2021-09-03T11:22:13.941-0400 [INFO] provider.terraform-provider-databricks_v0.3.7: Explicit and implicit attributes: host, token: timestamp=2021-09-03T11:22:13.941-0400
2021-09-03T11:22:13.941-0400 [DEBUG] ReferenceTransformer: "data.databricks_node_type.smallest" references: []
2021-09-03T11:22:13.942-0400 [DEBUG] ReferenceTransformer: "data.databricks_spark_version.latest_lts" references: []
2021-09-03T11:22:13.943-0400 [INFO] provider.terraform-provider-databricks_v0.3.7: Using directly configured host+token authentication: timestamp=2021-09-03T11:22:13.943-0400
2021-09-03T11:22:13.943-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/spark-versions: timestamp=2021-09-03T11:22:13.943-0400
2021-09-03T11:22:14.010-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/list-node-types: timestamp=2021-09-03T11:22:14.010-0400
2021-09-03T11:22:14.344-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"versions": [
{
"key": "8.2.x-scala2.12",
"name": "8.2 (includes Apache Spark 3.1.1, Scala 2.12)"
},
{
"key": "6.4.x-esr-scala2.11",
"name": "6.4 Extended Support (includes Apache Spark 2.4.5, Scala 2.11)"
},
{
"key": "7.3.x-cpu-ml-scala2.12",
"name": "7.3 LTS ML (includes Apache Spark 3.0.1, Scala 2.12)"
},
{
"key": "8.0.x-cpu-ml-scala2.12",
"name": "8.0 ML (includes Apache Spark 3.1.1, Scala 2.12)"
},
{
"key": "7.3.x-hls-scala2.12",
"name": "7.3 LTS Genomics (includes Apache Spark 3.0.1, Scala 2.12)"
},
{
"key": "8.2.x-cpu-ml-scala2.12",
"name": "8.2 ML (includes Apache Spark 3.1.1, Scala 2.12)"
},
{
"key": "7.3.x-gpu-ml-scala2.12",
"name": "7.3 LTS ML (includes Apache Spark 3.0.1, GPU, Scala 2.12)"
},
{
"key": "8.4.x-photon-scala2.12",
"name": "8.4 Photon (includes Apache Spark 3.1.2, Scala 2.12)"
},
{
"key": "8.1.x-gpu-ml-scala2.12",
... (2460 more bytes) <- GET /clusters/spark-versions: timestamp=2021-09-03T11:22:14.344-0400
2021-09-03T11:22:14.345-0400 [WARN] Provider "provider["registry.terraform.io/databrickslabs/databricks"]" produced an unexpected new value for data.databricks_spark_version.latest_lts.
- .genomics: was null, but now cty.False
- .gpu: was null, but now cty.False
- .beta: was null, but now cty.False
- .latest: was null, but now cty.True
- .ml: was null, but now cty.False
- .scala: was null, but now cty.StringVal("2.12")
2021-09-03T11:22:14.365-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"node_types": [
{
"category": "General Purpose",
"description": "Standard_DS3_v2",
"display_order": 0,
"instance_type_id": "Standard_DS3_v2",
"is_deprecated": false,
"is_encrypted_in_transit": false,
"is_graviton": false,
"is_hidden": false,
"is_io_cache_enabled": false,
"memory_mb": 14336,
"node_info": {
"available_core_quota": 346,
"total_core_quota": 350
},
"node_instance_type": {
"instance_family": "Standard DSv2 Family vCPUs",
"instance_type_id": "Standard_DS3_v2",
"local_disk_size_gb": 28,
"local_disks": 1,
"local_nvme_disk_size_gb": 0,
"local_nvme_disks": 0,
"swap_size": "10g"
},
"node_type_id": "Standard_DS3_v2",
"num_cores": 4,
"num_gpus": 0,
"photon_driver_capable": false,
"photon_worker_capable": false,
"support_cluster_tags": true,
"support_ebs_volumes": true,
"support_port_forwarding": tr... (123283 more bytes) <- GET /clusters/list-node-types: timestamp=2021-09-03T11:22:14.365-0400
2021-09-03T11:22:14.367-0400 [DEBUG] ReferenceTransformer: "databricks_cluster.shared_autoscaling" references: []
2021-09-03T11:22:14.375-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:22:14.375-0400
2021-09-03T11:22:14.460-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"autoscale": {
"max_workers": 10,
"min_workers": 2
},
"autotermination_minutes": 20,
"azure_attributes": {
"availability": "ON_DEMAND_AZURE",
"first_on_demand": 1,
"spot_bid_max_price": -1
},
"cluster_cores": 12,
"cluster_id": "0903-134511-debt715",
"cluster_memory_mb": 24576,
"cluster_name": "Shared Autoscaling",
"cluster_source": "API",
"creator_user_name": "@databricks.com",
"default_tags": {
"ClusterId": "0903-134511-debt715",
"ClusterName": "Shared Autoscaling",
"Creator": "@databricks.com",
"MLWorkspaceLinkUpdateTime": "07/15/2020 17:01:19 +00:00",
"Owner": "Vincent Fortier",
"OwnerEmail": "@databricks.com",
"Vendor": "Databricks"
},
"disk_spec": {},
"driver": {
"host_private_ip": "10.139.0.5",
"instance_id": "833d346b68834c7e8f03023a98702df4",
"node_id": "f2791b13e1a946f49b9c344f7c7aeb89",
"private_ip": "10.139.64.5",
"public_dns": "52.138.15.202",
"start_... (1208 more bytes) <- GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:22:14.460-0400
2021-09-03T11:22:14.462-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: POST /clusters/events {
"cluster_id": "0903-134511-debt715",
"event_types": [
"PINNED",
"UNPINNED"
],
"limit": 1,
"order": "DESC"
}: timestamp=2021-09-03T11:22:14.462-0400
2021-09-03T11:22:14.634-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"total_count": 0
} <- POST /clusters/events: timestamp=2021-09-03T11:22:14.634-0400
2021-09-03T11:22:14.635-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Waiting for state to become: [success]: timestamp=2021-09-03T11:22:14.635-0400
2021-09-03T11:22:14.635-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /libraries/cluster-status?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:22:14.635-0400
2021-09-03T11:22:14.729-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"cluster_id": "0903-134511-debt715",
"library_statuses": [
{
"is_library_for_all_clusters": false,
"library": {
"pypi": {
"package": "numpy"
}
},
"status": "INSTALLED"
}
]
} <- GET /libraries/cluster-status?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:22:14.728-0400
2021-09-03T11:22:14.736-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: k='azure_attributes.#', old='1', new='0': timestamp=2021-09-03T11:22:14.736-0400
2021-09-03T11:22:14.736-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Disable removal of empty block: timestamp=2021-09-03T11:22:14.736-0400
2021-09-03T11:22:14.737-0400 [WARN] Provider "registry.terraform.io/databrickslabs/databricks" produced an invalid plan for databricks_cluster.shared_autoscaling, but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .enable_local_disk_encryption: planned value cty.False for a non-computed attribute
- .is_pinned: planned value cty.False for a non-computed attribute
- .num_workers: planned value cty.NumberIntVal(0) for a non-computed attribute
- .azure_attributes: block count in plan (1) disagrees with count in config (0)
2021-09-03T11:22:14.739-0400 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-09-03T11:22:14.741-0400 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 pid=50606
2021-09-03T11:22:14.741-0400 [DEBUG] provider: plugin exited
2021-09-03T11:22:14.741-0400 [INFO] backend/local: plan operation completed
databricks_cluster.shared_autoscaling: Refreshing state... [id=0903-134511-debt715]

databricks_cluster.shared_autoscaling will be updated in-place

~ resource "databricks_cluster" "shared_autoscaling" {

TF_LOG=DEBUG terraform plan | grep databricks | sed -E 's/^.* plugin[^:]+: (.*)$/\1/'

2021-09-03T11:23:48.570-0400 [DEBUG] Adding temp file log sink: /var/folders/n_/z7nz8hxn4n95nyhyvzb5jt4h0000gp/T/terraform-log396815294
2021-09-03T11:23:48.570-0400 [INFO] Terraform version: 1.0.5
2021-09-03T11:23:48.570-0400 [INFO] Go runtime version: go1.16.4
2021-09-03T11:23:48.570-0400 [INFO] CLI args: []string{"/usr/local/bin/terraform", "apply"}
2021-09-03T11:23:48.571-0400 [DEBUG] Attempting to open CLI config file: /Users/vincent.fortier/.terraformrc
2021-09-03T11:23:48.571-0400 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2021-09-03T11:23:48.571-0400 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2021-09-03T11:23:48.571-0400 [DEBUG] ignoring non-existing provider search directory /Users/vincent.fortier/.terraform.d/plugins
2021-09-03T11:23:48.571-0400 [DEBUG] ignoring non-existing provider search directory /Users/vincent.fortier/Library/Application Support/io.terraform/plugins
2021-09-03T11:23:48.571-0400 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2021-09-03T11:23:48.572-0400 [INFO] CLI command args: []string{"apply"}
2021-09-03T11:23:48.590-0400 [DEBUG] New state was assigned lineage "164005ea-6081-6112-8a59-1a848187304a"
2021-09-03T11:23:48.674-0400 [DEBUG] checking for provisioner in "."
2021-09-03T11:23:48.676-0400 [DEBUG] checking for provisioner in "/usr/local/bin"
2021-09-03T11:23:48.676-0400 [INFO] Failed to read plugin lock file .terraform/plugins/darwin_amd64/lock.json: open .terraform/plugins/darwin_amd64/lock.json: no such file or directory
2021-09-03T11:23:48.680-0400 [INFO] backend/local: starting Apply operation
2021-09-03T11:23:48.690-0400 [DEBUG] created provider logger: level=debug
2021-09-03T11:23:48.690-0400 [INFO] provider: configuring client automatic mTLS
2021-09-03T11:23:48.738-0400 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 args=[.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7]
2021-09-03T11:23:48.742-0400 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 pid=50631
2021-09-03T11:23:48.743-0400 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7
2021-09-03T11:23:48.762-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Databricks Terraform Provider (experimental)
2021-09-03T11:23:48.762-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:23:48.762-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Version 0.3.7
2021-09-03T11:23:48.762-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:23:48.763-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs
2021-09-03T11:23:48.763-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:23:48.765-0400 [INFO] provider.terraform-provider-databricks_v0.3.7: configuring server automatic mTLS: timestamp=2021-09-03T11:23:48.764-0400
2021-09-03T11:23:48.809-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: plugin address: network=unix address=/var/folders/n_/z7nz8hxn4n95nyhyvzb5jt4h0000gp/T/plugin098951050 timestamp=2021-09-03T11:23:48.808-0400
2021-09-03T11:23:48.809-0400 [DEBUG] provider: using plugin: version=5
2021-09-03T11:23:48.874-0400 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-09-03T11:23:48.875-0400 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 pid=50631
2021-09-03T11:23:48.875-0400 [DEBUG] provider: plugin exited
2021-09-03T11:23:48.877-0400 [INFO] terraform: building graph: GraphTypeValidate
2021-09-03T11:23:48.880-0400 [DEBUG] ProviderTransformer: "data.databricks_spark_version.latest_lts" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/databrickslabs/databricks"]
2021-09-03T11:23:48.880-0400 [DEBUG] ProviderTransformer: "databricks_cluster.shared_autoscaling" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/databrickslabs/databricks"]
2021-09-03T11:23:48.880-0400 [DEBUG] ProviderTransformer: "data.databricks_node_type.smallest" (*terraform.NodeValidatableResource) needs provider["registry.terraform.io/databrickslabs/databricks"]
2021-09-03T11:23:48.882-0400 [DEBUG] ReferenceTransformer: "data.databricks_node_type.smallest" references: []
2021-09-03T11:23:48.882-0400 [DEBUG] ReferenceTransformer: "data.databricks_spark_version.latest_lts" references: []
2021-09-03T11:23:48.882-0400 [DEBUG] ReferenceTransformer: "provider["registry.terraform.io/databrickslabs/databricks"]" references: []
2021-09-03T11:23:48.882-0400 [DEBUG] ReferenceTransformer: "databricks_cluster.shared_autoscaling" references: [data.databricks_spark_version.latest_lts data.databricks_node_type.smallest]
2021-09-03T11:23:48.883-0400 [DEBUG] Starting graph walk: walkValidate
2021-09-03T11:23:48.884-0400 [DEBUG] created provider logger: level=debug
2021-09-03T11:23:48.884-0400 [INFO] provider: configuring client automatic mTLS
2021-09-03T11:23:48.918-0400 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 args=[.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7]
2021-09-03T11:23:48.922-0400 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 pid=50632
2021-09-03T11:23:48.922-0400 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7
2021-09-03T11:23:48.942-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Databricks Terraform Provider (experimental)
2021-09-03T11:23:48.942-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:23:48.942-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Version 0.3.7
2021-09-03T11:23:48.942-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:23:48.942-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs
2021-09-03T11:23:48.942-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:23:48.944-0400 [INFO] provider.terraform-provider-databricks_v0.3.7: configuring server automatic mTLS: timestamp=2021-09-03T11:23:48.944-0400
2021-09-03T11:23:48.983-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: plugin address: address=/var/folders/n_/z7nz8hxn4n95nyhyvzb5jt4h0000gp/T/plugin844291167 network=unix timestamp=2021-09-03T11:23:48.983-0400
2021-09-03T11:23:48.984-0400 [DEBUG] provider: using plugin: version=5
2021-09-03T11:23:49.047-0400 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-09-03T11:23:49.048-0400 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 pid=50632
2021-09-03T11:23:49.048-0400 [DEBUG] provider: plugin exited
2021-09-03T11:23:49.048-0400 [INFO] backend/local: apply calling Plan
2021-09-03T11:23:49.048-0400 [INFO] terraform: building graph: GraphTypePlan
2021-09-03T11:23:49.050-0400 [DEBUG] ProviderTransformer: "databricks_cluster.shared_autoscaling (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/databrickslabs/databricks"]
2021-09-03T11:23:49.050-0400 [DEBUG] ProviderTransformer: "data.databricks_node_type.smallest (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/databrickslabs/databricks"]
2021-09-03T11:23:49.050-0400 [DEBUG] ProviderTransformer: "data.databricks_spark_version.latest_lts (expand)" (*terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/databrickslabs/databricks"]
2021-09-03T11:23:49.050-0400 [DEBUG] ReferenceTransformer: "databricks_cluster.shared_autoscaling (expand)" references: [data.databricks_spark_version.latest_lts (expand) data.databricks_node_type.smallest (expand)]
2021-09-03T11:23:49.050-0400 [DEBUG] ReferenceTransformer: "data.databricks_node_type.smallest (expand)" references: []
2021-09-03T11:23:49.050-0400 [DEBUG] ReferenceTransformer: "data.databricks_spark_version.latest_lts (expand)" references: []
2021-09-03T11:23:49.050-0400 [DEBUG] ReferenceTransformer: "provider["registry.terraform.io/databrickslabs/databricks"]" references: []
2021-09-03T11:23:49.051-0400 [DEBUG] Starting graph walk: walkPlan
2021-09-03T11:23:49.051-0400 [DEBUG] created provider logger: level=debug
2021-09-03T11:23:49.051-0400 [INFO] provider: configuring client automatic mTLS
2021-09-03T11:23:49.087-0400 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 args=[.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7]
2021-09-03T11:23:49.091-0400 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 pid=50633
2021-09-03T11:23:49.091-0400 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7
2021-09-03T11:23:49.112-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Databricks Terraform Provider (experimental)
2021-09-03T11:23:49.112-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:23:49.112-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Version 0.3.7
2021-09-03T11:23:49.112-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:23:49.112-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs
2021-09-03T11:23:49.112-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:23:49.115-0400 [INFO] provider.terraform-provider-databricks_v0.3.7: configuring server automatic mTLS: timestamp=2021-09-03T11:23:49.114-0400
2021-09-03T11:23:49.155-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: plugin address: address=/var/folders/n_/z7nz8hxn4n95nyhyvzb5jt4h0000gp/T/plugin077300508 network=unix timestamp=2021-09-03T11:23:49.155-0400
2021-09-03T11:23:49.155-0400 [DEBUG] provider: using plugin: version=5
2021-09-03T11:23:49.212-0400 [WARN] ValidateProviderConfig from "provider["registry.terraform.io/databrickslabs/databricks"]" changed the config value, but that value is unused
2021-09-03T11:23:49.213-0400 [INFO] provider.terraform-provider-databricks_v0.3.7: Explicit and implicit attributes: host, token: timestamp=2021-09-03T11:23:49.213-0400
2021-09-03T11:23:49.214-0400 [DEBUG] ReferenceTransformer: "data.databricks_node_type.smallest" references: []
2021-09-03T11:23:49.215-0400 [DEBUG] ReferenceTransformer: "data.databricks_spark_version.latest_lts" references: []
2021-09-03T11:23:49.218-0400 [INFO] provider.terraform-provider-databricks_v0.3.7: Using directly configured host+token authentication: timestamp=2021-09-03T11:23:49.218-0400
2021-09-03T11:23:49.218-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/list-node-types: timestamp=2021-09-03T11:23:49.218-0400
2021-09-03T11:23:49.285-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/spark-versions: timestamp=2021-09-03T11:23:49.285-0400
2021-09-03T11:23:49.692-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"versions": [
{
"key": "8.2.x-scala2.12",
"name": "8.2 (includes Apache Spark 3.1.1, Scala 2.12)"
},
{
"key": "6.4.x-esr-scala2.11",
"name": "6.4 Extended Support (includes Apache Spark 2.4.5, Scala 2.11)"
},
{
"key": "7.3.x-cpu-ml-scala2.12",
"name": "7.3 LTS ML (includes Apache Spark 3.0.1, Scala 2.12)"
},
{
"key": "8.0.x-cpu-ml-scala2.12",
"name": "8.0 ML (includes Apache Spark 3.1.1, Scala 2.12)"
},
{
"key": "7.3.x-hls-scala2.12",
"name": "7.3 LTS Genomics (includes Apache Spark 3.0.1, Scala 2.12)"
},
{
"key": "8.2.x-cpu-ml-scala2.12",
"name": "8.2 ML (includes Apache Spark 3.1.1, Scala 2.12)"
},
{
"key": "7.3.x-gpu-ml-scala2.12",
"name": "7.3 LTS ML (includes Apache Spark 3.0.1, GPU, Scala 2.12)"
},
{
"key": "8.4.x-photon-scala2.12",
"name": "8.4 Photon (includes Apache Spark 3.1.2, Scala 2.12)"
},
{
"key": "8.1.x-gpu-ml-scala2.12",
... (2460 more bytes) <- GET /clusters/spark-versions: timestamp=2021-09-03T11:23:49.691-0400
2021-09-03T11:23:49.693-0400 [WARN] Provider "provider["registry.terraform.io/databrickslabs/databricks"]" produced an unexpected new value for data.databricks_spark_version.latest_lts.
- .genomics: was null, but now cty.False
- .gpu: was null, but now cty.False
- .scala: was null, but now cty.StringVal("2.12")
- .beta: was null, but now cty.False
- .latest: was null, but now cty.True
- .ml: was null, but now cty.False
2021-09-03T11:23:49.709-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"node_types": [
{
"category": "General Purpose",
"description": "Standard_DS3_v2",
"display_order": 0,
"instance_type_id": "Standard_DS3_v2",
"is_deprecated": false,
"is_encrypted_in_transit": false,
"is_graviton": false,
"is_hidden": false,
"is_io_cache_enabled": false,
"memory_mb": 14336,
"node_info": {
"available_core_quota": 346,
"total_core_quota": 350
},
"node_instance_type": {
"instance_family": "Standard DSv2 Family vCPUs",
"instance_type_id": "Standard_DS3_v2",
"local_disk_size_gb": 28,
"local_disks": 1,
"local_nvme_disk_size_gb": 0,
"local_nvme_disks": 0,
"swap_size": "10g"
},
"node_type_id": "Standard_DS3_v2",
"num_cores": 4,
"num_gpus": 0,
"photon_driver_capable": false,
"photon_worker_capable": false,
"support_cluster_tags": true,
"support_ebs_volumes": true,
"support_port_forwarding": tr... (123283 more bytes) <- GET /clusters/list-node-types: timestamp=2021-09-03T11:23:49.708-0400
2021-09-03T11:23:49.711-0400 [DEBUG] ReferenceTransformer: "databricks_cluster.shared_autoscaling" references: []
2021-09-03T11:23:49.717-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:23:49.717-0400
2021-09-03T11:23:49.802-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"autoscale": {
"max_workers": 10,
"min_workers": 2
},
"autotermination_minutes": 20,
"azure_attributes": {
"availability": "ON_DEMAND_AZURE",
"first_on_demand": 1,
"spot_bid_max_price": -1
},
"cluster_cores": 12,
"cluster_id": "0903-134511-debt715",
"cluster_memory_mb": 24576,
"cluster_name": "Shared Autoscaling",
"cluster_source": "API",
"creator_user_name": "@databricks.com",
"default_tags": {
"ClusterId": "0903-134511-debt715",
"ClusterName": "Shared Autoscaling",
"Creator": "@databricks.com",
"MLWorkspaceLinkUpdateTime": "07/15/2020 17:01:19 +00:00",
"Owner": "Vincent Fortier",
"OwnerEmail": "@databricks.com",
"Vendor": "Databricks"
},
"disk_spec": {},
"driver": {
"host_private_ip": "10.139.0.5",
"instance_id": "833d346b68834c7e8f03023a98702df4",
"node_id": "f2791b13e1a946f49b9c344f7c7aeb89",
"private_ip": "10.139.64.5",
"public_dns": "52.138.15.202",
"start_... (1208 more bytes) <- GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:23:49.801-0400
2021-09-03T11:23:49.803-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: POST /clusters/events {
"cluster_id": "0903-134511-debt715",
"event_types": [
"PINNED",
"UNPINNED"
],
"limit": 1,
"order": "DESC"
}: timestamp=2021-09-03T11:23:49.802-0400
2021-09-03T11:23:49.984-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"total_count": 0
} <- POST /clusters/events: timestamp=2021-09-03T11:23:49.984-0400
2021-09-03T11:23:49.985-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Waiting for state to become: [success]: timestamp=2021-09-03T11:23:49.985-0400
2021-09-03T11:23:49.985-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /libraries/cluster-status?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:23:49.985-0400
2021-09-03T11:23:50.081-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"cluster_id": "0903-134511-debt715",
"library_statuses": [
{
"is_library_for_all_clusters": false,
"library": {
"pypi": {
"package": "numpy"
}
},
"status": "INSTALLED"
}
]
} <- GET /libraries/cluster-status?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:23:50.081-0400
2021-09-03T11:23:50.088-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: k='azure_attributes.#', old='1', new='0': timestamp=2021-09-03T11:23:50.088-0400
2021-09-03T11:23:50.088-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Disable removal of empty block: timestamp=2021-09-03T11:23:50.088-0400
2021-09-03T11:23:50.090-0400 [WARN] Provider "registry.terraform.io/databrickslabs/databricks" produced an invalid plan for databricks_cluster.shared_autoscaling, but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .enable_local_disk_encryption: planned value cty.False for a non-computed attribute
- .num_workers: planned value cty.NumberIntVal(0) for a non-computed attribute
- .is_pinned: planned value cty.False for a non-computed attribute
- .azure_attributes: block count in plan (1) disagrees with count in config (0)
2021-09-03T11:23:50.092-0400 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-09-03T11:23:50.095-0400 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 pid=50633
2021-09-03T11:23:50.095-0400 [DEBUG] provider: plugin exited
2021-09-03T11:23:50.101-0400 [DEBUG] command: asking for input: "\nDo you want to perform these actions?"
yes
2021-09-03T11:23:54.296-0400 [INFO] backend/local: apply calling Apply
2021-09-03T11:23:54.296-0400 [INFO] terraform: building graph: GraphTypeApply
2021-09-03T11:23:54.298-0400 [DEBUG] ProviderTransformer: "databricks_cluster.shared_autoscaling (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/databrickslabs/databricks"]
2021-09-03T11:23:54.298-0400 [DEBUG] ProviderTransformer: "data.databricks_node_type.smallest (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/databrickslabs/databricks"]
2021-09-03T11:23:54.298-0400 [DEBUG] ProviderTransformer: "data.databricks_spark_version.latest_lts (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/databrickslabs/databricks"]
2021-09-03T11:23:54.299-0400 [DEBUG] ProviderTransformer: "databricks_cluster.shared_autoscaling" (*terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/databrickslabs/databricks"]
2021-09-03T11:23:54.299-0400 [DEBUG] ReferenceTransformer: "data.databricks_spark_version.latest_lts (expand)" references: []
2021-09-03T11:23:54.299-0400 [DEBUG] ReferenceTransformer: "databricks_cluster.shared_autoscaling" references: [data.databricks_spark_version.latest_lts (expand) data.databricks_node_type.smallest (expand)]
2021-09-03T11:23:54.299-0400 [DEBUG] ReferenceTransformer: "provider["registry.terraform.io/databrickslabs/databricks"]" references: []
2021-09-03T11:23:54.299-0400 [DEBUG] ReferenceTransformer: "databricks_cluster.shared_autoscaling (expand)" references: []
2021-09-03T11:23:54.299-0400 [DEBUG] ReferenceTransformer: "data.databricks_node_type.smallest (expand)" references: []
2021-09-03T11:23:54.300-0400 [DEBUG] Starting graph walk: walkApply
2021-09-03T11:23:54.300-0400 [DEBUG] created provider logger: level=debug
2021-09-03T11:23:54.300-0400 [INFO] provider: configuring client automatic mTLS
2021-09-03T11:23:54.341-0400 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 args=[.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7]
2021-09-03T11:23:54.345-0400 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7 pid=50635
2021-09-03T11:23:54.345-0400 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/databrickslabs/databricks/0.3.7/darwin_amd64/terraform-provider-databricks_v0.3.7
2021-09-03T11:23:54.365-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Databricks Terraform Provider (experimental)
2021-09-03T11:23:54.365-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:23:54.365-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Version 0.3.7
2021-09-03T11:23:54.365-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:23:54.365-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs
2021-09-03T11:23:54.365-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7:
2021-09-03T11:23:54.367-0400 [INFO] provider.terraform-provider-databricks_v0.3.7: configuring server automatic mTLS: timestamp=2021-09-03T11:23:54.367-0400
2021-09-03T11:23:54.407-0400 [DEBUG] provider: using plugin: version=5
2021-09-03T11:23:54.407-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: plugin address: address=/var/folders/n_/z7nz8hxn4n95nyhyvzb5jt4h0000gp/T/plugin499993326 network=unix timestamp=2021-09-03T11:23:54.407-0400
2021-09-03T11:23:54.462-0400 [WARN] ValidateProviderConfig from "provider["registry.terraform.io/databrickslabs/databricks"]" changed the config value, but that value is unused
2021-09-03T11:23:54.463-0400 [INFO] provider.terraform-provider-databricks_v0.3.7: Explicit and implicit attributes: host, token: timestamp=2021-09-03T11:23:54.463-0400
2021-09-03T11:23:54.469-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: k='azure_attributes.#', old='1', new='0': timestamp=2021-09-03T11:23:54.469-0400
2021-09-03T11:23:54.469-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Disable removal of empty block: timestamp=2021-09-03T11:23:54.469-0400
2021-09-03T11:23:54.470-0400 [WARN] Provider "registry.terraform.io/databrickslabs/databricks" produced an invalid plan for databricks_cluster.shared_autoscaling, but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .enable_local_disk_encryption: planned value cty.False for a non-computed attribute
- .num_workers: planned value cty.NumberIntVal(0) for a non-computed attribute
- .is_pinned: planned value cty.False for a non-computed attribute
- .azure_attributes: block count in plan (1) disagrees with count in config (0)
2021-09-03T11:23:54.471-0400 [INFO] Starting apply for databricks_cluster.shared_autoscaling
2021-09-03T11:23:54.472-0400 [DEBUG] databricks_cluster.shared_autoscaling: applying the planned Update change
2021-09-03T11:23:54.474-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Cluster state has changed!: timestamp=2021-09-03T11:23:54.474-0400
2021-09-03T11:23:54.474-0400 [INFO] provider.terraform-provider-databricks_v0.3.7: Using directly configured host+token authentication: timestamp=2021-09-03T11:23:54.474-0400
2021-09-03T11:23:54.474-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:23:54.474-0400
2021-09-03T11:23:54.882-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"autoscale": {
"max_workers": 10,
"min_workers": 2
},
"autotermination_minutes": 20,
"azure_attributes": {
"availability": "ON_DEMAND_AZURE",
"first_on_demand": 1,
"spot_bid_max_price": -1
},
"cluster_cores": 12,
"cluster_id": "0903-134511-debt715",
"cluster_memory_mb": 24576,
"cluster_name": "Shared Autoscaling",
"cluster_source": "API",
"creator_user_name": "vincent.fortier@databricks.com",
"default_tags": {
"ClusterId": "0903-134511-debt715",
"ClusterName": "Shared Autoscaling",
"Creator": "vincent.fortier@databricks.com",
"MLWorkspaceLinkUpdateTime": "07/15/2020 17:01:19 +00:00",
"Owner": "Vincent Fortier",
"OwnerEmail": "vincent.fortier@databricks.com",
"Vendor": "Databricks"
},
"disk_spec": {},
"driver": {
"host_private_ip": "10.139.0.5",
"instance_id": "833d346b68834c7e8f03023a98702df4",
"node_id": "f2791b13e1a946f49b9c344f7c7aeb89",
"private_ip": "10.139.64.5",
"public_dns": "52.138.15.202",
"start_... (1208 more bytes) <- GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:23:54.882-0400
2021-09-03T11:23:54.883-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: POST /clusters/edit {
"autoscale": {
"max_workers": 10,
"min_workers": 1
},
"autotermination_minutes": 20,
"azure_attributes": {
"availability": "ON_DEMAND_AZURE",
"first_on_demand": 1,
"spot_bid_max_price": -1
},
"cluster_id": "0903-134511-debt715",
"cluster_name": "Shared Autoscaling",
"driver_node_type_id": "Standard_F4s",
"enable_elastic_disk": true,
"node_type_id": "Standard_F4s",
"num_workers": 0,
"spark_conf": {
"spark.databricks.service.server.enabled": "true"
},
"spark_version": "7.3.x-scala2.12"
}: timestamp=2021-09-03T11:23:54.883-0400
2021-09-03T11:23:55.104-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {} <- POST /clusters/edit: timestamp=2021-09-03T11:23:55.104-0400
2021-09-03T11:23:55.104-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:23:55.104-0400
2021-09-03T11:23:55.271-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"autoscale": {
"max_workers": 10,
"min_workers": 1
},
"autotermination_minutes": 20,
"azure_attributes": {
"availability": "ON_DEMAND_AZURE",
"first_on_demand": 1,
"spot_bid_max_price": -1
},
"cluster_id": "0903-134511-debt715",
"cluster_name": "Shared Autoscaling",
"cluster_source": "API",
"creator_user_name": "vincent.fortier@databricks.com",
"default_tags": {
"ClusterId": "0903-134511-debt715",
"ClusterName": "Shared Autoscaling",
"Creator": "vincent.fortier@databricks.com",
"MLWorkspaceLinkUpdateTime": "07/15/2020 17:01:19 +00:00",
"Owner": "Vincent Fortier",
"OwnerEmail": "vincent.fortier@databricks.com",
"Vendor": "Databricks"
},
"disk_spec": {},
"driver": {
"host_private_ip": "10.139.0.5",
"instance_id": "833d346b68834c7e8f03023a98702df4",
"node_id": "f2791b13e1a946f49b9c344f7c7aeb89",
"private_ip": "10.139.64.5",
"public_dns": "52.138.15.202",
"start_timestamp": 1630682367328
},
"driver_instance_sou... (1175 more bytes) <- GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:23:55.271-0400
2021-09-03T11:23:55.271-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Waiting for state to become: [success]: timestamp=2021-09-03T11:23:55.271-0400
2021-09-03T11:23:55.271-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:23:55.271-0400
2021-09-03T11:23:55.358-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"autoscale": {
"max_workers": 10,
"min_workers": 1
},
"autotermination_minutes": 20,
"azure_attributes": {
"availability": "ON_DEMAND_AZURE",
"first_on_demand": 1,
"spot_bid_max_price": -1
},
"cluster_id": "0903-134511-debt715",
"cluster_name": "Shared Autoscaling",
"cluster_source": "API",
"creator_user_name": "vincent.fortier@databricks.com",
"default_tags": {
"ClusterId": "0903-134511-debt715",
"ClusterName": "Shared Autoscaling",
"Creator": "vincent.fortier@databricks.com",
"MLWorkspaceLinkUpdateTime": "07/15/2020 17:01:19 +00:00",
"Owner": "Vincent Fortier",
"OwnerEmail": "vincent.fortier@databricks.com",
"Vendor": "Databricks"
},
"disk_spec": {},
"driver": {
"host_private_ip": "10.139.0.5",
"instance_id": "833d346b68834c7e8f03023a98702df4",
"node_id": "f2791b13e1a946f49b9c344f7c7aeb89",
"private_ip": "10.139.64.5",
"public_dns": "52.138.15.202",
"start_timestamp": 1630682367328
},
"driver_instance_sou... (1175 more bytes) <- GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:23:55.358-0400
2021-09-03T11:23:55.358-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Cluster 0903-134511-debt715 is RESTARTING: Removing 3 nodes.: timestamp=2021-09-03T11:23:55.358-0400
2021-09-03T11:23:55.859-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:23:55.859-0400
2021-09-03T11:23:55.943-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"autoscale": {
"max_workers": 10,
"min_workers": 1
},
"autotermination_minutes": 20,
"azure_attributes": {
"availability": "ON_DEMAND_AZURE",
"first_on_demand": 1,
"spot_bid_max_price": -1
},
"cluster_id": "0903-134511-debt715",
"cluster_name": "Shared Autoscaling",
"cluster_source": "API",
"creator_user_name": "vincent.fortier@databricks.com",
"default_tags": {
"ClusterId": "0903-134511-debt715",
"ClusterName": "Shared Autoscaling",
"Creator": "vincent.fortier@databricks.com",
"MLWorkspaceLinkUpdateTime": "07/15/2020 17:01:19 +00:00",
"Owner": "Vincent Fortier",
"OwnerEmail": "vincent.fortier@databricks.com",
"Vendor": "Databricks"
},
"disk_spec": {},
"driver": {
"host_private_ip": "10.139.0.5",
"instance_id": "833d346b68834c7e8f03023a98702df4",
"node_id": "f2791b13e1a946f49b9c344f7c7aeb89",
"private_ip": "10.139.64.5",
"public_dns": "52.138.15.202",
"start_timestamp": 1630682367328
},
"driver_instance_sou... (1175 more bytes) <- GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:23:55.942-0400
2021-09-03T11:23:55.943-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Cluster 0903-134511-debt715 is RESTARTING: Removing 3 nodes.: timestamp=2021-09-03T11:23:55.942-0400
2021-09-03T11:23:56.945-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:23:56.945-0400
2021-09-03T11:23:57.030-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"autoscale": {
"max_workers": 10,
"min_workers": 1
},
"autotermination_minutes": 20,
"azure_attributes": {
"availability": "ON_DEMAND_AZURE",
"first_on_demand": 1,
"spot_bid_max_price": -1
},
"cluster_id": "0903-134511-debt715",
"cluster_name": "Shared Autoscaling",
"cluster_source": "API",
"creator_user_name": "vincent.fortier@databricks.com",
"default_tags": {
"ClusterId": "0903-134511-debt715",
"ClusterName": "Shared Autoscaling",
"Creator": "vincent.fortier@databricks.com",
"MLWorkspaceLinkUpdateTime": "07/15/2020 17:01:19 +00:00",
"Owner": "Vincent Fortier",
"OwnerEmail": "vincent.fortier@databricks.com",
"Vendor": "Databricks"
},
"disk_spec": {},
"driver": {
"host_private_ip": "10.139.0.5",
"instance_id": "833d346b68834c7e8f03023a98702df4",
"node_id": "f2791b13e1a946f49b9c344f7c7aeb89",
"private_ip": "10.139.64.5",
"public_dns": "52.138.15.202",
"start_timestamp": 1630682367328
},
"driver_instance_sou... (663 more bytes) <- GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:23:57.030-0400
2021-09-03T11:23:57.030-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Cluster 0903-134511-debt715 is RESTARTING: Finding instances for new nodes, acquiring more instances if necessary: timestamp=2021-09-03T11:23:57.030-0400
2021-09-03T11:23:59.033-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:23:59.033-0400
2021-09-03T11:23:59.169-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"autoscale": {
"max_workers": 10,
"min_workers": 1
},
"autotermination_minutes": 20,
"azure_attributes": {
"availability": "ON_DEMAND_AZURE",
"first_on_demand": 1,
"spot_bid_max_price": -1
},
"cluster_id": "0903-134511-debt715",
"cluster_name": "Shared Autoscaling",
"cluster_source": "API",
"creator_user_name": "vincent.fortier@databricks.com",
"default_tags": {
"ClusterId": "0903-134511-debt715",
"ClusterName": "Shared Autoscaling",
"Creator": "vincent.fortier@databricks.com",
"MLWorkspaceLinkUpdateTime": "07/15/2020 17:01:19 +00:00",
"Owner": "Vincent Fortier",
"OwnerEmail": "vincent.fortier@databricks.com",
"Vendor": "Databricks"
},
"disk_spec": {},
"driver": {
"host_private_ip": "10.139.0.5",
"instance_id": "833d346b68834c7e8f03023a98702df4",
"node_id": "f2791b13e1a946f49b9c344f7c7aeb89",
"private_ip": "10.139.64.5",
"public_dns": "52.138.15.202",
"start_timestamp": 1630682367328
},
"driver_instance_sou... (663 more bytes) <- GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:23:59.169-0400
2021-09-03T11:23:59.169-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Cluster 0903-134511-debt715 is RESTARTING: Finding instances for new nodes, acquiring more instances if necessary: timestamp=2021-09-03T11:23:59.169-0400
2021-09-03T11:24:03.172-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:03.172-0400
2021-09-03T11:24:03.261-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"autoscale": {
"max_workers": 10,
"min_workers": 1
},
"autotermination_minutes": 20,
"azure_attributes": {
"availability": "ON_DEMAND_AZURE",
"first_on_demand": 1,
"spot_bid_max_price": -1
},
"cluster_id": "0903-134511-debt715",
"cluster_name": "Shared Autoscaling",
"cluster_source": "API",
"creator_user_name": "vincent.fortier@databricks.com",
"default_tags": {
"ClusterId": "0903-134511-debt715",
"ClusterName": "Shared Autoscaling",
"Creator": "vincent.fortier@databricks.com",
"MLWorkspaceLinkUpdateTime": "07/15/2020 17:01:19 +00:00",
"Owner": "Vincent Fortier",
"OwnerEmail": "vincent.fortier@databricks.com",
"Vendor": "Databricks"
},
"disk_spec": {},
"driver": {
"host_private_ip": "10.139.0.5",
"instance_id": "833d346b68834c7e8f03023a98702df4",
"node_id": "f2791b13e1a946f49b9c344f7c7aeb89",
"private_ip": "10.139.64.5",
"public_dns": "52.138.15.202",
"start_timestamp": 1630682367328
},
"driver_instance_sou... (612 more bytes) <- GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:03.261-0400
2021-09-03T11:24:03.262-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Cluster 0903-134511-debt715 is RESTARTING: Setting up 3 nodes.: timestamp=2021-09-03T11:24:03.261-0400
2021-09-03T11:24:11.265-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:11.264-0400
2021-09-03T11:24:11.359-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"autoscale": {
"max_workers": 10,
"min_workers": 1
},
"autotermination_minutes": 20,
"azure_attributes": {
"availability": "ON_DEMAND_AZURE",
"first_on_demand": 1,
"spot_bid_max_price": -1
},
"cluster_id": "0903-134511-debt715",
"cluster_name": "Shared Autoscaling",
"cluster_source": "API",
"creator_user_name": "vincent.fortier@databricks.com",
"default_tags": {
"ClusterId": "0903-134511-debt715",
"ClusterName": "Shared Autoscaling",
"Creator": "vincent.fortier@databricks.com",
"MLWorkspaceLinkUpdateTime": "07/15/2020 17:01:19 +00:00",
"Owner": "Vincent Fortier",
"OwnerEmail": "vincent.fortier@databricks.com",
"Vendor": "Databricks"
},
"disk_spec": {},
"driver_instance_source": {
"node_type_id": "Standard_F4s"
},
"driver_node_type_id": "Standard_F4s",
"enable_elastic_disk": true,
"enable_local_disk_encryption": false,
"init_scripts_safe_mode": false,
"instance_source": {
"node_type_id": "Standard_F4s"
},
"last_... (318 more bytes) <- GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:11.359-0400
2021-09-03T11:24:11.359-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Cluster 0903-134511-debt715 is RESTARTING: Starting Spark: timestamp=2021-09-03T11:24:11.359-0400
2021-09-03T11:24:21.359-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:21.359-0400
2021-09-03T11:24:21.496-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"autoscale": {
"max_workers": 10,
"min_workers": 1
},
"autotermination_minutes": 20,
"azure_attributes": {
"availability": "ON_DEMAND_AZURE",
"first_on_demand": 1,
"spot_bid_max_price": -1
},
"cluster_id": "0903-134511-debt715",
"cluster_name": "Shared Autoscaling",
"cluster_source": "API",
"creator_user_name": "vincent.fortier@databricks.com",
"default_tags": {
"ClusterId": "0903-134511-debt715",
"ClusterName": "Shared Autoscaling",
"Creator": "vincent.fortier@databricks.com",
"MLWorkspaceLinkUpdateTime": "07/15/2020 17:01:19 +00:00",
"Owner": "Vincent Fortier",
"OwnerEmail": "vincent.fortier@databricks.com",
"Vendor": "Databricks"
},
"disk_spec": {},
"driver_instance_source": {
"node_type_id": "Standard_F4s"
},
"driver_node_type_id": "Standard_F4s",
"enable_elastic_disk": true,
"enable_local_disk_encryption": false,
"init_scripts_safe_mode": false,
"instance_source": {
"node_type_id": "Standard_F4s"
},
"last_... (318 more bytes) <- GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:21.496-0400
2021-09-03T11:24:21.496-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Cluster 0903-134511-debt715 is RESTARTING: Starting Spark: timestamp=2021-09-03T11:24:21.496-0400
2021-09-03T11:24:31.500-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:31.500-0400
2021-09-03T11:24:31.582-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"autoscale": {
"max_workers": 10,
"min_workers": 1
},
"autotermination_minutes": 20,
"azure_attributes": {
"availability": "ON_DEMAND_AZURE",
"first_on_demand": 1,
"spot_bid_max_price": -1
},
"cluster_id": "0903-134511-debt715",
"cluster_name": "Shared Autoscaling",
"cluster_source": "API",
"creator_user_name": "vincent.fortier@databricks.com",
"default_tags": {
"ClusterId": "0903-134511-debt715",
"ClusterName": "Shared Autoscaling",
"Creator": "vincent.fortier@databricks.com",
"MLWorkspaceLinkUpdateTime": "07/15/2020 17:01:19 +00:00",
"Owner": "Vincent Fortier",
"OwnerEmail": "vincent.fortier@databricks.com",
"Vendor": "Databricks"
},
"disk_spec": {},
"driver_instance_source": {
"node_type_id": "Standard_F4s"
},
"driver_node_type_id": "Standard_F4s",
"enable_elastic_disk": true,
"enable_local_disk_encryption": false,
"init_scripts_safe_mode": false,
"instance_source": {
"node_type_id": "Standard_F4s"
},
"last_... (318 more bytes) <- GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:31.582-0400
2021-09-03T11:24:31.582-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Cluster 0903-134511-debt715 is RESTARTING: Starting Spark: timestamp=2021-09-03T11:24:31.582-0400
2021-09-03T11:24:41.586-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:41.585-0400
2021-09-03T11:24:41.681-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"autoscale": {
"max_workers": 10,
"min_workers": 1
},
"autotermination_minutes": 20,
"azure_attributes": {
"availability": "ON_DEMAND_AZURE",
"first_on_demand": 1,
"spot_bid_max_price": -1
},
"cluster_cores": 12,
"cluster_id": "0903-134511-debt715",
"cluster_memory_mb": 24576,
"cluster_name": "Shared Autoscaling",
"cluster_source": "API",
"creator_user_name": "vincent.fortier@databricks.com",
"default_tags": {
"ClusterId": "0903-134511-debt715",
"ClusterName": "Shared Autoscaling",
"Creator": "vincent.fortier@databricks.com",
"MLWorkspaceLinkUpdateTime": "07/15/2020 17:01:19 +00:00",
"Owner": "Vincent Fortier",
"OwnerEmail": "vincent.fortier@databricks.com",
"Vendor": "Databricks"
},
"disk_spec": {},
"driver": {
"host_private_ip": "10.139.0.5",
"instance_id": "833d346b68834c7e8f03023a98702df4",
"node_id": "010c635eda314534b6b4db16d3baf7e1",
"private_ip": "10.139.64.5",
"public_dns": "52.138.15.202",
"start_... (1208 more bytes) <- GET /clusters/get?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:41.681-0400
2021-09-03T11:24:41.681-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Cluster 0903-134511-debt715 is RUNNING:: timestamp=2021-09-03T11:24:41.681-0400
2021-09-03T11:24:41.681-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /libraries/cluster-status?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:41.681-0400
2021-09-03T11:24:41.779-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"cluster_id": "0903-134511-debt715",
"library_statuses": [
{
"is_library_for_all_clusters": false,
"library": {
"pypi": {
"package": "numpy"
}
},
"status": "INSTALLING"
}
]
} <- GET /libraries/cluster-status?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:41.779-0400
2021-09-03T11:24:41.779-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: POST /libraries/uninstall {
"cluster_id": "0903-134511-debt715",
"libraries": [
{
"pypi": {
"package": "numpy"
}
}
]
}: timestamp=2021-09-03T11:24:41.779-0400
2021-09-03T11:24:41.928-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {} <- POST /libraries/uninstall: timestamp=2021-09-03T11:24:41.928-0400
2021-09-03T11:24:41.928-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: Waiting for state to become: [success]: timestamp=2021-09-03T11:24:41.928-0400
2021-09-03T11:24:41.928-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /libraries/cluster-status?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:41.928-0400
2021-09-03T11:24:42.029-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"cluster_id": "0903-134511-debt715",
"library_statuses": [
{
"is_library_for_all_clusters": false,
"library": {
"pypi": {
"package": "numpy"
}
},
"status": "INSTALLING"
}
]
} <- GET /libraries/cluster-status?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:42.028-0400
2021-09-03T11:24:42.529-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /libraries/cluster-status?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:42.529-0400
2021-09-03T11:24:42.622-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"cluster_id": "0903-134511-debt715",
"library_statuses": [
{
"is_library_for_all_clusters": false,
"library": {
"pypi": {
"package": "numpy"
}
},
"status": "INSTALLING"
}
]
} <- GET /libraries/cluster-status?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:42.622-0400
2021-09-03T11:24:43.623-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /libraries/cluster-status?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:43.623-0400
2021-09-03T11:24:43.712-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"cluster_id": "0903-134511-debt715",
"library_statuses": [
{
"is_library_for_all_clusters": false,
"library": {
"pypi": {
"package": "numpy"
}
},
"status": "INSTALLING"
}
]
} <- GET /libraries/cluster-status?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:43.712-0400
2021-09-03T11:24:45.713-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /libraries/cluster-status?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:45.713-0400
2021-09-03T11:24:45.802-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"cluster_id": "0903-134511-debt715",
"library_statuses": [
{
"is_library_for_all_clusters": false,
"library": {
"pypi": {
"package": "numpy"
}
},
"status": "INSTALLING"
}
]
} <- GET /libraries/cluster-status?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:45.802-0400
2021-09-03T11:24:49.804-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: GET /libraries/cluster-status?cluster_id=0903-134511-debt715: timestamp=2021-09-03T11:24:49.804-0400
2021-09-03T11:24:49.911-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: 200 OK {
"cluster_id": "0903-134511-debt715",
"library_statuses": [
{
"is_library_for_all_clusters": false,
"library": {
"pypi": {
"package": "numpy"
}
},

If Terraform produced a panic, please provide a link to a GitHub Gist containing the output of the crash.log.

Important Factoids

Are there anything atypical about your accounts that we should know?
@vfortierdatabricks
Copy link
Author

I can see that line: 2021-09-03T11:24:41.779-0400 [DEBUG] provider.terraform-provider-databricks_v0.3.7: POST /libraries/uninstall { so clearly the ignore block does not work.

@nfx
Copy link
Contributor

nfx commented Sep 3, 2021

Duplicate of #599 (essentially)

Ignoring blocks is part of Terraform, not provider

@nfx nfx closed this as completed Sep 3, 2021
@jdafoe1
Copy link

jdafoe1 commented Sep 3, 2021

I'm not familiar with the internal workings of Terraform vs provider. Having said that, can you elaborate on why this is a terraform issue and not a provider issue?

Terraform does in fact ignore the changes (i.e. when databricks libraries change outside of terraform, terraform successfully ignores those changes in it's planned output)

However, if a different property on the cluster changes, unrelated to libraries, for example spark_config... Terraform reports that only spark_config needs to be updated in its planned output. Good. But when the API call to change the spark_config runs, it also removes libraries.

From this behavior (again, I'm not an expert), it seems to me that terraform is behaving correctly. It does ignore library changes in planned output. But the API calls from the Databricks provider are still removing the libraries.

@nfx nfx mentioned this issue Nov 10, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nfx @vfortierdatabricks @jdafoe1