[ISSUE] databricks_job: cannot update new_cluster from node_type_id to instance_pool_id

[marekbrysa]

Configuration

See reproducing test case: marekbrysa@b30d115

Expected Behavior

Existing job with new_cluster can be updated to switch from using node_type_id to instance_pool_id.
Only instance_pool_id is present in the request sent to the API.

Actual Behavior

Applying fails with Error: cannot update job: The field 'node_type_id' cannot be supplied when an instance pool ID is provided.
In the request sent to API, both node_type_id and instance_pool_id are sent, even thought only instance_pool_id is present in new HCL.

Steps to Reproduce

See reproducing test case: marekbrysa@b30d115

The test fails with:

Error:      	Not equal:
        	            	expected: map[string]interface {}{"job_id":789, "new_settings":map[string]interface {}{"max_concurrent_runs":1, "max_retries":3, "min_retry_interval_millis":5000, "name":"Featurizer New", "new_cluster":map[string]interface {}{"driver_instance_pool_id":"instance-pool-driver", "instance_pool_id":"instance-pool-worker", "num_workers":1, "spark_version":"spark-1"}, "retry_on_timeout":true}}
        	            	actual  : map[string]interface {}{"job_id":789, "new_settings":map[string]interface {}{"max_concurrent_runs":1, "max_retries":3, "min_retry_interval_millis":5000, "name":"Featurizer New", "new_cluster":map[string]interface {}{"driver_instance_pool_id":"instance-pool-driver", "driver_node_type_id":"driver-node-type-id", "instance_pool_id":"instance-pool-worker", "node_type_id":"node-type-id", "num_workers":1, "spark_version":"spark-1"}, "retry_on_timeout":true}}

        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -7,5 +7,7 @@
        	            	   (string) (len=4) "name": (string) (len=14) "Featurizer New",
        	            	-  (string) (len=11) "new_cluster": (map[string]interface {}) (len=4) {
        	            	+  (string) (len=11) "new_cluster": (map[string]interface {}) (len=6) {
        	            	    (string) (len=23) "driver_instance_pool_id": (string) (len=20) "instance-pool-driver",
        	            	+   (string) (len=19) "driver_node_type_id": (string) (len=19) "driver-node-type-id",
        	            	    (string) (len=16) "instance_pool_id": (string) (len=20) "instance-pool-worker",
        	            	+   (string) (len=12) "node_type_id": (string) (len=12) "node-type-id",
        	            	    (string) (len=11) "num_workers": (float64) 1,
        	Test:       	TestResourceJobUpdate_NodeTypeToInstancePool
        	Messages:   	json strings do not match

Terraform and provider versions

master

[marekbrysa]

I have a hacky fix to disable computed tag on the properties: marekbrysa@746c7a8 (@mgyucht suggested)
@alexott do you know why computed was there?

[alexott]

@marekbrysa because backend fills values in case if they are missing. Like, if you specify instance pool, then node types from instance pool are filling the node_type_id, etc. For exporter I implemented quite complex logic for deciding if specific field should be omitted or not. Similarly should be done for clusters, but it's not so simple. I think that we had similar issues open already

[marekbrysa]

@alexott thanks for the explanation, what solution do you propose?

Naively, the logic should be something like: "if instance_pool_id is provided in HCL and node_type_id is not provided in HCL then do not send node_type_id in the job reset request (regardless of instance state or result of Read)". Where would be the best place for such logic?

[alexott]

It should be done inside the Update function of the databricks_cluster or databricks_job resource. Someone should code it in Go