databricks warehouses set-workspace-warehouse-config errors with enable_serverless_compute is required

[antonsteenvoorden]

Describe the issue

databricks warehouses set-workspace-warehouse-config fails when providing a JSON.
The error is enable_serverless_compute is required even though this is part of the JSON on the root.

Steps to reproduce the behavior

1. create JSON file sql-warehouse-global-settings.json with contents

{
  "security_policy":"DATA_ACCESS_CONTROL",
  "enable_serverless_compute": false,
  "data_access_config": [
    {
      "key": "spark.sql.warehouse.dir",
      "value": "/mnt/ADLS/spark-warehouse"
    }
  ],
  "enabled_warehouse_types": [
    {
      "enabled":true,
      "warehouse_type":"CLASSIC"
    },
    {
      "enabled":true,
      "warehouse_type":"PRO"
    }
  ],
  "sql_configuration_parameters": {}
}

2. Run databricks warehouses set-workspace-warehouse-config --json @./sql-warehouse-global-settings.json
3. See error

The same happens when:

1. Omitting enable_serverless_compute from the JSON
2. Adding an entry to enabled_warehouse_types with {"enabled":true, "warehouse_type":"SERVERLESS"}
3. Enabling serverless compute via the workspace settings in the databricks UI and retrying all of the above

Expected Behavior

To have set the warehouse configuration. Also according to the API documentation this is not part of the payload.

Actual Behavior

It appears the JSON is validated against something undocumented and causes an error.
Additionally, this is also not returned in the json when calling databricks warehouses get-workspace-warehouse-config.

OS and CLI version

OS: MacOS Sonoma 14.4.1
Databricks-CLI: v0.222.0

Is this a regression?

Did this work in a previous version of the CLI? If so, which versions did you try?

Debug Logs

14:56:17  INFO start pid=84298 version=0.222.0 args="/opt/homebrew/bin/databricks, warehouses, set-workspace-warehouse-config, --json, @./config/sql-warehouse-global-settings.json, --debug"
14:56:18 DEBUG PUT /api/2.0/sql/config/warehouses
> {
>   "data_access_config": [
>     {
>       "key": "spark.sql.warehouse.dir",
>       "value": "/mnt/ADLS/spark-warehouse"
>     },
>   ],
>   "enabled_warehouse_types": [
>     {
>       "enabled": true,
>       "warehouse_type": "CLASSIC"
>     },
>     {
>       "enabled": true,
>       "warehouse_type": "PRO"
>     }
>   ],
>   "security_policy": "DATA_ACCESS_CONTROL",
>   "sql_configuration_parameters": {}
> }
< HTTP/2.0 400 Bad Request
< {
<   "error_code": "INVALID_PARAMETER_VALUE",
<   "message": "enable_serverless_compute is required."
< } pid=84298 sdk=true
14:56:18 DEBUG non-retriable error: enable_serverless_compute is required. pid=84298 sdk=true
Error: enable_serverless_compute is required.
14:56:18 ERROR failed execution pid=84298 exit_code=1 error="enable_serverless_compute is required."

[pietern]

Thanks for reporting.

It is possible that this field hasn't been added to our API specs yet. I have flagged this with the relevant team.

In the mean time you can work around the issue by using the databricks api command and making a raw API call to /api/2.0/sql/config/warehouses with the same body.

[andrewnester]

This seems to be related to the issue in Go SDK when fales boolean values marked as omitempty are being dropped from request body as a result of unmarshalling

databricks/databricks-sdk-go#1002

[andrewnester]

The fix for passing false values in JSON has been merged in #1771 and will be released in the next CLI release