Databricks App with dynamic values from the target variables

[ne250040]

I have a databricks app that works well with hardcoded values on the app.yml file. However when i specify a variable on this file, it does not get resolved to the value from target variable. I dont see an example for doing this. I was able to make use of sql-warehouse to get the warehouse-id, however the custom variables dont work similar way. does the CLI support the custom variables yet? Is there another way to pass variables dynamically?

The way i have structured the files:

databricks.yml:

bundle:
  name: etl_deploy

include:
  - common/*.yml

variables:
  db_connection:
    description: connections to databases
    default: "unknown"

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://11111111.4.gcp.databricks.com/
    variables:
      short_env_name: dev
      permission_level: CAN_MANAGE
      schedule_state: PAUSED
      db_connection: salesforce_dev
  qal:
    mode: production
    default: true
    workspace:
      host: https://1234567.1.gcp.databricks.com/
    variables:
      short_env_name: qal
      permission_level: CAN_RUN
      schedule_state: UNPAUSED
      db_connection: salesforce_uat

Streamlit app's app.yml file:

command:
  - "streamlit"
  - "run"
  - "app.py"

env:
  - name: SQL_WAREHOUSE_ID
    valueFrom: sql-warehouse
  - name: DB_CONNECTION
    value: ${var.db_connection}

resources data.app.yml file

resources:
  apps:
    query-data:
      name: query-data
      description: 'To get sales data from the delta table'

      source_code_path: ../app

      resources:
        - name: "sql-warehouse"
          sql_warehouse:
            id: ${var.sql_warehouse_id}
            permission: "CAN_MANAGE"
      permissions:
        - level: CAN_MANAGE
          group_name: ${var.access_group}
​```

[andrewnester]

@ne250040, thanks for reporting this! Indeed, this is currently the case, as it's not possible to parameterise the app.yaml configuration using bundle configuration. We're working on this and planning to release it in the next couple of months. I'll keep this thread updated.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[nathan-daniels]

So I also ran into this problem. I came up with a workaround that isn't great but gets the job done. I created a script in databricks asset bundle that will fill in a app.yaml.template file.

set -e
TARGET=$DATABRICKS_BUNDLE_TARGET

# Get resolved bundle configuration
BUNDLE_JSON=$(databricks bundle -t "$TARGET" validate --output json)

# Extract catalog and schema from resources
APP_CATALOG=$(echo "$BUNDLE_JSON" | jq -r '.resources.schemas["process-management"].catalog_name // empty')
APP_SCHEMA=$(echo "$BUNDLE_JSON" | jq -r '.resources.schemas["process-management"].name // empty')

# Get fields_username from bundle variables
APP_FIELDS_USERNAME=$(echo "$BUNDLE_JSON" | jq -r '.variables.fields_username.value // empty')

# Validate required values are not empty - fail if any are missing
if [ -z "$APP_CATALOG" ] || [ -z "$APP_SCHEMA" ] || [ -z "$APP_FIELDS_USERNAME" ]; then
  echo "Error: Failed to extract required values from bundle configuration" >&2
  echo "APP_CATALOG: '$APP_CATALOG'" >&2
  echo "APP_SCHEMA: '$APP_SCHEMA'" >&2
  echo "APP_FIELDS_USERNAME: '$APP_FIELDS_USERNAME'" >&2
  exit 1
fi

# Export for envsubst
export APP_CATALOG APP_SCHEMA APP_FIELDS_USERNAME

# Generate app.yaml from template using envsubst
envsubst < apps/unit_breakdown/app.yaml.template > apps/unit_breakdown/app.yaml

echo "Generated apps/unit_breakdown/app.yaml for target $TARGET"
echo "  APP_CATALOG: $APP_CATALOG"
echo "  APP_SCHEMA: $APP_SCHEMA"
echo "  APP_FIELDS_USERNAME: $APP_FIELDS_USERNAME"

There are probably ways to do this in a generic way, but it was good enough to get me unstuck. Just an idea for anyone to use who comes across this issue, until this feature is available in the cli.