-
Notifications
You must be signed in to change notification settings - Fork 136
direct: support quality monitors #4278
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
c6d641b
direct: add support for quality_monitors
denik 24766ad
update
denik b03a6da
update create test
denik 63dd220
update tests
denik 28d7c0a
udate
denik cdf469c
clean up
denik 1ff26ec
update_assets_dir
denik 04a5b89
clean up
denik 13ddde6
add change_table_name
denik c270618
update
denik b20dd2e
clean up
denik 1b5cab1
clean up
denik f7b9fa9
update
denik 28b05ec
update
denik dd7a24d
update changelog
denik a05c26f
disable on terraform
denik 0443c7f
update out.fields.txt
denik 76f4ec7
rewrite script to use CLI so that it does not depends on sdk installed
denik 438ed49
update
denik 0745ab1
clean up
denik d33c7e5
clean up tests
denik c555af3
clean up
denik 906aed8
update test output
denik File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| #!/usr/bin/env python3 | ||
| import json | ||
| import os | ||
| import subprocess | ||
| import sys | ||
| import time | ||
|
|
||
| table_name = sys.argv[1] | ||
| # Extract catalog.schema from table_name | ||
| parts = table_name.split(".") | ||
| if len(parts) != 3: | ||
| print(f"Invalid table name: {table_name}. Expected format: catalog.schema.table", file=sys.stderr) | ||
| sys.exit(1) | ||
|
|
||
| catalog_name = parts[0] | ||
| schema_name = parts[1] | ||
| full_schema_name = f"{catalog_name}.{schema_name}" | ||
|
|
||
| cli = os.environ.get("CLI", "databricks") | ||
|
|
||
|
|
||
| def run_cli(*args): | ||
| result = subprocess.run([cli, *args], capture_output=True, text=True) | ||
| return result | ||
|
|
||
|
|
||
| def execute_sql(warehouse_id, sql): | ||
| """Execute SQL using the API endpoint.""" | ||
| payload = json.dumps({"warehouse_id": warehouse_id, "statement": sql, "wait_timeout": "30s"}) | ||
| return run_cli("api", "post", "/api/2.0/sql/statements/", "--json", payload) | ||
|
|
||
|
|
||
| # Get warehouse ID from environment variable | ||
| warehouse_id = os.environ["TEST_DEFAULT_WAREHOUSE_ID"] | ||
|
|
||
| # Create a simple table | ||
| sql = f"CREATE TABLE IF NOT EXISTS {table_name} (id INT, value STRING, timestamp TIMESTAMP) USING DELTA" | ||
|
|
||
| result = execute_sql(warehouse_id, sql) | ||
| if result.returncode != 0: | ||
| print(f"Failed to create table: {result.stderr}", file=sys.stderr) | ||
| sys.exit(1) | ||
|
|
||
| print(f"Created table {table_name}") | ||
|
|
||
| # Insert some sample data so the monitor has something to analyze | ||
| insert_sql = f"""INSERT INTO {table_name} VALUES | ||
| (1, 'test1', current_timestamp()), | ||
| (2, 'test2', current_timestamp()), | ||
| (3, 'test3', current_timestamp())""" | ||
|
|
||
| result = execute_sql(warehouse_id, insert_sql) | ||
| if result.returncode != 0: | ||
| print(f"Failed to insert data: {result.stderr}", file=sys.stderr) | ||
| sys.exit(1) | ||
|
|
||
| print(f"Inserted sample data into {table_name}") | ||
|
|
||
| # Wait for table to be visible in Unity Catalog | ||
| for attempt in range(10): | ||
| result = run_cli("tables", "get", table_name) | ||
| if result.returncode == 0: | ||
| table_info = json.loads(result.stdout) | ||
| print(f"Table {table_name} is now visible (catalog_name={table_info.get('catalog_name')})") | ||
| break | ||
| if attempt < 9: | ||
| time.sleep(1) | ||
| else: | ||
| print(f"Warning: Table may not be immediately visible: {result.stderr}", file=sys.stderr) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,6 @@ | ||
| Local = true | ||
| Cloud = false | ||
|
|
||
| [[Repls]] | ||
| Old = '"dashboard_id":"[0-9a-f]+",' | ||
| New = '"dashboard_id":"(redacted)",' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
11 changes: 11 additions & 0 deletions
11
acceptance/bundle/resources/quality_monitors/change_assets_dir/databricks.yml.tmpl
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| bundle: | ||
| name: quality-monitor-update-$UNIQUE_NAME | ||
|
|
||
| resources: | ||
| quality_monitors: | ||
| monitor1: | ||
| table_name: main.qm_test_$UNIQUE_NAME.test_table | ||
| assets_dir: /Workspace/Users/$CURRENT_USER_NAME/monitor_assets_$UNIQUE_NAME | ||
| output_schema_name: main.qm_test_$UNIQUE_NAME | ||
| snapshot: {} | ||
| warehouse_id: $TEST_DEFAULT_WAREHOUSE_ID |
6 changes: 6 additions & 0 deletions
6
acceptance/bundle/resources/quality_monitors/change_assets_dir/out.deploy.direct.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
|
|
||
| >>> errcode [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/quality-monitor-update-[UNIQUE_NAME]/default/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! |
14 changes: 14 additions & 0 deletions
14
...tance/bundle/resources/quality_monitors/change_assets_dir/out.deploy.requests.direct.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| { | ||
| "method": "DELETE", | ||
| "path": "/api/2.1/unity-catalog/tables/main.qm_test_[UNIQUE_NAME].test_table/monitor" | ||
| } | ||
| { | ||
| "method": "POST", | ||
| "path": "/api/2.1/unity-catalog/tables/main.qm_test_[UNIQUE_NAME].test_table/monitor", | ||
| "body": { | ||
| "assets_dir": "/Workspace/Users/[USERNAME]/monitor_assets2_[UNIQUE_NAME]", | ||
| "output_schema_name": "main.qm_test_[UNIQUE_NAME]", | ||
| "snapshot": {}, | ||
| "warehouse_id": "[TEST_DEFAULT_WAREHOUSE_ID]" | ||
| } | ||
| } | ||
9 changes: 9 additions & 0 deletions
9
...ce/bundle/resources/quality_monitors/change_assets_dir/out.deploy.requests.terraform.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| { | ||
| "method": "PUT", | ||
| "path": "/api/2.1/unity-catalog/tables/main.qm_test_[UNIQUE_NAME].test_table/monitor", | ||
| "body": { | ||
| "dashboard_id": "(redacted)", | ||
| "output_schema_name": "main.qm_test_[UNIQUE_NAME]", | ||
| "snapshot": {} | ||
| } | ||
| } |
22 changes: 22 additions & 0 deletions
22
acceptance/bundle/resources/quality_monitors/change_assets_dir/out.deploy.terraform.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
|
|
||
| >>> errcode [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/quality-monitor-update-[UNIQUE_NAME]/default/files... | ||
| Deploying resources... | ||
| Error: terraform apply: exit status 1 | ||
|
|
||
| Error: Provider produced inconsistent result after apply | ||
|
|
||
| When applying changes to databricks_quality_monitor.monitor1, provider | ||
| "provider[\"registry.terraform.io/databricks/databricks\"]" produced an | ||
| unexpected new value: .assets_dir: was | ||
| cty.StringVal("/Workspace/Users/[USERNAME]/monitor_assets2_[UNIQUE_NAME]"), | ||
| but now | ||
| cty.StringVal("/Workspace/Users/[USERNAME]/monitor_assets_[UNIQUE_NAME]"). | ||
|
|
||
| This is a bug in the provider, which should be reported in the provider's own | ||
| issue tracker. | ||
|
|
||
|
|
||
| Updating deployment state... | ||
|
|
||
| Exit code: 1 |
48 changes: 48 additions & 0 deletions
48
acceptance/bundle/resources/quality_monitors/change_assets_dir/out.plan.direct.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
|
|
||
| >>> errcode [CLI] bundle plan -o json | ||
| { | ||
| "plan_version": 2, | ||
| "cli_version": "[DEV_VERSION]", | ||
| "lineage": "[UUID]", | ||
| "serial": 1, | ||
| "plan": { | ||
| "resources.quality_monitors.monitor1": { | ||
| "action": "recreate", | ||
| "new_state": { | ||
| "value": { | ||
| "assets_dir": "/Workspace/Users/[USERNAME]/monitor_assets2_[UNIQUE_NAME]", | ||
| "output_schema_name": "main.qm_test_[UNIQUE_NAME]", | ||
| "snapshot": {}, | ||
| "table_name": "main.qm_test_[UNIQUE_NAME].test_table", | ||
| "warehouse_id": "[TEST_DEFAULT_WAREHOUSE_ID]" | ||
| } | ||
| }, | ||
| "remote_state": { | ||
| "assets_dir": "/Workspace/Users/[USERNAME]/monitor_assets_[UNIQUE_NAME]", | ||
| "dashboard_id": "(redacted)", | ||
| "drift_metrics_table_name": "main.qm_test_[UNIQUE_NAME].test_table_drift_metrics", | ||
| "monitor_version": 0, | ||
| "output_schema_name": "main.qm_test_[UNIQUE_NAME]", | ||
| "profile_metrics_table_name": "main.qm_test_[UNIQUE_NAME].test_table_profile_metrics", | ||
| "snapshot": {}, | ||
| "status": "MONITOR_STATUS_ACTIVE", | ||
| "table_name": "main.qm_test_[UNIQUE_NAME].test_table" | ||
| }, | ||
| "changes": { | ||
| "assets_dir": { | ||
| "action": "recreate", | ||
| "reason": "field_triggers", | ||
| "old": "/Workspace/Users/[USERNAME]/monitor_assets_[UNIQUE_NAME]", | ||
| "new": "/Workspace/Users/[USERNAME]/monitor_assets2_[UNIQUE_NAME]", | ||
| "remote": "/Workspace/Users/[USERNAME]/monitor_assets_[UNIQUE_NAME]" | ||
| }, | ||
| "warehouse_id": { | ||
| "action": "skip", | ||
| "reason": "config_only", | ||
| "old": "[TEST_DEFAULT_WAREHOUSE_ID]", | ||
| "new": "[TEST_DEFAULT_WAREHOUSE_ID]" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } |
5 changes: 5 additions & 0 deletions
5
acceptance/bundle/resources/quality_monitors/change_assets_dir/out.plan.direct.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
|
|
||
| >>> errcode [CLI] bundle plan | ||
| recreate quality_monitors.monitor1 | ||
|
|
||
| Plan: 1 to add, 0 to change, 1 to delete, 0 unchanged |
11 changes: 11 additions & 0 deletions
11
acceptance/bundle/resources/quality_monitors/change_assets_dir/out.plan.terraform.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
|
|
||
| >>> errcode [CLI] bundle plan -o json | ||
| { | ||
| "plan_version": 2, | ||
| "cli_version": "[DEV_VERSION]", | ||
| "plan": { | ||
| "resources.quality_monitors.monitor1": { | ||
| "action": "update" | ||
| } | ||
| } | ||
| } |
5 changes: 5 additions & 0 deletions
5
acceptance/bundle/resources/quality_monitors/change_assets_dir/out.plan.terraform.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
|
|
||
| >>> errcode [CLI] bundle plan | ||
| update quality_monitors.monitor1 | ||
|
|
||
| Plan: 0 to add, 1 to change, 0 to delete, 0 unchanged |
3 changes: 3 additions & 0 deletions
3
...ance/bundle/resources/quality_monitors/change_assets_dir/out.plan_after_deploy.direct.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
|
|
||
| >>> errcode [CLI] bundle plan | ||
| Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged |
5 changes: 5 additions & 0 deletions
5
...e/bundle/resources/quality_monitors/change_assets_dir/out.plan_after_deploy.terraform.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
|
|
||
| >>> errcode [CLI] bundle plan | ||
| update quality_monitors.monitor1 | ||
|
|
||
| Plan: 0 to add, 1 to change, 0 to delete, 0 unchanged |
6 changes: 6 additions & 0 deletions
6
acceptance/bundle/resources/quality_monitors/change_assets_dir/out.test.toml
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
35 changes: 35 additions & 0 deletions
35
acceptance/bundle/resources/quality_monitors/change_assets_dir/output.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
|
|
||
| >>> [CLI] schemas create qm_test_[UNIQUE_NAME] main -o json | ||
| { | ||
| "full_name": "main.qm_test_[UNIQUE_NAME]" | ||
| } | ||
|
|
||
| >>> [CLI] schemas create qm_test_[UNIQUE_NAME]_2 main -o json | ||
| { | ||
| "full_name": "main.qm_test_[UNIQUE_NAME]_2" | ||
| } | ||
|
|
||
| >>> create_table.py main.qm_test_[UNIQUE_NAME].test_table | ||
| Created table main.qm_test_[UNIQUE_NAME].test_table | ||
| Inserted sample data into main.qm_test_[UNIQUE_NAME].test_table | ||
| Table main.qm_test_[UNIQUE_NAME].test_table is now visible (catalog_name=main) | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/quality-monitor-update-[UNIQUE_NAME]/default/files... | ||
| Deploying resources... | ||
| Updating deployment state... | ||
| Deployment complete! | ||
|
|
||
| >>> [CLI] bundle plan | ||
| Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged | ||
|
|
||
| >>> print_requests.py ^//import-file/ ^//workspace/ ^//telemetry-ext | ||
|
|
||
| >>> [CLI] bundle destroy --auto-approve | ||
| The following resources will be deleted: | ||
| delete resources.quality_monitors.monitor1 | ||
|
|
||
| All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/quality-monitor-update-[UNIQUE_NAME]/default | ||
|
|
||
| Deleting files... | ||
| Destroy complete! |
30 changes: 30 additions & 0 deletions
30
acceptance/bundle/resources/quality_monitors/change_assets_dir/script
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| SCHEMA_NAME="main.qm_test_${UNIQUE_NAME}" | ||
| TABLE_NAME="${SCHEMA_NAME}.test_table" | ||
|
|
||
| trace $CLI schemas create "qm_test_${UNIQUE_NAME}" main -o json | jq '{full_name}' | ||
| trace $CLI schemas create "qm_test_${UNIQUE_NAME}_2" main -o json | jq '{full_name}' | ||
| trace create_table.py "$TABLE_NAME" | ||
|
|
||
| cleanup() { | ||
| trace $CLI bundle destroy --auto-approve | ||
| $CLI schemas delete "$SCHEMA_NAME" --force 2>/dev/null || true | ||
| $CLI schemas delete "qm_test_${UNIQUE_NAME}_2" --force 2>/dev/null || true | ||
| rm -f out.requests.txt | ||
| } | ||
| trap cleanup EXIT | ||
|
|
||
| envsubst < databricks.yml.tmpl > databricks.yml | ||
|
|
||
| trace $CLI bundle deploy | ||
|
|
||
| trace $CLI bundle plan | contains.py "1 unchanged" | ||
|
|
||
| update_file.py databricks.yml "assets_dir: /Workspace/Users/$CURRENT_USER_NAME/monitor_assets_$UNIQUE_NAME" "assets_dir: /Workspace/Users/$CURRENT_USER_NAME/monitor_assets2_$UNIQUE_NAME" | ||
|
|
||
| trace errcode $CLI bundle plan &> out.plan.$DATABRICKS_BUNDLE_ENGINE.txt | ||
| trace errcode $CLI bundle plan -o json &> out.plan.$DATABRICKS_BUNDLE_ENGINE.json | ||
|
|
||
| rm out.requests.txt | ||
| trace errcode $CLI bundle deploy &> out.deploy.$DATABRICKS_BUNDLE_ENGINE.txt | ||
| trace print_requests.py '^//import-file/' '^//workspace/' '^//telemetry-ext' > out.deploy.requests.$DATABRICKS_BUNDLE_ENGINE.json | ||
| trace errcode $CLI bundle plan &> out.plan_after_deploy.$DATABRICKS_BUNDLE_ENGINE.txt |
11 changes: 11 additions & 0 deletions
11
acceptance/bundle/resources/quality_monitors/change_output_schema_name/databricks.yml.tmpl
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| bundle: | ||
| name: quality-monitor-update-$UNIQUE_NAME | ||
|
|
||
| resources: | ||
| quality_monitors: | ||
| monitor1: | ||
| table_name: main.qm_test_$UNIQUE_NAME.test_table | ||
| assets_dir: /Workspace/Users/$CURRENT_USER_NAME/monitor_assets_$UNIQUE_NAME | ||
| output_schema_name: main.qm_test_$UNIQUE_NAME | ||
| snapshot: {} | ||
| warehouse_id: $TEST_DEFAULT_WAREHOUSE_ID |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like the update now causes a recreation instead of the actual update, which leads to a quality monitor URL / id change. This URL / ID might be referenced from other places (outside of bundle scope, like let's say some dashboard)
Is this recreation intentional?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, because assets_dir is not available on update request for quality monitors, so it simply cannot be updated.
https://docs.databricks.com/api/workspace/qualitymonitors/update