Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test failure: test_running_real_assessment_job_ext_hms #1616

Closed
github-actions bot opened this issue May 3, 2024 · 7 comments
Closed

Test failure: test_running_real_assessment_job_ext_hms #1616

github-actions bot opened this issue May 3, 2024 · 7 comments
Assignees
@JCZuurmond

Comments

@github-actions
Copy link

github-actions bot commented May 3, 2024

❌ test_running_real_assessment_job_ext_hms: KeyError: 'ucx_tnIh' (25m7.398s) 
KeyError: 'ucx_tnIh'
[gw8] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-jwad@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-jwad@example.com')], entitlements=[], external_id=None, groups=[], id='3594324638715029', name=Name(family_name=None, given_name='sdk-jwad@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-jwad@example.com')
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_tnIh: https://DATABRICKS_HOST#setting/accounts/groups/373244128321231
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_tnIh', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='373244128321231', members=[ComplexValue(display='sdk-jwad@example.com', primary=None, ref='Users/3594324638715029', type=None, value='3594324638715029')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_tnIh: https://accounts.CLOUD_ENVdatabricks.net/users/groups/1001775630102736/members
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_tnIh', entitlements=[], external_id=None, groups=[], id='1001775630102736', members=[ComplexValue(display='sdk-jwad@example.com', primary=None, ref='Users/3594324638715029', type=None, value='3594324638715029')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/000BA07CC000327C
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='000BA07CC000327C')
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 000BA07CC000327C [group_name admins CAN_USE] -> [group_name ucx_tnIh CAN_USE]
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sxsox
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sxsox: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sxsox
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sxsox', metastore_id=None, name='ucx_sxsox', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:15 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.gE8a/config.yml) doesn't exist.
07:15 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:15 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:15 INFO [databricks.labs.ucx.install] Fetching installations...
07:15 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:15 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:15 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:15 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:15 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+2720240503071553
07:15 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sxsox
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.external_locations (location STRING NOT NULL... (40 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.udfs (catalog STRING NOT NULL, database STRI... (239 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sxsox.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sxsox.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sxsox.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sxsox.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sxsox.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:17 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.gE8a/README for the next steps.
07:17 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/552932542669137
07:30 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/552932542669137
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-jwad@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-jwad@example.com')], entitlements=[], external_id=None, groups=[], id='3594324638715029', name=Name(family_name=None, given_name='sdk-jwad@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-jwad@example.com')
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_tnIh: https://DATABRICKS_HOST#setting/accounts/groups/373244128321231
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_tnIh', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='373244128321231', members=[ComplexValue(display='sdk-jwad@example.com', primary=None, ref='Users/3594324638715029', type=None, value='3594324638715029')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_tnIh: https://accounts.CLOUD_ENVdatabricks.net/users/groups/1001775630102736/members
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_tnIh', entitlements=[], external_id=None, groups=[], id='1001775630102736', members=[ComplexValue(display='sdk-jwad@example.com', primary=None, ref='Users/3594324638715029', type=None, value='3594324638715029')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/000BA07CC000327C
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='000BA07CC000327C')
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 000BA07CC000327C [group_name admins CAN_USE] -> [group_name ucx_tnIh CAN_USE]
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sxsox
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sxsox: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sxsox
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sxsox', metastore_id=None, name='ucx_sxsox', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:15 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.gE8a/config.yml) doesn't exist.
07:15 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:15 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:15 INFO [databricks.labs.ucx.install] Fetching installations...
07:15 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:15 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:15 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:15 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:15 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+2720240503071553
07:15 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sxsox
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.external_locations (location STRING NOT NULL... (40 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.udfs (catalog STRING NOT NULL, database STRI... (239 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sxsox.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sxsox.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sxsox.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sxsox.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sxsox.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sxsox.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:17 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.gE8a/README for the next steps.
07:17 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/552932542669137
07:30 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/552932542669137
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 000BA07CC000327C [group_name admins CAN_USE] -> [group_name ucx_tnIh CAN_USE]
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='000BA07CC000327C')
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while cluster policy CreatePolicyResponse(policy_id='000BA07CC000327C') teardown: Can't find a cluster policy with id: 000BA07CC000327C.
07:30 INFO [databricks.labs.ucx.install] Deleting UCX v0.22.1+2720240503073003 from https://DATABRICKS_HOST
07:30 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sxsox
07:30 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sxsox CASCADE
07:30 INFO [databricks.labs.ucx.install] Deleting jobs
07:30 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=1068898461395856.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=277236911283661.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=1083879690919004.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=450536702360503.
07:30 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=904346105919442.
07:30 INFO [databricks.labs.ucx.install] Deleting failing job_id=752593203937251.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=1092339076953781.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=1034103158179352.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=411333076634864.
07:30 INFO [databricks.labs.ucx.install] Deleting assessment job_id=552932542669137.
07:30 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:30 INFO [databricks.labs.ucx.install] Deleting secret scope
07:30 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='sdk-jwad@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-jwad@example.com')], entitlements=[], external_id=None, groups=[], id='3594324638715029', name=Name(family_name=None, given_name='sdk-jwad@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-jwad@example.com')
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace user User(active=True, display_name='sdk-jwad@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-jwad@example.com')], entitlements=[], external_id=None, groups=[], id='3594324638715029', name=Name(family_name=None, given_name='sdk-jwad@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-jwad@example.com') teardown: None User with id 3594324638715029 not found.
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_tnIh', entitlements=[], external_id=None, groups=[], id='1001775630102736', members=[ComplexValue(display='sdk-jwad@example.com', primary=None, ref='Users/3594324638715029', type=None, value='3594324638715029')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_tnIh', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='373244128321231', members=[ComplexValue(display='sdk-jwad@example.com', primary=None, ref='Users/3594324638715029', type=None, value='3594324638715029')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace group Group(display_name='ucx_tnIh', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='373244128321231', members=[ComplexValue(display='sdk-jwad@example.com', primary=None, ref='Users/3594324638715029', type=None, value='3594324638715029')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]) teardown: None Group with id 373244128321231 not found.
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sxsox', metastore_id=None, name='ucx_sxsox', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:30 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sxsox CASCADE
[gw8] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

Running from nightly #49
@github-actions github-actions bot added the bug label May 3, 2024
@github-actions GitHub Actions
Copy link
Author

github-actions bot commented May 4, 2024

❌ test_running_real_assessment_job_ext_hms: KeyError: 'ucx_ORS8' (25m36.24s) 
KeyError: 'ucx_ORS8'
[gw8] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-unve@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-unve@example.com')], entitlements=[], external_id=None, groups=[], id='49835715119411', name=Name(family_name=None, given_name='sdk-unve@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-unve@example.com')
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_ORS8: https://DATABRICKS_HOST#setting/accounts/groups/282498917932873
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_ORS8', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='282498917932873', members=[ComplexValue(display='sdk-unve@example.com', primary=None, ref='Users/49835715119411', type=None, value='49835715119411')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_ORS8: https://accounts.CLOUD_ENVdatabricks.net/users/groups/903899026305652/members
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_ORS8', entitlements=[], external_id=None, groups=[], id='903899026305652', members=[ComplexValue(display='sdk-unve@example.com', primary=None, ref='Users/49835715119411', type=None, value='49835715119411')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/00183E02C8D02B37
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='00183E02C8D02B37')
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 00183E02C8D02B37 [group_name admins CAN_USE] -> [group_name ucx_ORS8 CAN_USE]
07:04 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sypx9
07:16 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sypx9: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sypx9
07:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sypx9', metastore_id=None, name='ucx_sypx9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:16 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.neCs/config.yml) doesn't exist.
07:16 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:16 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:16 INFO [databricks.labs.ucx.install] Fetching installations...
07:16 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:16 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:16 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+3220240504071657
07:16 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sypx9
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.external_locations (location STRING NOT NULL... (40 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.udfs (catalog STRING NOT NULL, database STRI... (239 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:18 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.neCs/README for the next steps.
07:18 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/1056246959260187
07:30 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/1056246959260187
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-unve@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-unve@example.com')], entitlements=[], external_id=None, groups=[], id='49835715119411', name=Name(family_name=None, given_name='sdk-unve@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-unve@example.com')
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_ORS8: https://DATABRICKS_HOST#setting/accounts/groups/282498917932873
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_ORS8', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='282498917932873', members=[ComplexValue(display='sdk-unve@example.com', primary=None, ref='Users/49835715119411', type=None, value='49835715119411')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_ORS8: https://accounts.CLOUD_ENVdatabricks.net/users/groups/903899026305652/members
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_ORS8', entitlements=[], external_id=None, groups=[], id='903899026305652', members=[ComplexValue(display='sdk-unve@example.com', primary=None, ref='Users/49835715119411', type=None, value='49835715119411')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/00183E02C8D02B37
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='00183E02C8D02B37')
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 00183E02C8D02B37 [group_name admins CAN_USE] -> [group_name ucx_ORS8 CAN_USE]
07:04 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sypx9
07:16 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sypx9: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sypx9
07:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sypx9', metastore_id=None, name='ucx_sypx9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:16 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.neCs/config.yml) doesn't exist.
07:16 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:16 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:16 INFO [databricks.labs.ucx.install] Fetching installations...
07:16 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:16 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:16 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+3220240504071657
07:16 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sypx9
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.external_locations (location STRING NOT NULL... (40 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.udfs (catalog STRING NOT NULL, database STRI... (239 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sypx9.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sypx9.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:18 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.neCs/README for the next steps.
07:18 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/1056246959260187
07:30 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/1056246959260187
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 00183E02C8D02B37 [group_name admins CAN_USE] -> [group_name ucx_ORS8 CAN_USE]
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='00183E02C8D02B37')
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while cluster policy CreatePolicyResponse(policy_id='00183E02C8D02B37') teardown: Can't find a cluster policy with id: 00183E02C8D02B37.
07:30 INFO [databricks.labs.ucx.install] Deleting UCX v0.22.1+3220240504073003 from https://DATABRICKS_HOST
07:30 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sypx9
07:30 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sypx9 CASCADE
07:30 INFO [databricks.labs.ucx.install] Deleting jobs
07:30 INFO [databricks.labs.ucx.install] Deleting failing job_id=587841249730879.
07:30 INFO [databricks.labs.ucx.install] Deleting assessment job_id=1056246959260187.
07:30 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=409633535547014.
07:30 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=199580646083585.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=912505395576906.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=174812062070925.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=1387661723379.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=80407497854529.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=382570426554417.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=75205767086363.
07:30 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:30 INFO [databricks.labs.ucx.install] Deleting secret scope
07:30 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='sdk-unve@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-unve@example.com')], entitlements=[], external_id=None, groups=[], id='49835715119411', name=Name(family_name=None, given_name='sdk-unve@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-unve@example.com')
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace user User(active=True, display_name='sdk-unve@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-unve@example.com')], entitlements=[], external_id=None, groups=[], id='49835715119411', name=Name(family_name=None, given_name='sdk-unve@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-unve@example.com') teardown: None User with id 49835715119411 not found.
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_ORS8', entitlements=[], external_id=None, groups=[], id='903899026305652', members=[ComplexValue(display='sdk-unve@example.com', primary=None, ref='Users/49835715119411', type=None, value='49835715119411')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_ORS8', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='282498917932873', members=[ComplexValue(display='sdk-unve@example.com', primary=None, ref='Users/49835715119411', type=None, value='49835715119411')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace group Group(display_name='ucx_ORS8', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='282498917932873', members=[ComplexValue(display='sdk-unve@example.com', primary=None, ref='Users/49835715119411', type=None, value='49835715119411')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]) teardown: None Group with id 282498917932873 not found.
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sypx9', metastore_id=None, name='ucx_sypx9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:30 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sypx9 CASCADE
[gw8] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

Running from nightly #50

@github-actions GitHub Actions
Copy link
Author

github-actions bot commented May 5, 2024

❌ test_running_real_assessment_job_ext_hms: KeyError: 'ucx_wWSV' (24m44.187s) 
KeyError: 'ucx_wWSV'
[gw9] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-n7ts@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-n7ts@example.com')], entitlements=[], external_id=None, groups=[], id='3753449921302703', name=Name(family_name=None, given_name='sdk-n7ts@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-n7ts@example.com')
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_wWSV: https://DATABRICKS_HOST#setting/accounts/groups/599730207407029
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_wWSV', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='599730207407029', members=[ComplexValue(display='sdk-n7ts@example.com', primary=None, ref='Users/3753449921302703', type=None, value='3753449921302703')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_wWSV: https://accounts.CLOUD_ENVdatabricks.net/users/groups/806626359434077/members
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_wWSV', entitlements=[], external_id=None, groups=[], id='806626359434077', members=[ComplexValue(display='sdk-n7ts@example.com', primary=None, ref='Users/3753449921302703', type=None, value='3753449921302703')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/001342C3CE3F5A01
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='001342C3CE3F5A01')
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 001342C3CE3F5A01 [group_name admins CAN_USE] -> [group_name ucx_wWSV CAN_USE]
07:04 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sjzkr
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sjzkr: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sjzkr
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sjzkr', metastore_id=None, name='ucx_sjzkr', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:15 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Dg3s/config.yml) doesn't exist.
07:15 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:15 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:15 INFO [databricks.labs.ucx.install] Fetching installations...
07:15 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:15 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:15 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:15 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:15 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+3220240505071556
07:15 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sjzkr
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.external_locations (location STRING NOT NULL... (40 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.udfs (catalog STRING NOT NULL, database STRI... (239 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:17 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Dg3s/README for the next steps.
07:17 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/255392742700621
07:28 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/255392742700621
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-n7ts@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-n7ts@example.com')], entitlements=[], external_id=None, groups=[], id='3753449921302703', name=Name(family_name=None, given_name='sdk-n7ts@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-n7ts@example.com')
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_wWSV: https://DATABRICKS_HOST#setting/accounts/groups/599730207407029
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_wWSV', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='599730207407029', members=[ComplexValue(display='sdk-n7ts@example.com', primary=None, ref='Users/3753449921302703', type=None, value='3753449921302703')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_wWSV: https://accounts.CLOUD_ENVdatabricks.net/users/groups/806626359434077/members
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_wWSV', entitlements=[], external_id=None, groups=[], id='806626359434077', members=[ComplexValue(display='sdk-n7ts@example.com', primary=None, ref='Users/3753449921302703', type=None, value='3753449921302703')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:04 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/001342C3CE3F5A01
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='001342C3CE3F5A01')
07:04 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 001342C3CE3F5A01 [group_name admins CAN_USE] -> [group_name ucx_wWSV CAN_USE]
07:04 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sjzkr
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sjzkr: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sjzkr
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sjzkr', metastore_id=None, name='ucx_sjzkr', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:15 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Dg3s/config.yml) doesn't exist.
07:15 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:15 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:15 INFO [databricks.labs.ucx.install] Fetching installations...
07:15 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:15 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:15 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:15 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:15 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+3220240505071556
07:15 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sjzkr
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.external_locations (location STRING NOT NULL... (40 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:15 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.udfs (catalog STRING NOT NULL, database STRI... (239 more bytes)
07:16 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sjzkr.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:17 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sjzkr.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:17 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Dg3s/README for the next steps.
07:17 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/255392742700621
07:28 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/255392742700621
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 001342C3CE3F5A01 [group_name admins CAN_USE] -> [group_name ucx_wWSV CAN_USE]
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='001342C3CE3F5A01')
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while cluster policy CreatePolicyResponse(policy_id='001342C3CE3F5A01') teardown: Can't find a cluster policy with id: 001342C3CE3F5A01.
07:28 INFO [databricks.labs.ucx.install] Deleting UCX v0.22.1+3220240505072853 from https://DATABRICKS_HOST
07:28 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sjzkr
07:28 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sjzkr CASCADE
07:28 INFO [databricks.labs.ucx.install] Deleting jobs
07:28 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=803662989067337.
07:28 INFO [databricks.labs.ucx.install] Deleting assessment job_id=255392742700621.
07:28 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=574141874107506.
07:28 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=828472765882381.
07:28 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=596970312271364.
07:28 INFO [databricks.labs.ucx.install] Deleting failing job_id=125935612576049.
07:28 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=235375964036838.
07:28 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=369025870963211.
07:28 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=127488673308845.
07:28 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=175964184317291.
07:28 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:28 INFO [databricks.labs.ucx.install] Deleting secret scope
07:28 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='sdk-n7ts@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-n7ts@example.com')], entitlements=[], external_id=None, groups=[], id='3753449921302703', name=Name(family_name=None, given_name='sdk-n7ts@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-n7ts@example.com')
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace user User(active=True, display_name='sdk-n7ts@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-n7ts@example.com')], entitlements=[], external_id=None, groups=[], id='3753449921302703', name=Name(family_name=None, given_name='sdk-n7ts@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-n7ts@example.com') teardown: None User with id 3753449921302703 not found.
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
07:28 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_wWSV', entitlements=[], external_id=None, groups=[], id='806626359434077', members=[ComplexValue(display='sdk-n7ts@example.com', primary=None, ref='Users/3753449921302703', type=None, value='3753449921302703')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
07:29 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_wWSV', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='599730207407029', members=[ComplexValue(display='sdk-n7ts@example.com', primary=None, ref='Users/3753449921302703', type=None, value='3753449921302703')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:29 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace group Group(display_name='ucx_wWSV', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='599730207407029', members=[ComplexValue(display='sdk-n7ts@example.com', primary=None, ref='Users/3753449921302703', type=None, value='3753449921302703')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]) teardown: None Group with id 599730207407029 not found.
07:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:29 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
07:29 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sjzkr', metastore_id=None, name='ucx_sjzkr', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:29 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sjzkr CASCADE
[gw9] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

Running from nightly #51

@github-actions GitHub Actions
Copy link
Author

github-actions bot commented May 6, 2024

❌ test_running_real_assessment_job_ext_hms: KeyError: 'ucx_D5jw' (25m26.539s) 
KeyError: 'ucx_D5jw'
[gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-3jwh@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-3jwh@example.com')], entitlements=[], external_id=None, groups=[], id='5593186882414224', name=Name(family_name=None, given_name='sdk-3jwh@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-3jwh@example.com')
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_D5jw: https://DATABRICKS_HOST#setting/accounts/groups/430512368140466
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_D5jw', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='430512368140466', members=[ComplexValue(display='sdk-3jwh@example.com', primary=None, ref='Users/5593186882414224', type=None, value='5593186882414224')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_D5jw: https://accounts.CLOUD_ENVdatabricks.net/users/groups/363376592364516/members
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_D5jw', entitlements=[], external_id=None, groups=[], id='363376592364516', members=[ComplexValue(display='sdk-3jwh@example.com', primary=None, ref='Users/5593186882414224', type=None, value='5593186882414224')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/000796608048429E
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='000796608048429E')
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 000796608048429E [group_name admins CAN_USE] -> [group_name ucx_D5jw CAN_USE]
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_smrhe
07:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_smrhe: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_smrhe
07:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_smrhe', metastore_id=None, name='ucx_smrhe', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:18 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ckVZ/config.yml) doesn't exist.
07:18 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:18 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:18 INFO [databricks.labs.ucx.install] Fetching installations...
07:18 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:18 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+3220240506071809
07:18 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_smrhe
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.external_locations (location STRING NOT NULL... (40 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.udfs (catalog STRING NOT NULL, database STRI... (239 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:20 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ckVZ/README for the next steps.
07:20 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/1020303679279872
07:30 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/1020303679279872
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-3jwh@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-3jwh@example.com')], entitlements=[], external_id=None, groups=[], id='5593186882414224', name=Name(family_name=None, given_name='sdk-3jwh@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-3jwh@example.com')
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_D5jw: https://DATABRICKS_HOST#setting/accounts/groups/430512368140466
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_D5jw', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='430512368140466', members=[ComplexValue(display='sdk-3jwh@example.com', primary=None, ref='Users/5593186882414224', type=None, value='5593186882414224')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_D5jw: https://accounts.CLOUD_ENVdatabricks.net/users/groups/363376592364516/members
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_D5jw', entitlements=[], external_id=None, groups=[], id='363376592364516', members=[ComplexValue(display='sdk-3jwh@example.com', primary=None, ref='Users/5593186882414224', type=None, value='5593186882414224')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/000796608048429E
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='000796608048429E')
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 000796608048429E [group_name admins CAN_USE] -> [group_name ucx_D5jw CAN_USE]
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_smrhe
07:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_smrhe: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_smrhe
07:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_smrhe', metastore_id=None, name='ucx_smrhe', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:18 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ckVZ/config.yml) doesn't exist.
07:18 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:18 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:18 INFO [databricks.labs.ucx.install] Fetching installations...
07:18 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:18 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+3220240506071809
07:18 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_smrhe
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.external_locations (location STRING NOT NULL... (40 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.udfs (catalog STRING NOT NULL, database STRI... (239 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_smrhe.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_smrhe.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:20 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ckVZ/README for the next steps.
07:20 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/1020303679279872
07:30 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/1020303679279872
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 000796608048429E [group_name admins CAN_USE] -> [group_name ucx_D5jw CAN_USE]
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='000796608048429E')
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while cluster policy CreatePolicyResponse(policy_id='000796608048429E') teardown: Can't find a cluster policy with id: 000796608048429E.
07:30 INFO [databricks.labs.ucx.install] Deleting UCX v0.22.1+3220240506073023 from https://DATABRICKS_HOST
07:30 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_smrhe
07:30 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_smrhe CASCADE
07:30 INFO [databricks.labs.ucx.install] Deleting jobs
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=647769373469020.
07:30 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=985369021765508.
07:30 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=412118750744371.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=822388136460936.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=1088464868943184.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=76918293174651.
07:30 INFO [databricks.labs.ucx.install] Deleting failing job_id=839658168553027.
07:30 INFO [databricks.labs.ucx.install] Deleting assessment job_id=1020303679279872.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=995324220637486.
07:30 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=476358037385030.
07:30 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:30 INFO [databricks.labs.ucx.install] Deleting secret scope
07:30 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='sdk-3jwh@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-3jwh@example.com')], entitlements=[], external_id=None, groups=[], id='5593186882414224', name=Name(family_name=None, given_name='sdk-3jwh@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-3jwh@example.com')
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace user User(active=True, display_name='sdk-3jwh@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-3jwh@example.com')], entitlements=[], external_id=None, groups=[], id='5593186882414224', name=Name(family_name=None, given_name='sdk-3jwh@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-3jwh@example.com') teardown: None User with id 5593186882414224 not found.
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_D5jw', entitlements=[], external_id=None, groups=[], id='363376592364516', members=[ComplexValue(display='sdk-3jwh@example.com', primary=None, ref='Users/5593186882414224', type=None, value='5593186882414224')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_D5jw', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='430512368140466', members=[ComplexValue(display='sdk-3jwh@example.com', primary=None, ref='Users/5593186882414224', type=None, value='5593186882414224')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace group Group(display_name='ucx_D5jw', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='430512368140466', members=[ComplexValue(display='sdk-3jwh@example.com', primary=None, ref='Users/5593186882414224', type=None, value='5593186882414224')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]) teardown: None Group with id 430512368140466 not found.
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
07:30 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_smrhe', metastore_id=None, name='ucx_smrhe', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:30 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_smrhe CASCADE
[gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

Running from nightly #52

@JCZuurmond JCZuurmond self-assigned this May 6, 2024
@JCZuurmond
Copy link
Member

Running this integration test in isolation works on an up-to-date main, let's wait another to see if the issue remains.

@JCZuurmond JCZuurmond mentioned this issue May 7, 2024
Merged
11 tasks
github-merge-queue bot pushed a commit that referenced this issue May 7, 2024
@JCZuurmond
Prefix random test group names with capital G (#1646)
641fbbb 
## Changes
Similarly to the prefix catalogs with "C", tables with "T" and schemas
with "S", also prefix random test group names with "G".

### Linked issues
Hopefully it narrows the KeyError in: #1616

### Functionality 

- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs ucx ...`
- [ ] added a new workflow
- [ ] modified existing workflow: `...`
- [ ] added a new table
- [ ] modified existing table: `...`

### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->

- [ ] manually tested
- [ ] added unit tests
- [ ] added integration tests
- [ ] verified on staging environment (screenshot attached)
github-merge-queue bot pushed a commit that referenced this issue May 7, 2024
@JCZuurmond
Prefix random test group names with capital G (#1646)
0a47514 
## Changes
Similarly to the prefix catalogs with "C", tables with "T" and schemas
with "S", also prefix random test group names with "G".

### Linked issues
Hopefully it narrows the KeyError in: #1616

### Functionality 

- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs ucx ...`
- [ ] added a new workflow
- [ ] modified existing workflow: `...`
- [ ] added a new table
- [ ] modified existing table: `...`

### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->

- [ ] manually tested
- [ ] added unit tests
- [ ] added integration tests
- [ ] verified on staging environment (screenshot attached)
@github-actions GitHub Actions
Copy link
Author

github-actions bot commented May 8, 2024

❌ test_running_real_assessment_job_ext_hms: KeyError: 'ucx_GIu4b' (28m11.568s) 
KeyError: 'ucx_GIu4b'
[gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-mrv6@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-mrv6@example.com')], entitlements=[], external_id=None, groups=[], id='4059398050832766', name=Name(family_name=None, given_name='sdk-mrv6@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-mrv6@example.com')
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_GIu4b: https://DATABRICKS_HOST#setting/accounts/groups/1011098775732898
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_GIu4b', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='1011098775732898', members=[ComplexValue(display='sdk-mrv6@example.com', primary=None, ref='Users/4059398050832766', type=None, value='4059398050832766')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_GIu4b: https://accounts.CLOUD_ENVdatabricks.net/users/groups/111040381670897/members
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_GIu4b', entitlements=[], external_id=None, groups=[], id='111040381670897', members=[ComplexValue(display='sdk-mrv6@example.com', primary=None, ref='Users/4059398050832766', type=None, value='4059398050832766')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/0006C10A86B07D44
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='0006C10A86B07D44')
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 0006C10A86B07D44 [group_name admins CAN_USE] -> [group_name ucx_GIu4b CAN_USE]
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_s7f5d
07:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s7f5d: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s7f5d
07:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s7f5d', metastore_id=None, name='ucx_s7f5d', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:18 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XZNr/config.yml) doesn't exist.
07:18 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:18 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:18 INFO [databricks.labs.ucx.install] Fetching installations...
07:18 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:18 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+5920240508071846
07:18 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_s7f5d
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.external_locations (location STRING NOT NULL... (40 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.workflow_problems (job_id LONG NOT NULL, job... (227 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.udfs (catalog STRING NOT NULL, database STRI... (288 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:21 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XZNr/README for the next steps.
07:21 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/763983188280201
07:34 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/763983188280201
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-mrv6@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-mrv6@example.com')], entitlements=[], external_id=None, groups=[], id='4059398050832766', name=Name(family_name=None, given_name='sdk-mrv6@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-mrv6@example.com')
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_GIu4b: https://DATABRICKS_HOST#setting/accounts/groups/1011098775732898
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_GIu4b', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='1011098775732898', members=[ComplexValue(display='sdk-mrv6@example.com', primary=None, ref='Users/4059398050832766', type=None, value='4059398050832766')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_GIu4b: https://accounts.CLOUD_ENVdatabricks.net/users/groups/111040381670897/members
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_GIu4b', entitlements=[], external_id=None, groups=[], id='111040381670897', members=[ComplexValue(display='sdk-mrv6@example.com', primary=None, ref='Users/4059398050832766', type=None, value='4059398050832766')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:06 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/0006C10A86B07D44
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='0006C10A86B07D44')
07:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 0006C10A86B07D44 [group_name admins CAN_USE] -> [group_name ucx_GIu4b CAN_USE]
07:06 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_s7f5d
07:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s7f5d: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s7f5d
07:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s7f5d', metastore_id=None, name='ucx_s7f5d', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:18 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XZNr/config.yml) doesn't exist.
07:18 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:18 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:18 INFO [databricks.labs.ucx.install] Fetching installations...
07:18 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:18 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 INFO [databricks.labs.ucx.install] Installing UCX v0.22.1+5920240508071846
07:18 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_s7f5d
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.external_locations (location STRING NOT NULL... (40 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.workflow_problems (job_id LONG NOT NULL, job... (227 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.udfs (catalog STRING NOT NULL, database STRI... (288 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_s7f5d.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_s7f5d.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:21 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XZNr/README for the next steps.
07:21 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/763983188280201
07:34 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/763983188280201
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 0006C10A86B07D44 [group_name admins CAN_USE] -> [group_name ucx_GIu4b CAN_USE]
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='0006C10A86B07D44')
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while cluster policy CreatePolicyResponse(policy_id='0006C10A86B07D44') teardown: Can't find a cluster policy with id: 0006C10A86B07D44.
07:34 INFO [databricks.labs.ucx.install] Deleting UCX v0.22.1+5920240508073412 from https://DATABRICKS_HOST
07:34 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_s7f5d
07:34 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_s7f5d CASCADE
07:34 INFO [databricks.labs.ucx.install] Deleting jobs
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=508300475131428.
07:34 INFO [databricks.labs.ucx.install] Deleting failing job_id=166634836047744.
07:34 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=226671084118256.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=336203416402049.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=146741978055920.
07:34 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=680320723792549.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=1061588380583787.
07:34 INFO [databricks.labs.ucx.install] Deleting assessment job_id=763983188280201.
07:34 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=421491137745054.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=930913717937964.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=149368825494921.
07:34 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:34 INFO [databricks.labs.ucx.install] Deleting secret scope
07:34 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='sdk-mrv6@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-mrv6@example.com')], entitlements=[], external_id=None, groups=[], id='4059398050832766', name=Name(family_name=None, given_name='sdk-mrv6@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-mrv6@example.com')
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace user User(active=True, display_name='sdk-mrv6@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-mrv6@example.com')], entitlements=[], external_id=None, groups=[], id='4059398050832766', name=Name(family_name=None, given_name='sdk-mrv6@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-mrv6@example.com') teardown: None User with id 4059398050832766 not found.
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_GIu4b', entitlements=[], external_id=None, groups=[], id='111040381670897', members=[ComplexValue(display='sdk-mrv6@example.com', primary=None, ref='Users/4059398050832766', type=None, value='4059398050832766')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_GIu4b', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='1011098775732898', members=[ComplexValue(display='sdk-mrv6@example.com', primary=None, ref='Users/4059398050832766', type=None, value='4059398050832766')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace group Group(display_name='ucx_GIu4b', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='1011098775732898', members=[ComplexValue(display='sdk-mrv6@example.com', primary=None, ref='Users/4059398050832766', type=None, value='4059398050832766')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]) teardown: None Group with id 1011098775732898 not found.
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s7f5d', metastore_id=None, name='ucx_s7f5d', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:34 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_s7f5d CASCADE
[gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

Running from nightly #54

@JCZuurmond JCZuurmond mentioned this issue May 8, 2024
Merged
11 tasks
github-merge-queue bot pushed a commit that referenced this issue May 8, 2024
@JCZuurmond
Assert if group name is in cluster policy (#1665)
f321e41 
## Changes
Assert if group name is in cluster policy

### Linked issues
Does not resolve the issue, but asserts the key to avoid the key error
in #1616

### Functionality 

- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs ucx ...`
- [ ] added a new workflow
- [ ] modified existing workflow: `...`
- [ ] added a new table
- [ ] modified existing table: `...`

### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->

- [ ] manually tested
- [ ] added unit tests
- [ ] added integration tests
- [ ] verified on staging environment (screenshot attached)
@JCZuurmond JCZuurmond mentioned this issue May 8, 2024
Merged
11 tasks
@github-actions GitHub Actions
Copy link
Author

github-actions bot commented May 9, 2024

❌ test_running_real_assessment_job_ext_hms: AssertionError: Group ucx_Gfrhs not found in cluster policy (29m25.923s) 
AssertionError: Group ucx_Gfrhs not found in cluster policy
assert 'ucx_Gfrhs' in {'0a330eb5-dd51-4d97-b6e4-c474356b1d5d': <PermissionLevel.CAN_USE: 'CAN_USE'>}
 +  where 'ucx_Gfrhs' = Group(display_name='ucx_Gfrhs', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='all...[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]).display_name
[gw7] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-jrzt@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-jrzt@example.com')], entitlements=[], external_id=None, groups=[], id='7535334714784639', name=Name(family_name=None, given_name='sdk-jrzt@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-jrzt@example.com')
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_Gfrhs: https://DATABRICKS_HOST#setting/accounts/groups/898539675448914
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_Gfrhs', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='898539675448914', members=[ComplexValue(display='sdk-jrzt@example.com', primary=None, ref='Users/7535334714784639', type=None, value='7535334714784639')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_Gfrhs: https://accounts.CLOUD_ENVdatabricks.net/users/groups/70908068725976/members
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_Gfrhs', entitlements=[], external_id=None, groups=[], id='70908068725976', members=[ComplexValue(display='sdk-jrzt@example.com', primary=None, ref='Users/7535334714784639', type=None, value='7535334714784639')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/0006678BE3CF7725
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='0006678BE3CF7725')
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 0006678BE3CF7725 [group_name admins CAN_USE] -> [group_name ucx_Gfrhs CAN_USE]
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sn6ow
07:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sn6ow: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sn6ow
07:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sn6ow', metastore_id=None, name='ucx_sn6ow', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:18 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.LGMU/config.yml) doesn't exist.
07:18 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:18 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:18 INFO [databricks.labs.ucx.install] Fetching installations...
07:18 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:18 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 INFO [databricks.labs.ucx.install] Installing UCX v0.23.1+220240509071855
07:18 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sn6ow
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.external_locations (location STRING NOT NULL... (40 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.workflow_problems (job_id LONG NOT NULL, job... (227 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.udfs (catalog STRING NOT NULL, database STRI... (288 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:21 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.LGMU/README for the next steps.
07:21 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/805561307152876
07:34 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/805561307152876
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='sdk-jrzt@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-jrzt@example.com')], entitlements=[], external_id=None, groups=[], id='7535334714784639', name=Name(family_name=None, given_name='sdk-jrzt@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-jrzt@example.com')
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_Gfrhs: https://DATABRICKS_HOST#setting/accounts/groups/898539675448914
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_Gfrhs', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='898539675448914', members=[ComplexValue(display='sdk-jrzt@example.com', primary=None, ref='Users/7535334714784639', type=None, value='7535334714784639')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_Gfrhs: https://accounts.CLOUD_ENVdatabricks.net/users/groups/70908068725976/members
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_Gfrhs', entitlements=[], external_id=None, groups=[], id='70908068725976', members=[ComplexValue(display='sdk-jrzt@example.com', primary=None, ref='Users/7535334714784639', type=None, value='7535334714784639')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:05 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/0006678BE3CF7725
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='0006678BE3CF7725')
07:05 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 0006678BE3CF7725 [group_name admins CAN_USE] -> [group_name ucx_Gfrhs CAN_USE]
07:05 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA hive_metastore.ucx_sn6ow
07:18 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sn6ow: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sn6ow
07:18 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sn6ow', metastore_id=None, name='ucx_sn6ow', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:18 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.LGMU/config.yml) doesn't exist.
07:18 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:18 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:18 INFO [databricks.labs.ucx.install] Fetching installations...
07:18 INFO [databricks.labs.ucx.installer.policy] Setting up an external metastore
07:18 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:18 INFO [databricks.labs.ucx.install] Installing UCX v0.23.1+220240509071855
07:18 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE SCHEMA IF NOT EXISTS hive_metastore.ucx_sn6ow
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.CLOUD_ENV_service_principals (application_id STR... (107 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.clusters (cluster_id STRING NOT NULL, succes... (131 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.global_init_scripts (script_id STRING NOT NU... (120 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.jobs (job_id STRING NOT NULL, success LONG N... (79 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.pipelines (pipeline_id STRING NOT NULL, succ... (99 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.external_locations (location STRING NOT NULL... (40 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.mounts (name STRING NOT NULL, source STRING ... (21 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.grants (principal STRING NOT NULL, action_ty... (167 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.groups (id_in_workspace STRING NOT NULL, nam... (179 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.tables (catalog STRING NOT NULL, database ST... (222 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.table_size (catalog STRING NOT NULL, databas... (81 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.table_failures (catalog STRING NOT NULL, dat... (61 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.workspace_objects (path STRING NOT NULL, obj... (63 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.permissions (object_id STRING NOT NULL, obje... (57 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.submit_runs (run_ids STRING NOT NULL, hashed... (58 more bytes)
07:18 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.policies (policy_id STRING NOT NULL, policy_... (147 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.migration_status (src_schema STRING NOT NULL... (115 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.workflow_problems (job_id LONG NOT NULL, job... (227 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.udfs (catalog STRING NOT NULL, database STRI... (288 more bytes)
07:19 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE TABLE IF NOT EXISTS hive_metastore.ucx_sn6ow.logs (timestamp LONG NOT NULL, job_id LONG N... (179 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.objects AS SELECT "jobs" AS object_type, job_id ... (1639 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.grant_detail AS SELECT CASE WHEN anonymous_funct... (1037 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.table_estimates AS select concat(catalog, ".", d... (1227 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.misc_patterns AS SELECT col1 AS commandLanguage,... (559 more bytes)
07:20 DEBUG [tests.integration.test_ext_hms] [api][execute] CREATE OR REPLACE VIEW hive_metastore.ucx_sn6ow.code_patterns AS SELECT col1 AS pattern, col2 AS... (3796 more bytes)
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:21 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:21 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.LGMU/README for the next steps.
07:21 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/805561307152876
07:34 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/805561307152876
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 0006678BE3CF7725 [group_name admins CAN_USE] -> [group_name ucx_Gfrhs CAN_USE]
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='0006678BE3CF7725')
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while cluster policy CreatePolicyResponse(policy_id='0006678BE3CF7725') teardown: Can't find a cluster policy with id: 0006678BE3CF7725.
07:34 INFO [databricks.labs.ucx.install] Deleting UCX v0.23.1+220240509073432 from https://DATABRICKS_HOST
07:34 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_sn6ow
07:34 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sn6ow CASCADE
07:34 INFO [databricks.labs.ucx.install] Deleting jobs
07:34 INFO [databricks.labs.ucx.install] Deleting assessment job_id=805561307152876.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=888058956393416.
07:34 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=1120896398887648.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=946932723019634.
07:34 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=1122088821949531.
07:34 INFO [databricks.labs.ucx.install] Deleting failing job_id=734032708287244.
07:34 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=313296447930976.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=147829886125175.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=520850033632636.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=209220968508595.
07:34 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=161822184494260.
07:34 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:34 INFO [databricks.labs.ucx.install] Deleting secret scope
07:34 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='sdk-jrzt@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-jrzt@example.com')], entitlements=[], external_id=None, groups=[], id='7535334714784639', name=Name(family_name=None, given_name='sdk-jrzt@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-jrzt@example.com')
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace user User(active=True, display_name='sdk-jrzt@example.com', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='sdk-jrzt@example.com')], entitlements=[], external_id=None, groups=[], id='7535334714784639', name=Name(family_name=None, given_name='sdk-jrzt@example.com'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='sdk-jrzt@example.com') teardown: None User with id 7535334714784639 not found.
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_Gfrhs', entitlements=[], external_id=None, groups=[], id='70908068725976', members=[ComplexValue(display='sdk-jrzt@example.com', primary=None, ref='Users/7535334714784639', type=None, value='7535334714784639')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_Gfrhs', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='898539675448914', members=[ComplexValue(display='sdk-jrzt@example.com', primary=None, ref='Users/7535334714784639', type=None, value='7535334714784639')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace group Group(display_name='ucx_Gfrhs', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='898539675448914', members=[ComplexValue(display='sdk-jrzt@example.com', primary=None, ref='Users/7535334714784639', type=None, value='7535334714784639')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]) teardown: None Group with id 898539675448914 not found.
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
07:34 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sn6ow', metastore_id=None, name='ucx_sn6ow', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:34 DEBUG [tests.integration.test_ext_hms] [api][execute] DROP SCHEMA IF EXISTS hive_metastore.ucx_sn6ow CASCADE
[gw7] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

Running from nightly #55

github-merge-queue bot pushed a commit that referenced this issue May 9, 2024
@JCZuurmond
Fix test failing due to groups not being created yet (#1669)
7997179 
## Changes
(We think that) sometimes groups are not created before the test run,
e.g.
[this](https://github.com/databrickslabs/ucx/actions/runs/9002466231?pr=1532)
and [that](#1616). This PR
extends the `make_group` with an optional flag to wait for the groups to
be created before continuing with the test. To limit impact on
performance, we don't wait for the groups to be created by default.

### Linked issues
Hopefully esolves #1616

### Functionality 

- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs ucx ...`
- [ ] added a new workflow
- [ ] modified existing workflow: `...`
- [ ] added a new table
- [ ] modified existing table: `...`

### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->

- [x] manually tested
- [ ] added unit tests
- [ ] added integration tests
- [ ] verified on staging environment (screenshot attached)
@JCZuurmond
Copy link
Member

Not present anymore in latest test run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JCZuurmond JCZuurmond

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@JCZuurmond