Skip to content

Commit

Permalink
Remove WorkflowLinter
Browse files Browse the repository at this point in the history
  • Loading branch information
JCZuurmond committed Oct 22, 2024
1 parent f58aa24 commit 77b3dc0
Show file tree
Hide file tree
Showing 4 changed files with 9 additions and 50 deletions.
22 changes: 3 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -401,9 +401,8 @@ which can be used for further analysis and decision-making through the [assessme
9. `assess_pipelines`: This task scans through all the Pipelines and identifies those pipelines that have Azure Service Principals embedded in their configurations. A list of all the pipelines with matching configurations is stored in the `$inventory.pipelines` table.
10. `assess_azure_service_principals`: This task scans through all the clusters configurations, cluster policies, job cluster configurations, Pipeline configurations, and Warehouse configuration and identifies all the Azure Service Principals who have been given access to the Azure storage accounts via spark configurations referred in those entities. The list of all the Azure Service Principals referred in those configurations is saved in the `$inventory.azure_service_principals` table.
11. `assess_global_init_scripts`: This task scans through all the global init scripts and identifies if there is an Azure Service Principal who has been given access to the Azure storage accounts via spark configurations referred in those scripts.
12. `assess_dashboards`: This task scans through all the dashboards and analyzes embedded queries for migration problems. It also collects direct filesystem access patterns that require attention.
13. `assess_workflows`: This task scans through all the jobs and tasks and analyzes notebooks and files for migration problems. It also collects direct filesystem access patterns that require attention.

12. `assess_dashboards`: This task scans through all the dashboards and analyzes embedded queries for migration problems which it persists in `$inventory_database.query_problems`. It also collects direct filesystem access patterns that require attention which it persists in `$inventory_database.directfs_in_queries`.
13. `assess_workflows`: This task scans through all the jobs and tasks and analyzes notebooks and files for migration problems which it persists in `$inventory_database.workflow_problems`. It also collects direct filesystem access patterns that require attention which it persists in `$inventory_database.directfs_in_paths`.

![report](docs/assessment-report.png)

Expand Down Expand Up @@ -726,25 +725,10 @@ in the Migration dashboard.

[[back to top](#databricks-labs-ucx)]

## Jobs Static Code Analysis Workflow

The `workflow-linter` workflow lints accessible code from 2 sources:
- all workflows/jobs present in the workspace
- all dashboards/queries present in the workspace
The linting emits problems indicating what to resolve for making the code Unity Catalog compatible.
The linting also locates direct filesystem access that need to be migrated.

Once the workflow completes:
- problems are stored in the `$inventory_database.workflow_problems`/`$inventory_database.query_problems` table
- direct filesystem access are stored in the `$inventory_database.directfs_in_paths`/`$inventory_database.directfs_in_queries` table
- all the above are displayed in the Migration dashboard.
### Linter message codes

![code compatibility problems](docs/code_compatibility_problems.png)

[[back to top](#databricks-labs-ucx)]

### Linter message codes

Here's the detailed explanation of the linter message codes:

#### `cannot-autofix-table-reference`
Expand Down
12 changes: 6 additions & 6 deletions src/databricks/labs/ucx/assessment/workflows.py
Original file line number Diff line number Diff line change
Expand Up @@ -184,20 +184,20 @@ def crawl_groups(self, ctx: RuntimeContext):
ctx.group_manager.snapshot()

@job_task
def assess_workflows(self, ctx: RuntimeContext):
"""Scans all jobs for migration issues in notebooks jobs.
def assess_dashboards(self, ctx: RuntimeContext):
"""Scans all dashboards for migration issues in SQL code of embedded widgets.
Also, stores direct filesystem accesses for display in the migration dashboard.
"""
ctx.workflow_linter.refresh_report(ctx.sql_backend, ctx.inventory_database)
ctx.query_linter.refresh_report(ctx.sql_backend, ctx.inventory_database)

@job_task
def assess_dashboards(self, ctx: RuntimeContext):
"""Scans all dashboards for migration issues in SQL code of embedded widgets.
def assess_workflows(self, ctx: RuntimeContext):
"""Scans all jobs for migration issues in notebooks jobs.
Also, stores direct filesystem accesses for display in the migration dashboard.
"""
ctx.query_linter.refresh_report(ctx.sql_backend, ctx.inventory_database)
ctx.workflow_linter.refresh_report(ctx.sql_backend, ctx.inventory_database)


class Failing(Workflow):
Expand Down
2 changes: 0 additions & 2 deletions src/databricks/labs/ucx/runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@
)
from databricks.labs.ucx.progress.workflows import MigrationProgress
from databricks.labs.ucx.recon.workflows import MigrationRecon
from databricks.labs.ucx.source_code.workflows import WorkflowLinter
from databricks.labs.ucx.workspace_access.workflows import (
GroupMigration,
PermissionsMigrationAPI,
Expand Down Expand Up @@ -58,7 +57,6 @@ def all(cls):
ScanTablesInMounts(),
MigrateTablesInMounts(),
PermissionsMigrationAPI(),
WorkflowLinter(),
MigrationRecon(),
Failing(),
]
Expand Down
23 changes: 0 additions & 23 deletions src/databricks/labs/ucx/source_code/workflows.py

This file was deleted.

0 comments on commit 77b3dc0

Please sign in to comment.