Added a crawler for creating an inventory of Azure Service Principals #326

dipankarkush-db · 2023-09-28T20:37:38Z

No description provided.

Fixes #249

codecov · 2023-09-28T20:39:47Z

Codecov Report

Merging #326 (f57bc26) into main (d7cc195) will increase coverage by 0.32%.
The diff coverage is 84.79%.

@@            Coverage Diff             @@
##             main     #326      +/-   ##
==========================================
+ Coverage   83.48%   83.80%   +0.32%     
==========================================
  Files          31       31              
  Lines        2555     2810     +255     
  Branches      448      532      +84     
==========================================
+ Hits         2133     2355     +222     
- Misses        319      332      +13     
- Partials      103      123      +20

Files	Coverage Δ
src/databricks/labs/ucx/runtime.py	`50.53% <43.75%> (-2.03%)`	⬇️
src/databricks/labs/ucx/assessment/crawlers.py	`82.86% <87.44%> (+9.20%)`	⬆️

Fixes #249

src/databricks/labs/ucx/assessment/crawlers.py

Fixes #249

tests/unit/assessment/test_assessment.py

src/databricks/labs/ucx/assessment/crawlers.py

Fixes #249

…ix/azure-SPNs-inventory # Conflicts: # src/databricks/labs/ucx/assessment/crawlers.py # tests/unit/assessment/test_assessment.py

Fixes #249

nfx

yes, _list_all_cluster_with_spn_in_spark_conf in AzureServicePrincipalCrawler is a bit better. see other comments

src/databricks/labs/ucx/assessment/crawlers.py

tests/unit/assessment/test_assessment.py

Fixes #249

gitguardian · 2023-09-29T18:43:37Z

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	Secret	Commit	Filename
8289420	Generic High Entropy Secret	`784c7f9`	tests/unit/assessment/test_assessment.py	View secret
8289420	Generic High Entropy Secret	`020f631`	tests/unit/assessment/test_assessment.py	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!}

Fixes #249

nfx

tiny comment

tests/integration/assessment/test_assessment.py

Fixes #249

dipankarkush-db · 2023-10-08T07:06:27Z

This looks mostly OK, but I don't see any scanning of init scripts to see if the same is being set in init scripts.
Here's what I think is missing:

Check cluster scoped init scripts for spark config

Check global init scripts for spark config

Do we want to handle scanning the init scripts (cluster and global) separately? There could be other entries in the init scripts (such as installation of packages) that we might be interested and its content are relevant for all cloud platform. We could run a dependent job after the current jobs to capture the details from init scripts and if any matching spark config for Azure is found then append to the cluster, job and Azure SPN tables. If agreed, then i could create a new issue.

Added the following -

Azure SPN present in cluster scope init scripts for job and cluster and if yes then flag those clusters
Check all the global init scripts and flag all those which have Azure SPN
List of global init script is also added in the dashboard

Did not do - List of all Azure SPNs from all the init scripts. Lets discuss about it. It makes sense to me to add a separate issue for the same and merge this one.

dipankarkush-db · 2023-10-08T08:08:52Z

This looks mostly OK, but I don't see any scanning of init scripts to see if the same is being set in init scripts.

Here's what I think is missing:

Check cluster scoped init scripts for spark config

Check global init scripts for spark config

Fixes #249

src/databricks/labs/ucx/mixins/fixtures.py

Fixes #249

# Version changelog ## 0.4.0 * Added exception handling for secret scope not found. ([#418](#418)). * Added a crawler for creating an inventory of Azure Service Principals ([#326](#326)). * Added check if account group already exists during failure recovery ([#446](#446)). * Added checking for index out of range. ([#429](#429)). * Added hyperlink to UCX releases in the main readme ([#408](#408)). * Added integration test to check backup groups get deleted ([#387](#387)). * Added logging of errors during threadpool operations. ([#376](#376)). * Added recovery mode for workspace-local groups from temporary groups ([#435](#435)). * Added support for migrating Legacy Table ACLs from workspace-local to account-level groups ([#412](#412)). * Added detection for installations of unreleased versions ([#399](#399)). * Decoupled `PermissionsManager` from `GroupMigrationToolkit` ([#407](#407)). * Enabled debug logging for every job task run through a file, which is accessible from both workspace UI and Databricks CLI ([#426](#426)). * Ensured that table exists, even when crawlers produce zero records ([#373](#373)). * Extended test suite for HMS->HMS TACL migration ([#439](#439)). * Fixed handling of secret scope responses ([#431](#431)). * Fixed `crawl_permissions` task to respect 'workspace_start_path' config ([#444](#444)). * Fixed broken logic in `parallel` module and applied hardened error handling design for parallel code ([#405](#405)). * Fixed codecov.io reporting ([#403](#403)). * Fixed integration tests for crawlers ([#379](#379)). * Improved README.py and logging messages ([#433](#433)). * Improved cleanup for workspace backup groups by adding more retries on errors ([#375](#375)). * Improved dashboard queries to show unsupported storage types. ([#398](#398)). * Improved documentation for readme notebook ([#257](#257)). * Improved test coverage for installer ([#371](#371)). * Introduced deterministic `env_or_skip` fixture for integration tests ([#396](#396)). * Made HMS & UC fixtures return `CatalogInfo`, `SchemaInfo`, and `TableInfo` ([#409](#409)). * Merge `workspace_access.Crawler` and `workspace_access.Applier` interfaces to `workspace_access.AclSupport` ([#436](#436)). * Moved examples to docs ([#404](#404)). * Properly isolated integration testing for workflows on an existing shared cluster ([#414](#414)). * Removed thread pool for any IAM Group removals and additions ([#394](#394)). * Replace plus char with minus in version tag for GCP dev installation of UCX ([#420](#420)). * Run integration tests on shared clusters for a faster devloop ([#397](#397)). * Show difference between serverless and PRO warehouses during installation ([#385](#385)). * Split `migrate-groups` workflow into three different stages for reliability ([#442](#442)). * Use groups instead of usernames in code owners file ([#389](#389)).

…#326)

# Version changelog ## 0.4.0 * Added exception handling for secret scope not found. ([#418](#418)). * Added a crawler for creating an inventory of Azure Service Principals ([#326](#326)). * Added check if account group already exists during failure recovery ([#446](#446)). * Added checking for index out of range. ([#429](#429)). * Added hyperlink to UCX releases in the main readme ([#408](#408)). * Added integration test to check backup groups get deleted ([#387](#387)). * Added logging of errors during threadpool operations. ([#376](#376)). * Added recovery mode for workspace-local groups from temporary groups ([#435](#435)). * Added support for migrating Legacy Table ACLs from workspace-local to account-level groups ([#412](#412)). * Added detection for installations of unreleased versions ([#399](#399)). * Decoupled `PermissionsManager` from `GroupMigrationToolkit` ([#407](#407)). * Enabled debug logging for every job task run through a file, which is accessible from both workspace UI and Databricks CLI ([#426](#426)). * Ensured that table exists, even when crawlers produce zero records ([#373](#373)). * Extended test suite for HMS->HMS TACL migration ([#439](#439)). * Fixed handling of secret scope responses ([#431](#431)). * Fixed `crawl_permissions` task to respect 'workspace_start_path' config ([#444](#444)). * Fixed broken logic in `parallel` module and applied hardened error handling design for parallel code ([#405](#405)). * Fixed codecov.io reporting ([#403](#403)). * Fixed integration tests for crawlers ([#379](#379)). * Improved README.py and logging messages ([#433](#433)). * Improved cleanup for workspace backup groups by adding more retries on errors ([#375](#375)). * Improved dashboard queries to show unsupported storage types. ([#398](#398)). * Improved documentation for readme notebook ([#257](#257)). * Improved test coverage for installer ([#371](#371)). * Introduced deterministic `env_or_skip` fixture for integration tests ([#396](#396)). * Made HMS & UC fixtures return `CatalogInfo`, `SchemaInfo`, and `TableInfo` ([#409](#409)). * Merge `workspace_access.Crawler` and `workspace_access.Applier` interfaces to `workspace_access.AclSupport` ([#436](#436)). * Moved examples to docs ([#404](#404)). * Properly isolated integration testing for workflows on an existing shared cluster ([#414](#414)). * Removed thread pool for any IAM Group removals and additions ([#394](#394)). * Replace plus char with minus in version tag for GCP dev installation of UCX ([#420](#420)). * Run integration tests on shared clusters for a faster devloop ([#397](#397)). * Show difference between serverless and PRO warehouses during installation ([#385](#385)). * Split `migrate-groups` workflow into three different stages for reliability ([#442](#442)). * Use groups instead of usernames in code owners file ([#389](#389)).

dipankarkush-db added 2 commits September 28, 2023 16:32

Added crawler for Azure Service principals.

9a1a32f

Fixes #249

Added crawler for Azure Service principals.

c31a48b

Fixes #249

dipankarkush-db added 2 commits September 28, 2023 19:02

Added crawler for Azure Service principals.

183c12b

Fixes #249

Added crawler for Azure Service principals.

d499d08

Fixes #249

nfx requested changes Sep 28, 2023

View reviewed changes

src/databricks/labs/ucx/assessment/crawlers.py Outdated Show resolved Hide resolved

src/databricks/labs/ucx/assessment/crawlers.py Outdated Show resolved Hide resolved

src/databricks/labs/ucx/assessment/crawlers.py Outdated Show resolved Hide resolved

Added crawler for Azure Service principals.

3aeeec8

Fixes #249

nfx requested changes Sep 29, 2023

View reviewed changes

tests/unit/assessment/test_assessment.py Outdated Show resolved Hide resolved

src/databricks/labs/ucx/assessment/crawlers.py Outdated Show resolved Hide resolved

src/databricks/labs/ucx/assessment/crawlers.py Outdated Show resolved Hide resolved

dipankarkush-db added 6 commits September 29, 2023 11:04

Added crawler for Azure Service principals.

00dfaba

Fixes #249

Merge remote-tracking branch 'origin/fix/azure-SPNs-inventory' into f…

81492f0

…ix/azure-SPNs-inventory # Conflicts: # src/databricks/labs/ucx/assessment/crawlers.py # tests/unit/assessment/test_assessment.py

Added crawler for Azure Service principals.

bb66cac

Fixes #249

Added crawler for Azure Service principals.

7c98fb2

Fixes #249

Added crawler for Azure Service principals.

87c7007

Fixes #249

Added crawler for Azure Service principals.

9aeaefe

Fixes #249

nfx requested changes Sep 29, 2023

View reviewed changes

Added crawler for Azure Service principals.

784c7f9

Fixes #249

dipankarkush-db added 13 commits September 29, 2023 14:47

Added crawler for Azure Service principals.

020f631

Fixes #249

Added crawler for Azure Service principals.

77bffb1

Fixes #249

Added crawler for Azure Service principals.

9592a24

Fixes #249

Added crawler for Azure Service principals.

ce9a3fa

Fixes #249

Added crawler for Azure Service principals.

6c45d1d

Fixes #249

Added crawler for Azure Service principals.

6b0a083

Fixes #249

Added crawler for Azure Service principals.

4037676

Fixes #249

Added crawler for Azure Service principals.

f8e9563

Fixes #249

Added crawler for Azure Service principals.

5793828

Fixes #249

Added crawler for Azure Service principals.

6fdcb5c

Fixes #249

Added crawler for Azure Service principals.

00a7897

Fixes #249

Added crawler for Azure Service principals.

0a77a59

Fixes #249

Added crawler for Azure Service principals.

bb8eace

Fixes #249

nfx requested changes Oct 7, 2023

View reviewed changes

tests/integration/assessment/test_assessment.py Outdated Show resolved Hide resolved

dipankarkush-db added 9 commits October 7, 2023 19:28

Merge branch 'main' into fix/azure-SPNs-inventory

924c1c9

Added changes for supporting Klass.

8e8f1a5

Fixes #249

Added changes for supporting Klass.

8d7dae2

Fixes #249

Added changes for supporting Klass.

80fdac6

Fixes #249

Added changes for supporting Klass.

eca081e

Fixes #249

Added changes for supporting Klass.

1d972c6

Fixes #249

Added changes for supporting Klass.

476b3d0

Fixes #249

Added changes for supporting Klass.

d27d10a

Fixes #249

Added changes for supporting Klass.

37905fa

Fixes #249

dipankarkush-db added 2 commits October 8, 2023 04:10

Added changes for supporting Klass.

e37728d

Fixes #249

Added changes for supporting Klass.

cc48e0c

Fixes #249

nfx approved these changes Oct 8, 2023

View reviewed changes

src/databricks/labs/ucx/mixins/fixtures.py Outdated Show resolved Hide resolved

src/databricks/labs/ucx/mixins/fixtures.py Show resolved Hide resolved

nfx enabled auto-merge October 8, 2023 15:50

dipankarkush-db added 4 commits October 8, 2023 13:27

Added changes for supporting Klass.

172d2fd

Fixes #249

Added changes for supporting Klass.

1aedfca

Fixes #249

Added changes for supporting Klass.

052aef4

Fixes #249

Added changes for supporting Klass.

f57bc26

Fixes #249

nfx changed the title ~~Added a Crawler for creating an inventory of Azure service principals~~ Added a crawler for creating an inventory of Azure Service Principals Oct 8, 2023

nfx merged commit 994c127 into main Oct 8, 2023
5 checks passed

dipankarkush-db mentioned this pull request Oct 8, 2023

Create an inventory of Azure Service Principals who has access to storage by scanning all init scripts #413

Closed

zpappa mentioned this pull request Oct 9, 2023

Add support for init scripts in crawling for Azure Service Principals #415

Closed

nfx mentioned this pull request Oct 12, 2023

Release v0.4.0 #447

Merged

nfx deleted the fix/azure-SPNs-inventory branch October 17, 2023 22:44

FastLee pushed a commit that referenced this pull request Oct 25, 2023

Added a crawler for creating an inventory of Azure Service Principals (…

aa9013e

…#326)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added a crawler for creating an inventory of Azure Service Principals #326

Added a crawler for creating an inventory of Azure Service Principals #326

dipankarkush-db commented Sep 28, 2023 •

edited by nfx

Loading

codecov bot commented Sep 28, 2023 •

edited

Loading

nfx left a comment

gitguardian bot commented Sep 29, 2023 •

edited

Loading

nfx left a comment

dipankarkush-db commented Oct 8, 2023

dipankarkush-db commented Oct 8, 2023

Added a crawler for creating an inventory of Azure Service Principals #326

Added a crawler for creating an inventory of Azure Service Principals #326

Conversation

dipankarkush-db commented Sep 28, 2023 • edited by nfx Loading

codecov bot commented Sep 28, 2023 • edited Loading

Codecov Report

nfx left a comment

Choose a reason for hiding this comment

gitguardian bot commented Sep 29, 2023 • edited Loading

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

nfx left a comment

Choose a reason for hiding this comment

dipankarkush-db commented Oct 8, 2023

dipankarkush-db commented Oct 8, 2023

dipankarkush-db commented Sep 28, 2023 •

edited by nfx

Loading

codecov bot commented Sep 28, 2023 •

edited

Loading

gitguardian bot commented Sep 29, 2023 •

edited

Loading