Added baseline for getting Azure Resource Role Assignments #764
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HariGS-DB
had a problem deploying
to
account-admin
GitHub Actions
Failure
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #764 +/- ##
==========================================
+ Coverage 84.15% 84.39% +0.23%
==========================================
Files 39 39
Lines 4601 4850 +249
Branches 859 907 +48
==========================================
+ Hits 3872 4093 +221
- Misses 528 545 +17
- Partials 201 212 +11 ☔ View full report in Codecov by Sentry. |
HariGS-DB
had a problem deploying
to
account-admin
GitHub Actions
Failure
HariGS-DB
had a problem deploying
to
account-admin
GitHub Actions
Failure
HariGS-DB
had a problem deploying
to
account-admin
GitHub Actions
Failure
HariGS-DB
had a problem deploying
to
account-admin
GitHub Actions
Failure
HariGS-DB
temporarily deployed
to
account-admin
GitHub Actions
Inactive
HariGS-DB
temporarily deployed
to
account-admin
GitHub Actions
Inactive
HariGS-DB
had a problem deploying
to
account-admin
GitHub Actions
Error
HariGS-DB
temporarily deployed
to
account-admin
GitHub Actions
Inactive
HariGS-DB
temporarily deployed
to
account-admin
GitHub Actions
Inactive
HariGS-DB
had a problem deploying
to
account-admin
GitHub Actions
Failure
HariGS-DB
had a problem deploying
to
account-admin
GitHub Actions
Failure
HariGS-DB
temporarily deployed
to
account-admin
GitHub Actions
Inactive
HariGS-DB
had a problem deploying
to
account-admin
GitHub Actions
Failure
HariGS-DB
temporarily deployed
to
account-admin
GitHub Actions
Inactive
HariGS-DB
had a problem deploying
to
account-admin
GitHub Actions
Failure
HariGS-DB
temporarily deployed
to
account-admin
GitHub Actions
Inactive
FastLee
pushed a commit
that referenced
this pull request
Jan 19, 2024
This pull request adds a new command `save-azure-storage-accounts` to the UCX CLI tool. This command identifies all storage accounts used by tables in a HMS metastore, identifies the corresponding service principals and their permissions on each storage account, and saves the data in the CSV file on workspace. The new method `AzureResourcePermissions.save_spn_permissions` performs this functionality. Additionally, the pull request adds the `AzureResources` class, which is used to fetch information about Azure resources such as subscriptions, storage accounts, and containers. The `AzureRoleAssignment` and `Principal` classes are also added to represent role assignments and their associated principals. Changes related to tests include: * Test cases are added for the new `save-azure-storage-accounts` command, including tests for valid and invalid subscription IDs, as well as tests for cases where there are no external tables, no Azure storage accounts, or no valid Azure storage accounts. * The test case for `test_azure_spn_info_without_secret` is updated to use the `create_autospec` function to create a mock `WorkspaceClient` object instead of creating a mock object manually. * The `test_move` function is updated to use the `patch` decorator to patch the `TableMove.move_tables` method instead of using a mock object. * The `test_save_azure_storage_accounts_no_ucx` test case is added to test the behavior when UCX is not installed. * The `test_save_azure_storage_accounts_not_azure` test case is added to test the behavior when the workspace is not on Azure. * The `test_save_azure_storage_accounts_no_azure_cli` test case is added to test the behavior when the Azure CLI authentication method is not used. * The `test_save_azure_storage_accounts_no_subscription_id` test case is added to test the behavior when the subscription ID is not provided. --------- Co-authored-by: Hari Selvarajan <hari.selvarajan@databricks.com>
nfx
added a commit
that referenced
this pull request
Jan 19, 2024
* Added `databricks labs ucx validate-groups-membership` command to validate groups to see if they have same membership across acount and workspace level ([#772](#772)). * Added baseline for getting Azure Resource Role Assignments ([#764](#764)). * Added issue and pull request templates ([#791](#791)). * Added linked issues to PR template ([#793](#793)). * Added optional `debug_truncate_bytes` parameter to the config and extend the default log truncation limit ([#782](#782)). * Added support for crawling grants and applying Hive Metastore UDF ACLs ([#812](#812)). * Changed Python requirement from 3.10.6 to 3.10 ([#805](#805)). * Extend error handling of delta issues in crawlers and hive metastore ([#795](#795)). * Fixed `databricks labs ucx repair-run` command to execute correctly ([#801](#801)). * Fixed handling of `DELTASHARING` table format ([#802](#802)). * Fixed listing of workflows via CLI ([#811](#811)). * Fixed logger import path for DEBUG notebook ([#792](#792)). * Fixed move table command to delete table/view regardless if permissions are present, skipping corrupted tables when crawling table size and making existing tests more stable ([#777](#777)). * Fixed the issue of `databricks labs ucx installations` and `databricks labs ucx manual-workspace-info` ([#814](#814)). * Increase the unit test coverage for cli.py ([#800](#800)). * Mount Point crawler lists /Volume with four variations which is confusing ([#779](#779)). * Updated README.md to remove mention of deprecated install.sh ([#781](#781)). * Updated `bug` issue template ([#797](#797)). * Fixed writing log readme in multiprocess safe way ([#794](#794)).
Merged
nfx
added a commit
that referenced
this pull request
Jan 19, 2024
* Added `databricks labs ucx validate-groups-membership` command to validate groups to see if they have same membership across acount and workspace level ([#772](#772)). * Added baseline for getting Azure Resource Role Assignments ([#764](#764)). * Added issue and pull request templates ([#791](#791)). * Added linked issues to PR template ([#793](#793)). * Added optional `debug_truncate_bytes` parameter to the config and extend the default log truncation limit ([#782](#782)). * Added support for crawling grants and applying Hive Metastore UDF ACLs ([#812](#812)). * Changed Python requirement from 3.10.6 to 3.10 ([#805](#805)). * Extend error handling of delta issues in crawlers and hive metastore ([#795](#795)). * Fixed `databricks labs ucx repair-run` command to execute correctly ([#801](#801)). * Fixed handling of `DELTASHARING` table format ([#802](#802)). * Fixed listing of workflows via CLI ([#811](#811)). * Fixed logger import path for DEBUG notebook ([#792](#792)). * Fixed move table command to delete table/view regardless if permissions are present, skipping corrupted tables when crawling table size and making existing tests more stable ([#777](#777)). * Fixed the issue of `databricks labs ucx installations` and `databricks labs ucx manual-workspace-info` ([#814](#814)). * Increase the unit test coverage for cli.py ([#800](#800)). * Mount Point crawler lists /Volume with four variations which is confusing ([#779](#779)). * Updated README.md to remove mention of deprecated install.sh ([#781](#781)). * Updated `bug` issue template ([#797](#797)). * Fixed writing log readme in multiprocess safe way ([#794](#794)).
This was referenced Jan 30, 2024
dmoore247
pushed a commit
that referenced
this pull request
Mar 23, 2024
* Added `databricks labs ucx validate-groups-membership` command to validate groups to see if they have same membership across acount and workspace level ([#772](#772)). * Added baseline for getting Azure Resource Role Assignments ([#764](#764)). * Added issue and pull request templates ([#791](#791)). * Added linked issues to PR template ([#793](#793)). * Added optional `debug_truncate_bytes` parameter to the config and extend the default log truncation limit ([#782](#782)). * Added support for crawling grants and applying Hive Metastore UDF ACLs ([#812](#812)). * Changed Python requirement from 3.10.6 to 3.10 ([#805](#805)). * Extend error handling of delta issues in crawlers and hive metastore ([#795](#795)). * Fixed `databricks labs ucx repair-run` command to execute correctly ([#801](#801)). * Fixed handling of `DELTASHARING` table format ([#802](#802)). * Fixed listing of workflows via CLI ([#811](#811)). * Fixed logger import path for DEBUG notebook ([#792](#792)). * Fixed move table command to delete table/view regardless if permissions are present, skipping corrupted tables when crawling table size and making existing tests more stable ([#777](#777)). * Fixed the issue of `databricks labs ucx installations` and `databricks labs ucx manual-workspace-info` ([#814](#814)). * Increase the unit test coverage for cli.py ([#800](#800)). * Mount Point crawler lists /Volume with four variations which is confusing ([#779](#779)). * Updated README.md to remove mention of deprecated install.sh ([#781](#781)). * Updated `bug` issue template ([#797](#797)). * Fixed writing log readme in multiprocess safe way ([#794](#794)).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request adds a new command
save-azure-storage-accountsto the UCX CLI tool. This command identifies all storage accounts used by tables in a HMS metastore, identifies the corresponding service principals and their permissions on each storage account, and saves the data in the CSV file on workspace. The new methodAzureResourcePermissions.save_spn_permissionsperforms this functionality.Additionally, the pull request adds the
AzureResourcesclass, which is used to fetch information about Azure resources such as subscriptions, storage accounts, and containers. TheAzureRoleAssignmentandPrincipalclasses are also added to represent role assignments and their associated principals.Changes related to tests include:
save-azure-storage-accountscommand, including tests for valid and invalid subscription IDs, as well as tests for cases where there are no external tables, no Azure storage accounts, or no valid Azure storage accounts.test_azure_spn_info_without_secretis updated to use thecreate_autospecfunction to create a mockWorkspaceClientobject instead of creating a mock object manually.test_movefunction is updated to use thepatchdecorator to patch theTableMove.move_tablesmethod instead of using a mock object.test_save_azure_storage_accounts_no_ucxtest case is added to test the behavior when UCX is not installed.test_save_azure_storage_accounts_not_azuretest case is added to test the behavior when the workspace is not on Azure.test_save_azure_storage_accounts_no_azure_clitest case is added to test the behavior when the Azure CLI authentication method is not used.test_save_azure_storage_accounts_no_subscription_idtest case is added to test the behavior when the subscription ID is not provided.