[FEATURE]: Create UC External Location, Schema, and Table Grants based on workspace-wide Azure SPN mount points

[nfx]

Problem statement

If we have Azure Blob Storage mount points, data is accessible for all clusters and warehouses in the workspace (at least for those people who have access to those warehouses/clusters). We need to migrate that to UC ACLs.

Related issues:

• [FEATURE]: Create UC External Location, Schema, and Table Grants based on workspace-wide Azure SPN mount points #94
• [FEATURE]: Create UC Storage Credential, Schema, and Table Grants based on AWS IAM role access #887
• [FEATURE]: Create UC Schema and Table Grants based on Legacy Table ACLs #340
• [EPIC] Multi-workspace installations #891
• [FEATURE]: Migrate AWS IAM Instance Profiles to UC Storage Credentials #862
• Migrate EXTERNAL tables from AWS Glue or other external HMS #337

Proposed Solution

• Get all permissions for all clusters and sql warehouses
• Create relevant permissions to External Locations that those mounts get converted into

[nfx]

use command executor to run scala code from from terraform exporter https://github.com/databricks/ucx/blob/main/tests/integration/test_installation.py#L76-L79

[zpappa]

Mounts should be going to volumes, not external locations.

@william-conti @nfx can we get a write up on this proposal?

[zpappa]

Does this only work for AWS?
What about if the mount point is in Azure or GCP?

[zpappa]

Is this part of the assessment? It's linked in a PRD as related to it, but it seems more as part of sync tables

Can you clarify @nfx?

[william-conti]

@zpappa Not very sure that mounts should automatically go to Volumes, what if there is an external table in there ? Could you please elaborate ?

Right now we've pushed the assessment part only

[pohlposition]

Is this Issue just for Mounts migrating to External Locations or does it also include Mounts migrating to Volumes?

I didn't see an issue for Volumes but can create one.

Both of these are dependent on Issue #100 (Create CREDENTIALs and EXTERNAL LOCATIONs)

The holistic set of things that need to be addressed are (in no particular order):

1. Crawl and record mount point paths (done)
2. Parse notebooks for absolute paths (s3://, s3a://, abfss://, etc.) and record them
3. Map all paths to either an external location or an external volume
4. Allow the path mapping to be modified if desired
5. Map External Locations to External Tables & old table namespace to UC table namespace (Given an HMS table name and database, get the new full name of table in UC #323)
6. Allow the Location to Table mapping to be modified if desired
7. Crawl and record instance profiles
8. Map all instance profiles to a CREDENTIAL
9. Highlight which paths might not have a CREDENTIAL mapping
10. Allow the credential mapping to be modified if desired
11. Indicate to the user which IAM Roles / Managed Identities / etc. need to be modified to (and how) for UC to assume them
12. Create CREDENTIALs
13. Create EXTERNAL LOCATIONs

• Need to figure out how to GRANT read vs write

14. Create EXTERNAL VOLUMEs

• Need to figure out how to GRANT read vs write

15. Create EXTERNAL TABLEs
16. Migrate TACL GRANTS to EXTERNAL TABLES
17. Have a separate method for users to move a VOLUME or TABLE to a different namespace if so desired (after workspace N has been upgraded)

[nfx]

@pritishpai when it's going to be done?

[pritishpai]

@pritishpai when it's going to be done?
Pushed latest changes to incorporate group ACLs over databases/tables over mounts.