Skip to content

Commit

Permalink
Added AWS IAM role support to `databricks labs ucx create-uber-princi…
Browse files Browse the repository at this point in the history
…pal` command (#993)

## Changes
Added CLI command `databricks labs ucx create-uber-principal` for
creating uber-IAM profile for performing external table migration on
AWS.

Logic:
* Stop if UCX migration cluster policy is not found
* Collect paths of all locations/paths used in tables (call
`external_location.snapshot`)
* If cluster policy has an existing iam instance profile/role specified,
then add/update migration policy providing access to the locations
* If cluster policy does not have iam instance profile/role specified,
then create new iam profile/role and migration policy, and add it to the
cluster policy

### Linked issues

Resolves #879 

Related issues:
- #976
- #693

### Functionality 

- [x] added new CLI command

### Tests

- [x] manually tested
- [x] added unit tests

### TODO
- [x] added integration tests
- [x] verified on staging environment (screenshot attached)

---------

Co-authored-by: Vuong <vuong.nguyen@databricks.com>
  • Loading branch information
2 people authored and dmoore247 committed Mar 23, 2024
1 parent 9bd1df3 commit a97df8b
Show file tree
Hide file tree
Showing 14 changed files with 1,347 additions and 872 deletions.
7 changes: 5 additions & 2 deletions labs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -110,10 +110,13 @@ commands:

- name: create-uber-principal
description: For azure cloud, creates a service principal and gives STORAGE BLOB READER access on all the storage account
used by tables in the workspace and stores the spn info in the UCX cluster policy.
used by tables in the workspace and stores the spn info in the UCX cluster policy. For aws,
it identifies all s3 buckets used by the Instance Profiles configured in the workspace.
flags:
- name: subscription-id
description: Subscription to scan storage account in
- name: aws-profile
description: AWS Profile to use for authentication

- name: validate-groups-membership
description: Validate groups to check if the groups at account level and workspace level have different memberships
Expand All @@ -139,4 +142,4 @@ commands:

- name: create-catalogs-schemas
description: Create UC external catalogs and schemas based on the destinations created from create_table_mapping command.
This command is supposed to be run before migrating tables to UC.
This command is supposed to be run before migrating tables to UC.
Loading

0 comments on commit a97df8b

Please sign in to comment.