Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added support for crawling grants and applying Hive Metastore UDF ACLs #812

Merged
merged 13 commits into from
Jan 18, 2024
Merged

Added support for crawling grants and applying Hive Metastore UDF ACLs #812

merged 13 commits into from
Jan 18, 2024

Conversation

mwojtyczka
Copy link
Contributor

@mwojtyczka mwojtyczka commented Jan 18, 2024 •
edited
Loading

Changes

Added support for crawling grants and applying Hive Metastore UDF ACLs.

Resolves #808

Functionality

  • added new crawler for udfs
  • added handling of udfs to the existing TableAclSupport
  • added new column to inventory permissions table for udfs

Tests

  • tested manually
  • added unit tests
  • added integration tests

@mwojtyczka mwojtyczka requested review from nfx, william-conti and a team January 18, 2024 14:27
@codecov Codecov
Copy link

codecov bot commented Jan 18, 2024 •
edited
Loading

Codecov Report

Attention: 7 lines in your changes are missing coverage. Please review.

Comparison is base (af80620) 84.07% compared to head (1969124) 84.20%.

Files Patch % Lines
src/databricks/labs/ucx/hive_metastore/udfs.py 91.66% 2 Missing and 3 partials ⚠️
src/databricks/labs/ucx/runtime.py 33.33% 2 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #812      +/-   ##
==========================================
+ Coverage   84.07%   84.20%   +0.13%     
==========================================
  Files          39       40       +1     
  Lines        4872     4952      +80     
  Branches      913      924      +11     
==========================================
+ Hits         4096     4170      +74     
- Misses        564      567       +3     
- Partials      212      215       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

nfx
nfx approved these changes Jan 18, 2024
View reviewed changes
Copy link
Collaborator

@nfx nfx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall lgtm, couple of nits though

src/databricks/labs/ucx/hive_metastore/udfs.py Outdated Show resolved Hide resolved
src/databricks/labs/ucx/hive_metastore/udfs.py Outdated Show resolved Hide resolved
src/databricks/labs/ucx/mixins/fixtures.py Outdated Show resolved Hide resolved
src/databricks/labs/ucx/mixins/fixtures.py Outdated Show resolved Hide resolved
tests/integration/hive_metastore/test_grants.py Outdated Show resolved Hide resolved
@mwojtyczka @nfx
Update src/databricks/labs/ucx/hive_metastore/udfs.py
5a31799 
review feedback implementation

Co-authored-by: Serge Smertin <259697+nfx@users.noreply.github.com>
@mwojtyczka mwojtyczka changed the title [FEATURE]: Crawl and Apply grants for Hive Metastore UDFs Crawl Hive Metastore UDFs and apply Table Acl grants for groups migration Jan 18, 2024
@mwojtyczka mwojtyczka changed the title Crawl Hive Metastore UDFs and apply Table Acl grants for groups migration Crawl Hive Metastore UDFs and apply Table Acl grants Jan 18, 2024
@mwojtyczka mwojtyczka changed the title Crawl Hive Metastore UDFs and apply Table Acl grants Crawl and apply Table Acl grants for Hive Metastore UDFs Jan 18, 2024
@mwojtyczka mwojtyczka changed the title Crawl and apply Table Acl grants for Hive Metastore UDFs Added support for crawling Hive Metastore Udf grants and applying ACLs Jan 18, 2024
@mwojtyczka mwojtyczka requested a review from nfx January 18, 2024 21:43
@mwojtyczka mwojtyczka changed the title Added support for crawling Hive Metastore Udf grants and applying ACLs Added support for crawling Hive Metastore UDF grants and applying ACLs Jan 18, 2024
@mwojtyczka mwojtyczka changed the title Added support for crawling Hive Metastore UDF grants and applying ACLs Added support for crawling grants and applying Hive Metastore UDF ACLs Jan 18, 2024
nfx
nfx approved these changes Jan 18, 2024
View reviewed changes
Copy link
Collaborator

@nfx nfx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@nfx nfx merged commit c9ee832 into main Jan 18, 2024
7 checks passed
@nfx nfx deleted the feature/functions_crawl_apply branch January 18, 2024 21:53
nfx added a commit that referenced this pull request Jan 19, 2024
@nfx
Release v0.10.0
10ea0b1 
* Added `databricks labs ucx validate-groups-membership` command to validate groups to see if they have same membership across acount and workspace level ([#772](#772)).
* Added baseline for getting Azure Resource Role Assignments ([#764](#764)).
* Added issue and pull request templates ([#791](#791)).
* Added linked issues to PR template ([#793](#793)).
* Added optional `debug_truncate_bytes` parameter to the config and extend the default log truncation limit ([#782](#782)).
* Added support for crawling grants and applying Hive Metastore UDF ACLs ([#812](#812)).
* Changed Python requirement from 3.10.6 to 3.10 ([#805](#805)).
* Extend error handling of delta issues in crawlers and hive metastore ([#795](#795)).
* Fixed `databricks labs ucx repair-run` command to execute correctly ([#801](#801)).
* Fixed handling of `DELTASHARING` table format ([#802](#802)).
* Fixed listing of workflows via CLI ([#811](#811)).
* Fixed logger import path for DEBUG notebook ([#792](#792)).
* Fixed move table command to delete table/view regardless if permissions are present, skipping corrupted tables when crawling table size and making existing tests more stable ([#777](#777)).
* Fixed the issue of `databricks labs ucx installations` and `databricks labs ucx manual-workspace-info` ([#814](#814)).
* Increase the unit test coverage for cli.py ([#800](#800)).
* Mount Point crawler lists /Volume with four variations which is confusing ([#779](#779)).
* Updated README.md to remove mention of deprecated install.sh ([#781](#781)).
* Updated `bug` issue template ([#797](#797)).
* Fixed writing log readme in multiprocess safe way ([#794](#794)).
@nfx nfx mentioned this pull request Jan 19, 2024
Merged
nfx added a commit that referenced this pull request Jan 19, 2024
@nfx
Release v0.10.0 (#819)
38256dc 
* Added `databricks labs ucx validate-groups-membership` command to
validate groups to see if they have same membership across acount and
workspace level
([#772](#772)).
* Added baseline for getting Azure Resource Role Assignments
([#764](#764)).
* Added issue and pull request templates
([#791](#791)).
* Added linked issues to PR template
([#793](#793)).
* Added optional `debug_truncate_bytes` parameter to the config and
extend the default log truncation limit
([#782](#782)).
* Added support for crawling grants and applying Hive Metastore UDF ACLs
([#812](#812)).
* Changed Python requirement from 3.10.6 to 3.10
([#805](#805)).
* Extend error handling of delta issues in crawlers and hive metastore
([#795](#795)).
* Fixed `databricks labs ucx repair-run` command to execute correctly
([#801](#801)).
* Fixed handling of `DELTASHARING` table format
([#802](#802)).
* Fixed listing of workflows via CLI
([#811](#811)).
* Fixed logger import path for DEBUG notebook
([#792](#792)).
* Fixed move table command to delete table/view regardless if
permissions are present, skipping corrupted tables when crawling table
size and making existing tests more stable
([#777](#777)).
* Fixed the issue of `databricks labs ucx installations` and `databricks
labs ucx manual-workspace-info`
([#814](#814)).
* Increase the unit test coverage for cli.py
([#800](#800)).
* Mount Point crawler lists /Volume with four variations which is
confusing ([#779](#779)).
* Updated README.md to remove mention of deprecated install.sh
([#781](#781)).
* Updated `bug` issue template
([#797](#797)).
* Fixed writing log readme in multiprocess safe way
([#794](#794)).
dmoore247 pushed a commit that referenced this pull request Mar 23, 2024
@nfx @dmoore247
Release v0.10.0 (#819)
d30eb4d 
* Added `databricks labs ucx validate-groups-membership` command to
validate groups to see if they have same membership across acount and
workspace level
([#772](#772)).
* Added baseline for getting Azure Resource Role Assignments
([#764](#764)).
* Added issue and pull request templates
([#791](#791)).
* Added linked issues to PR template
([#793](#793)).
* Added optional `debug_truncate_bytes` parameter to the config and
extend the default log truncation limit
([#782](#782)).
* Added support for crawling grants and applying Hive Metastore UDF ACLs
([#812](#812)).
* Changed Python requirement from 3.10.6 to 3.10
([#805](#805)).
* Extend error handling of delta issues in crawlers and hive metastore
([#795](#795)).
* Fixed `databricks labs ucx repair-run` command to execute correctly
([#801](#801)).
* Fixed handling of `DELTASHARING` table format
([#802](#802)).
* Fixed listing of workflows via CLI
([#811](#811)).
* Fixed logger import path for DEBUG notebook
([#792](#792)).
* Fixed move table command to delete table/view regardless if
permissions are present, skipping corrupted tables when crawling table
size and making existing tests more stable
([#777](#777)).
* Fixed the issue of `databricks labs ucx installations` and `databricks
labs ucx manual-workspace-info`
([#814](#814)).
* Increase the unit test coverage for cli.py
([#800](#800)).
* Mount Point crawler lists /Volume with four variations which is
confusing ([#779](#779)).
* Updated README.md to remove mention of deprecated install.sh
([#781](#781)).
* Updated `bug` issue template
([#797](#797)).
* Fixed writing log readme in multiprocess safe way
([#794](#794)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nfx nfx nfx approved these changes

@william-conti william-conti Awaiting requested review from william-conti

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEATURE]: Crawl and Apply grants for SQL UDFs
2 participants
@mwojtyczka @nfx