Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement] Support ADLS storage for Gravitino Iceberg catalog and Spark connector #5954

Closed
FANNG1 opened this issue Dec 23, 2024 · 0 comments · Fixed by #5952
Closed

[Improvement] Support ADLS storage for Gravitino Iceberg catalog and Spark connector #5954

FANNG1 opened this issue Dec 23, 2024 · 0 comments · Fixed by #5952
Assignees
@FANNG1
Labels
0.8.0 Release v0.8.0 improvement Improvements on everything

Comments

@FANNG1
Copy link
Contributor

FANNG1 commented Dec 23, 2024

What would you like to be improved?

Support ADLS storage for Gravitino Iceberg catalog and Spark connector

How should we improve?

add ADLS storage support
@FANNG1 FANNG1 added the improvement Improvements on everything label Dec 23, 2024
@FANNG1 FANNG1 closed this as completed Dec 23, 2024
@FANNG1 FANNG1 reopened this Dec 23, 2024
@FANNG1 FANNG1 self-assigned this Dec 23, 2024
@FANNG1 FANNG1 mentioned this issue Dec 23, 2024
Merged
@jerryshao jerryshao added the 0.8.0 Release v0.8.0 label Dec 24, 2024
Abyss-lord pushed a commit to Abyss-lord/gravitino that referenced this issue Dec 29, 2024
@FANNG1 @Abyss-lord
[apache#5954] feat(iceberg): Supports ADLS for Iceberg catalog and sp…
f6806bb 
…ark connector (apache#5952)

### What changes were proposed in this pull request?

1. Most code work is implemented in apache#5938 apache#5737 including catalog
properties convert and add Iceberg azure bundle jar, this PR mainly
about test and document.
2. Remove hidden properties of the cloud secret key from the Iceberg
catalog, as Gravitino doesn't have an unified security management yet
and Iceberg REST server need to fetch catalog cloud properties to
initiate `IcebergWrapper` dymaticly. Another benefit is spark connector
does not need to specify the secret key explictly.

Supports ADLS for Iceberg catalog and spark connector

### Why are the changes needed?
Fix: apache#5954 

### Does this PR introduce _any_ user-facing change?
Yes, the user no need to specify the cloud secret key in spark
connector.

### How was this patch tested?
test in local enviroment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@FANNG1 FANNG1

Labels
0.8.0 Release v0.8.0 improvement Improvements on everything
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[#5954] feat(iceberg): Supports ADLS for Iceberg catalog and spark connector FANNG1/gravitino
2 participants
@jerryshao @FANNG1