Refactor: Use per-request STS credentials #1629

dimas-b · 2025-05-20T20:35:14Z

No functional changes.

This is mostly to allow more storage integration
flexibility in downstream build.

This might also be useful for non-AWS storage.

singhpk234 · 2025-05-21T04:55:10Z

...core/src/main/java/org/apache/polaris/core/storage/aws/AwsCredentialsStorageIntegration.java

can we call this credentialProvider instead ?

singhpk234 · 2025-05-21T05:00:06Z

service/common/src/main/java/org/apache/polaris/service/storage/StorageConfiguration.java

any reason to keep Logger getter in this method rather outside ?

oops, it was just a copy-paste.... will fix.

Actually, refactoring this in current PR is awkward - I would not like to add a logger field to this interface. Adding a class looks like an overkill.

Would you mind if I moved this to a production readiness check in a follow-up PR?

sounds fair !

singhpk234 · 2025-05-21T05:05:22Z

service/common/src/main/java/org/apache/polaris/service/storage/StorageConfiguration.java

[orthogonal] Thoughts on supportingdynamically loading credential provider implementation ?

The idea (for now) is for custom builds to manage cred. provided via a custom impl. of PolarisStorageIntegrationProvider.

I suppose this will evolve as we move forward with @XJDKC 's proposal: https://lists.apache.org/thread/ph1tvn3lzvn8kh8fnhc6k585qmw2m12r

Yes, I have a branch that's based on the proposal, and it's working. I'm trying to clean it and add more tests, then I will open a PR. Welcome to review the PR and leave your comments!
https://github.com/XJDKC/polaris/tree/rxing-catalog-federation-sigv4-poc

snazy · 2025-05-21T07:52:58Z

service/common/src/testFixtures/java/org/apache/polaris/service/TestServices.java

Wildcard import?

good catch - fixed

No functional changes. This is mostly to allow more storage integration flexibility in downstream build. This might also be useful for non-AWS storage.

snazy · 2025-05-23T10:29:25Z

...kus/service/src/test/java/org/apache/polaris/service/quarkus/admin/PolarisAuthzTestBase.java

-import java.util.List;
-import java.util.Map;
-import java.util.Set;
+import java.util.*;


ping: wildcard import

snazy

no more wildcard imports -> +1 ;)

* fix(nightly-CI): Do not publish snapshots from forks (apache#1635) Adopt the `Nightly Build` workflow to not (try to) publish every night from forks. * main: Update dependency io.smallrye.config:smallrye-config-core to v3.13.0 (apache#1637) * Use echo to print script errors (apache#1648) * [HOTFIX] QUICKSTART (apache#1646) The change adds the following to fix Quick start experience : [1] ENV variables required by common assets after apache#1522 [2] New configs required to enable FILE based sources apache#1649 Co-authored-by: singhpk234 <singhpk234@users.noreply.github.com> Co-authored-by: pjanuario <pjanuario@users.noreply.github.com> * main: Update dependency gradle to v8.14.1 (apache#1652) * main: Update dependency gradle to v8.14.1 * Re-adopt PR to the project's needs --------- Co-authored-by: Robert Stupp <snazy@snazy.de> * [Policy Store] Add policyTypeCode to Slice/Index for Future Filtering Support and Update Policy Persistence Method (apache#1628) This PR adds policyTypeCode to the in-memory tree map store's slice and the SQL index on policy_mapping_records (already done in JDBC in apache#1468). This prepares for future features that need to filter efficiently by policy type, like listing all entities with a data compaction policy. It also updates the loadAllTargetsOnPolicy method to accept policyTypeCode, enabling it to use the new index for better performance. * fix(test): Do not let some more tests spam `/tmp` (apache#1651) * fix(test): Do not let some more tests not spam `/tmp` * `PolarisRestCatalogViewFileIntegrationTest` * `FileIOExceptionsTest` * `PolarisRestCatalogViewFileIntegrationTest` Changes the tests to leverage JUnit's `@TempDir`. Simplifies `PolarisEclipseLinkMetaStoreManagerTest` * review: rename the (now) abstract class * fix(testing): Do not let PolarisOverlappingTableTest spam `/tmp` (apache#1641) Changes the test to leverage JUnit's `@TempDir`. * Add CATALOG_MANAGE_METADATA to super privilege set of policy attachment privileges (apache#1643) * Fix quickstart doc with docker compose (apache#1610) * main: Update dependency boto3 to v1.38.22 (apache#1657) * Refactor IcebergCatalog to isolate internal state (apache#1659) Following up on apache#1694 * Restore `private` scope on internal fields in `IcebergCatalog` * Use a test-only setter instead of sub-classing to manage injecting test FileIO implementations * Refactor: Use per-request STS credentials (apache#1629) * Refactor: Use per-request STS credentials No functional changes. This is mostly to allow more storage integration flexibility in downstream build. This might also be useful for non-AWS storage. * fix and enforce more errorprone checks (apache#1663) enforces the following checks: https://errorprone.info/bugpattern/ObjectsHashCodePrimitive https://errorprone.info/bugpattern/OptionalMapToOptional https://errorprone.info/bugpattern/StringCharset https://errorprone.info/bugpattern/VariableNameSameAsType * Create a wrapper script to generate python client; regenerate the python client (apache#1347) As noted in apache#755 and elsewhere, the generated types in client/python are currently out of date. This introduces a script to regenerate them and a gradle task to run that script. I've also run the script, which necessitated several things to get tests passing: 1. There were small nonfunctional spec changes needed in order to keep the Python client working 2. The CLI and its tests required a few fixes to work with the updated Python client 3. Many of the regtests required fixes to work with the updated Python client * [Python Client] CI for Python client (Continue PR#1096) (apache#1639) Adds CI for python client. It does not include caching poetry step for now since we do not have poetry.lock (it is in .gitignore), see relevant discussion in: apache#1102 (comment), apache#1096 (comment), we can add that later * main: Update actions/setup-python action to v5 (apache#1671) * main: Update actions/checkout action to v4 (apache#1670) * main: Update python Docker tag to v3.13 (apache#1669) * main: Update dependency pytest to ~=7.4.4 (apache#1668) * main: Update dependency software.amazon.awssdk:bom to v2.31.50 (apache#1677) * main: Update dependency boto3 to v1.38.23 (apache#1667) * feat(build): make archive builds reproducible (apache#1664) See https://docs.gradle.org/current/userguide/working_with_files.html#sec:reproducible_archives * main: Update dependency io.prometheus:prometheus-metrics-exporter-servlet-jakarta to v1.3.8 (apache#1679) * NoSQL: adapt to change on oss/main * INFO: Last merged commit: 6ef8b3e --------- Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: ModEtchFill <50123102+ModEtchFill@users.noreply.github.com> Co-authored-by: Prashant Singh <35593236+singhpk234@users.noreply.github.com> Co-authored-by: singhpk234 <singhpk234@users.noreply.github.com> Co-authored-by: pjanuario <pjanuario@users.noreply.github.com> Co-authored-by: Honah (Jonas) J. <honahx@apache.org> Co-authored-by: MonkeyCanCode <yongzheng0809@gmail.com> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@dremio.com> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: Christopher Lambert <xn137@gmx.de> Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com>

dimas-b requested review from HonahX, MonkeyCanCode, RussellSpitzer, adutra, ashvina, collado-mike, dennishuo, ebyhr, eric-maynard, flyrain, jackye1995, jbonofre, snazy, takidau and vvcephei as code owners May 20, 2025 20:35

github-project-automation bot added this to Basic Kanban Board May 20, 2025

dimas-b requested review from pingtimeout and singhpk234 as code owners May 20, 2025 20:35

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board May 20, 2025

singhpk234 reviewed May 21, 2025

View reviewed changes

snazy reviewed May 21, 2025

View reviewed changes

dimas-b added 3 commits May 22, 2025 09:42

Refactor: Use per-request STS credentials

775750e

No functional changes. This is mostly to allow more storage integration flexibility in downstream build. This might also be useful for non-AWS storage.

review: fix imports

744a40d

review: rename vars to credentialsProvider

649c173

dimas-b force-pushed the sts-credentials branch from ef72a86 to 649c173 Compare May 22, 2025 13:42

dimas-b requested a review from ajantha-bhat as a code owner May 22, 2025 13:42

snazy reviewed May 23, 2025

View reviewed changes

review: fix more imports

27c7d84

snazy approved these changes May 23, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board May 23, 2025

dimas-b enabled auto-merge (squash) May 23, 2025 14:02

dimas-b merged commit 0bdc3f4 into apache:main May 23, 2025
6 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board May 23, 2025

dimas-b deleted the sts-credentials branch June 19, 2025 23:41

Refactor: Use per-request STS credentials #1629

Refactor: Use per-request STS credentials #1629

Uh oh!

Conversation

dimas-b commented May 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dimas-b May 22, 2025 •

edited

Loading