Enhancement : adding support for Aurora postgres AWS IAM authentication #2650

fabio-rizzo-01 · 2025-09-22T14:20:13Z

Adding ability to connect to AWS Aurora Postgres using IAM authentication (password-less DB connection).

Tests carried out:

Deployed image to EKS cluster with IRSA (IAM Role for Service Account) enabled. Polaris is able to connect to aurora database
Deployed image to minikube to validate username/password auth to Postgress DB still works

Both test cases were successful.

Added example configuration in properties file

singhpk234

Thanks @fabio-rizzo-01 for taking this up !

singhpk234 · 2025-09-22T15:40:57Z

persistence/relational-jdbc/build.gradle.kts


  implementation(libs.smallrye.common.annotation) // @Identifier
  implementation(libs.postgresql)
+  implementation("software.amazon.awssdk:url-connection-client")


why do we need url-connection-client specifically ? can this not work with apache http client ?
per this should work - https://docs.quarkiverse.io/quarkus-amazon-services/dev/amazon-rds.html#_configuring_rds_clients

either should work yes, you want me to change it?

How can the user select which client to use?

My impression from the quarkiverse docs is that the type of HTTP client is a build time decision that is implied by the dependency included in the build (url-connection-client vs. apache-client). The latter is already included in Polaris.

My preference would be to use the apache-client for Aurora too and avoid url-connection-client.

WDYT? CC: @snazy

it is at build time, I'll update to apache if this is the preferred way.

+1 to using apache-client from my side (reconfirming) 🙂

I'll update the PR

singhpk234 · 2025-09-22T15:44:42Z

runtime/defaults/src/main/resources/application.properties

+quarkus.devservices.enabled=false
+
+
+#AWS IAM postgres example


wondering linking this additionally https://docs.quarkiverse.io/quarkus-amazon-services/dev/amazon-rds.html#_configuration_reference would be helpful too

singhpk234 · 2025-09-22T15:51:55Z

runtime/defaults/src/main/resources/application.properties

+#quarkus.rds.credentials-provider.aws.use-quarkus-client="true"
+#quarkus.rds.credentials-provider.aws.username=dbusername


seems like this is from this section - https://docs.quarkiverse.io/quarkus-amazon-services/dev/amazon-rds.html#_iam_authentication_support_with_quarkus_credential_provider ?
did you had to change the trust relationships etc and IAM policy etc, would be helpful adding something about that here too !

wondering if we can move this to https://polaris.apache.org/in-dev/unreleased/metastores/ ? cc @flyrain @dimas-b thoughts ?

do you want me to add a comment with more details around the policy for the pod? or in the docs? let me know what details you want me to add

Instructions in docs would be nice, but IMHO, it can be a separate PR

persistence/relational-jdbc/build.gradle.kts

dimas-b · 2025-09-24T15:43:17Z

Thanks for your contribution, @fabio-rizzo-01 !

flyrain

Thanks a lot @fabio-rizzo-01 for working on it. I'm glad this was figured out. The PR looks good to me overall! My only concern is that users won't be able to figure out how database configure work from a bag of comments from the file application.properties. Here are complexity users will face:

Two options to configure JDBC db. 1. Username/Password 2. AWS IAM AuthN
Two application.properties files(admin/server) to configure them. And they have to be consistent with each other.

I'd propose to move these configures to a new section under "Relational JDBC" of the page https://polaris.apache.org/in-dev/unreleased/metastores/. With certain description like there are two options. option 1 ..., option 2 .... WDYT?

flyrain · 2025-09-25T22:54:18Z

runtime/defaults/src/main/resources/application.properties

 polaris.persistence.type=in-memory
-# polaris.persistence.type=relational-jdbc
-
+#polaris.persistence.type=relational-jdbc


minor: this change seems unnecessary

true, @fabio-rizzo-01 : would you mind reverting this line for the sake of proper tranceability should someone need to see when and why it was added?

flyrain · 2025-09-25T22:55:07Z

runtime/defaults/src/main/resources/application.properties

-
+#polaris.persistence.type=relational-jdbc
 polaris.secrets-manager.type=in-memory
+#disable localstack


nit: we usually have a space between # and content.

@fabio-rizzo-01 : could you fix this together with the other minor comment?

fabio-rizzo-01 · 2025-09-26T07:08:31Z

Thanks a lot @fabio-rizzo-01 for working on it. I'm glad this was figured out. The PR looks good to me overall! My only concern is that users won't be able to figure out how database configure work from a bag of comments from the file application.properties. Here are complexity users will face:

Two options to configure JDBC db. 1. Username/Password 2. AWS IAM AuthN

Two application.properties files(admin/server) to configure them. And they have to be consistent with each other.

I'd propose to move these configures to a new section under "Relational JDBC" of the page https://polaris.apache.org/in-dev/unreleased/metastores/. With certain description like there are two options. option 1 ..., option 2 .... WDYT?

shall we merge this and I'll create another PR for the docs?

dimas-b · 2025-09-26T15:46:34Z

I think doing doc changes in a follow-up PR makes sense 👍

flyrain · 2025-09-26T16:41:32Z

Thanks, @fabio-rizzo-01, for addressing the comment. I'm good with a follow-up doc PR. In that case, I suggest we remove the configuration changes from both application.properties files. We add a new jdbc connection option in this PR. We got two options now. Users might confused by which option to choose, how they set it, especially since neither file includes a complete setup for both.

fabio-rizzo-01 · 2025-09-26T16:53:16Z

Thanks, @fabio-rizzo-01, for addressing the comment. I'm good with a follow-up doc PR. In that case, I suggest we remove the configuration changes from both application.properties files. We add a new jdbc connection option in this PR. We got two options now. Users might confused by which option to choose, how they set it, especially since neither file includes a complete setup for both.

I can't remove all the properties these 2 need to be there: quarkus.rds.sync-client.type=apache quarkus.devservices.enabled=false

the app won't run, because it needs to know which client to use to connect otherwise it defaults to the url-connection-client that is not there and for the second one it will try to start up localstack. So I can remove the db config but not these 2.

singhpk234 · 2025-09-26T16:58:17Z

I can't remove all the properties these 2 need to be there: quarkus.rds.sync-client.type=apache quarkus.devservices.enabled=false

This seems reasonable if this is a build time property and all others are runtime properties. my understaing we were concerned mostly from two ways to configure runtime properties for the datasources.

Orthogonally it would be really nice to add a CHANGELOG.md entry for this ! Thank you @fabio-rizzo-01 for working on this one.

dimas-b · 2025-09-26T17:04:27Z

Good point about CHANGELOG (I missed that 😅 ). @fabio-rizzo-01 : could you add a brief notice that the RDS plugin is now enabled and allows connecting to Aurora DB? I do not think it needs to be too verbose since doc changes are coming later.

fabio-rizzo-01 · 2025-09-26T17:15:42Z

Good point about CHANGELOG (I missed that 😅 ). @fabio-rizzo-01 : could you add a brief notice that the RDS plugin is now enabled and allows connecting to Aurora DB? I do not think it needs to be too verbose since doc changes are coming later.

properties removed and updated changelog file

flyrain

+1 Thanks a lot @fabio-rizzo-01 for adding this!

This PR contains no functional and no user-facing change. It is merely a refactor to better organize auth code. Summary of changes: - Moved all internal authentication components to the `org.apache.polaris.service.auth.internal` package and subpackages - Reduced visibility of utility classes - Renamed `TokenBroker` class hierarchy to stick to the naming standard: `<Algorithm>JWTBroker` - Introduced `@PolarisImmutable` whenever appropriate - Removed unused `NoneTokenBrokerFactory` (we already have `DisabledOAuth2ApiService`) - Removed unused `TokenBrokerFactoryConfig` Enhancement : adding support for Aurora postgres AWS IAM authentication (apache#2650) Add support for postgres AWS IAM authentication using the `apache-client` lib. Remove unused `name` arg from findCatalogByName in PolarisAdminService (apache#2691) * remove unused name param * Rename for better readability Fix a race condition in sendNotification where concurrent parent-namespace creation causes failures (apache#2693) * Fix a race condition in sendNotification where concurrent parent-namespace creation causes failures The semantics of the createNonExistingNamespaces method used during sendNotification were supposed to be "create if needed". However, the behavior ended up surfacing an AlreadyExistsException if multiple concurrent sendNotification attempts were made for a brand-new namespace (where the notifications may be different tables). This would cause a table sync to fail if a sibling table was being synced at the same time, even though the new table should successfully get created under the shared namespace. * Also better future-proof the createNamespaceInternal logic by explicitly checking for ENTITY_ALREADY_EXISTS, per review suggestion. Log a less scary message since it's not an error scenario type of race condition, per review suggestion Client: add credential reset option (apache#2698) * Client: add credential reset option * Client: add credential reset option * Client: add credential reset option * Add integration testing * Fix lint fix(deps): update dependency software.amazon.awssdk:bom to v2.34.5 (apache#2702) fix(deps): update dependency com.gradleup.shadow:shadow-gradle-plugin to v9.2.2 (apache#2661) added Aurora postgres to metastore documentation

* (Based on PR#2223)Support Namespace/Table level RBAC for external passthrough catalogs (apache#2673) Creates missing synthetic entities for securables in external passthrough catalogs. Based on Option 1 discussed in the RBAC section of catalog federation design doc. In the future, we could remove calls to PolarisEntity.Builder() and replace them with entities fetched from the remote catalog. (enabling Option 2). --------- Co-authored-by: Pooja Nilangekar <poojan@umd.edu> * Docs: Add more details about v1 schema user to upgrade from 1.0 to 1.1 (apache#2674) * Site: The link https://iceberg.apache.org/concepts/catalog/ doesn't exist anymore. (apache#2683) * Docs: Add analytics for polaris.apache.org (apache#2676) * Make ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS configurable per catalog (apache#2688) * Update ENABLE_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS to be configurable per catalog * chore(deps): update postgres docker tag to v18 (apache#2692) * fix(deps): update dependency org.eclipse.persistence:eclipselink to v4.0.8 (apache#2682) * fix(deps): update dependency org.apache.logging.log4j:log4j-core to v2.25.2 (apache#2646) * chore(deps): update dependency openapi-generator-cli to v7.15.0 (apache#2410) * chore(deps): update dependency io.quarkus to v3.27.0 (apache#2663) Co-authored-by: Mend Renovate <bot@renovateapp.com> * Publish Develocity builds scans for PRs and local use (apache#2596) This PR enables Develocity build scans for all PRs and contributors w/o an Apache account. CI build scans in the `apache/polaris` repo against branches and tags and having access to the ASF's Develocity secret continue to publish to the ASF's Develocity instance (no behavioral change). All other build scans are published to Gradle's public Develocity instance: - Build scans from local developer (non-CI) runs are only published, if Gradle is invoked with the `--scan` option. - Build scans from or targeting another repository than `apache/polaris` do need be enabled explicity by accepting Gradle's terms of service, via a repository variable, because this is a decision of the owner of a repository. Advanced options to configure another Develocity server or project-ID are available (for non-`apache/polaris` repositories). Detailed instructions in the `README.md`. * Fix & enhancements to the Events API hierarchy (apache#2629) Summary of changes: - Turned `PolarisEventListener` into an interface to facilitate implementation / mocking - Added missing `implements PolarisEvent` to many event records - Removed unused method overrides - Added missing method overrides to `TestPolarisEventListener` * fix(deps): update dependency org.kordamp.gradle:jandex-gradle-plugin to v2.3.0 (apache#2694) * Auth: reorganize internal authentication components (apache#2634) This PR contains no functional and no user-facing change. It is merely a refactor to better organize auth code. Summary of changes: - Moved all internal authentication components to the `org.apache.polaris.service.auth.internal` package and subpackages - Reduced visibility of utility classes - Renamed `TokenBroker` class hierarchy to stick to the naming standard: `<Algorithm>JWTBroker` - Introduced `@PolarisImmutable` whenever appropriate - Removed unused `NoneTokenBrokerFactory` (we already have `DisabledOAuth2ApiService`) - Removed unused `TokenBrokerFactoryConfig` * Enhancement : adding support for Aurora postgres AWS IAM authentication (apache#2650) Add support for postgres AWS IAM authentication using the `apache-client` lib. * Remove unused `name` arg from findCatalogByName in PolarisAdminService (apache#2691) * remove unused name param * Rename for better readability * Fix a race condition in sendNotification where concurrent parent-namespace creation causes failures (apache#2693) * Fix a race condition in sendNotification where concurrent parent-namespace creation causes failures The semantics of the createNonExistingNamespaces method used during sendNotification were supposed to be "create if needed". However, the behavior ended up surfacing an AlreadyExistsException if multiple concurrent sendNotification attempts were made for a brand-new namespace (where the notifications may be different tables). This would cause a table sync to fail if a sibling table was being synced at the same time, even though the new table should successfully get created under the shared namespace. * Also better future-proof the createNamespaceInternal logic by explicitly checking for ENTITY_ALREADY_EXISTS, per review suggestion. Log a less scary message since it's not an error scenario type of race condition, per review suggestion * Client: add credential reset option (apache#2698) * Client: add credential reset option * Client: add credential reset option * Client: add credential reset option * Add integration testing * Fix lint * fix(deps): update dependency software.amazon.awssdk:bom to v2.34.5 (apache#2702) * fix(deps): update dependency com.gradleup.shadow:shadow-gradle-plugin to v9.2.2 (apache#2661) * Support S3 storage that does not have STS (apache#2672) * Support S3 storage that does not have STS This change is backward compatible with old catalogs that have storage configuration for S3 systems with STS. * Add new property to S3 storage config: `stsUnavailable` (defaults to "available"). * Do not call STS when unavailable in `AwsCredentialsStorageIntegration`, but still put other properties (e.g. s3.endpoint) into `AccessConfig` Relates to apache#2615 Relates apache#2207 * Docs/improve idp documentation (apache#2695) * Fix Github links in IDP documentation * Separate IDP docs for usage and development * - Add telemetry config example - Fix link to getting started from landing page - Fix mentioning role-arn as required * Fix some relative links (local Hugo resolves them properly, but PR auto checks still fails) * Docs: narrow down --role-arn usage for AWS S3 only; fix a link in keycloak guide. * Docs: fix a link in keycloak guide. * chore(deps): update gradle/actions digest to 748248d (apache#2708) * Client: fix integration testing (apache#2700) * Add fallback in case the VERSION table is not present (apache#2653) * initial commit * wire up * pastefix * change to postgres specific code * [Catalog Federation] Add feature flag to disallow setting sub-RBAC for federated catalog at catalog level (apache#2696) In apache#2688 (comment), we've identified that configuring polaris.config.enable-sub-catalog-rbac-for-federated-catalogs at catalog level should not be allowed in all cases, especially when the owner is not the same subject as the catalog user or admin. This PR add a feature flag, ALLOW_SETTING_SUB_CATALOG_RBAC_FOR_FEDERATED_CATALOGS to allow owner to disable catalog level setting polaris.config.enable-sub-catalog-rbac-for-federated-catalogs * Fix `delegationModes` parameter propagation in `createTableStaged()` (apache#2713) This is follow-up bugfix for apache#2589 The bugfix part apache#2711 is extracted here since apache#2711 proved to be non-trivial and may require extra time. * Use the `delegationModes` method parameter as intended (as opposed to a local constant). * Generate Request IDs (if not specified); Return Request ID as a Header (apache#2602) * fix(deps): update dependency org.junit:junit-bom to v5.14.0 (apache#2715) * NoSQL persistence: add Java/Vert.X executor abstraction layer (apache#2527) Provides an abstraction to submit asynchronous tasks, optionally with a delay or delay + repetition and implementations based on Java's `ThreadPoolExecutor` and Vert.X. * Fix RDS devservices config + adopt for `:polaris-admin:test` (apache#2723) Changes: * Disables devservices for `:polaris-admin` tests as well, which is necessary to _not_ spin up test containers. * Use the explicit devservices-config as everywhere else. The first bullet point can cause excessive memory usage, especially with more test classes, eventually killing the whole GH runner. * fix(deps): update dependency io.smallrye:jandex to v3.5.0 (apache#2722) * fix(deps): update dependency org.jboss.weld:weld-junit5 to v5.0.2.final (apache#2721) * chore(deps): update quay.io/keycloak/keycloak docker tag to v26.4.0 (apache#2719) * Last merged commit 4024557 * NoSQL: Minor-ish changes to "nodes" projects Adopt nodes projects to OSS PR content * NoSQL: adapt to async package rename * Build: remove unnecessary explicit vertx-core dependency The async-vertx implementation should not propagate a different Vert.X dependency than Quarkus provides. This wouldn't be an issue if we could just use `enforcedPlatform()` for all Quarkus-builds, but sadly we cannot for the spark-plugin-inttests. --------- Co-authored-by: Honah (Jonas) J. <honahx@apache.org> Co-authored-by: Pooja Nilangekar <poojan@umd.edu> Co-authored-by: Prashant Singh <35593236+singhpk234@users.noreply.github.com> Co-authored-by: JB Onofré <jbonofre@apache.org> Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: Alexandre Dutra <adutra@apache.org> Co-authored-by: fabio-rizzo-01 <fabio.rizzocascio@jpmorgan.com> Co-authored-by: Dennis Huo <7410123+dennishuo@users.noreply.github.com> Co-authored-by: Yong Zheng <yongzheng0809@gmail.com> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: olsoloviov <40199597+olsoloviov@users.noreply.github.com> Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com> Co-authored-by: Adnan Hemani <adnan.h@berkeley.edu>

adding support for postgres AWS IAM authentication

cc08657

github-project-automation bot added this to Basic Kanban Board Sep 22, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Sep 22, 2025

fabio-rizzo-01 added 2 commits September 22, 2025 15:54

updated jdbc url

0e6a7b8

removed example yml file used in ci

a8d5b12

singhpk234 self-requested a review September 22, 2025 15:35

singhpk234 reviewed Sep 22, 2025

View reviewed changes

moving to apache-client lib

1da5154

dimas-b reviewed Sep 24, 2025

View reviewed changes

persistence/relational-jdbc/build.gradle.kts Outdated Show resolved Hide resolved

moving dependencies to runtime/common

f98b87f

dimas-b previously approved these changes Sep 24, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Sep 24, 2025

flyrain reviewed Sep 25, 2025

View reviewed changes

updated comment formatting

49c8d55

fabio-rizzo-01 dismissed dimas-b’s stale review via 49c8d55 September 26, 2025 16:03

dimas-b previously approved these changes Sep 26, 2025

View reviewed changes

singhpk234 modified the milestones: 1.1.0, 1.2.0 Sep 26, 2025

updated change log and removed runtime properties

c9f5cb4

fabio-rizzo-01 dismissed dimas-b’s stale review via c9f5cb4 September 26, 2025 17:14

flyrain approved these changes Sep 26, 2025

View reviewed changes

dimas-b approved these changes Sep 26, 2025

View reviewed changes

dimas-b merged commit 965bd53 into apache:main Sep 26, 2025
14 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Sep 26, 2025

		#quarkus.rds.credentials-provider.aws.use-quarkus-client="true"
		#quarkus.rds.credentials-provider.aws.username=dbusername

Enhancement : adding support for Aurora postgres AWS IAM authentication #2650

Enhancement : adding support for Aurora postgres AWS IAM authentication #2650

Uh oh!

Conversation

fabio-rizzo-01 commented Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

singhpk234 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dimas-b commented Sep 24, 2025

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fabio-rizzo-01 commented Sep 26, 2025

Uh oh!

dimas-b commented Sep 26, 2025

Uh oh!

flyrain commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fabio-rizzo-01 commented Sep 26, 2025

Uh oh!

singhpk234 commented Sep 26, 2025

Uh oh!

dimas-b commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fabio-rizzo-01 commented Sep 26, 2025

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fabio-rizzo-01 commented Sep 22, 2025 •

edited

Loading

dimas-b Sep 22, 2025 •

edited

Loading

flyrain commented Sep 26, 2025 •

edited

Loading

dimas-b commented Sep 26, 2025 •

edited

Loading