feat(helm): Add support for external authentication #2104

adutra · 2025-07-15T16:19:18Z

No description provided.

adutra · 2025-07-16T09:38:46Z

helm/polaris/templates/_helpers.tpl

+- Escapes all backslashes.
+- Escapes all key termination charaters: '=', ':' and whitespace.
 */}}
-{{- define "polaris.dictToString" -}}


The template polaris.dictToString was unused.

adutra · 2025-07-16T09:41:28Z

helm/polaris/templates/_helpers.tpl

+{{- $_ = set $map (printf "%s.token-broker.max-token-generation" $prefix) (dig "tokenBroker" "maxTokenGeneration" "PT1H" $auth) -}}
+{{- $secretName := dig "tokenBroker" "secret" "name" "" $auth -}}
+{{- if $secretName -}}
+{{- $subpath := empty $realm | ternary "" (printf "%s/" (urlquery $realm)) -}}


This is a best-effort to produce valid filesystem paths if the realm ID contains weird characters. The only function available in Helm that vaguely serves this purpose is urlquery.

Another option is to output the hash the realm ID. It's safer, but less readable. Lmk if that's preferable.

urlquery LGTM 👍

adutra · 2025-07-16T10:32:49Z

helm/polaris/templates/_helpers.tpl


 {{/*
-Convert a dict into a string formed by a comma-separated list of key-value pairs: key1=value1,key2=value2, ...
+Escapes a property key to be used in a configmap, conforming with the Java parsisng rules for


We were missing until now a proper escape mechanism for config option keys. Before this change, any realm containing a key termination character such as a space or =, would produce invalid configuration.

dimas-b

LGTM overall 👍

dimas-b · 2025-07-16T13:54:46Z

helm/polaris/values.yaml

  tokenService:
    type: default
-  # -- The type of token broker to use. Two built-in types are supported: rsa-key-pair and symmetric-key.
+  # -- The `TokenBroker` implementation to use. Two built-in types are supported: rsa-key-pair and symmetric-key.


nit: TokenBroker is not applicable to the "external" auth type, right? Should we support disabling it?

tokenBroker and tokenService are indeed only relevant when using internal (or mixed) authentication.

Setting these options when using external auth doesn't hurt, but in practice, the configmap won't even contain those config options if the authentication type is external.

ah, thx for the explanation!.. I missed that behaviour 🤦

dimas-b · 2025-07-16T13:56:40Z

helm/polaris/values.yaml

+  client:
+    # -- The client ID to use when authenticating with the authentication server.
+    id: polaris
+    # -- The secret to pull the client secret from.


nit: the OIDC secret is optional, right?

Yes. If secret.name is nil, no client secret is included in the configuration. This follows a pattern that is already used for other secrets.

Do you think we could mention that in the doc comment for clarity? Current text feels like the secret might be required 🤔

Done. I seized the opportunity to clarify tokenBroker and tokenService as well.

dimas-b

LGTM 👍

MonkeyCanCode · 2025-07-16T19:07:54Z

helm/polaris/README.md


 | Key | Type | Default | Description |
 |-----|------|---------|-------------|
 | advancedConfig | object | `{}` | Advanced configuration. You can pass here any valid Polaris or Quarkus configuration property. Any property that is defined here takes precedence over all the other configuration values generated by this chart. Properties can be passed "flattened" or as nested YAML objects (see examples below). Note: values should be strings; avoid using numbers, booleans, or other types. |


Can we copy this to site/content/in-dev/unreleased/helm.md as well?

@MonkeyCanCode isn't this copy done automatically? Or should I make a manual copy-paste?

Okay I copied the file manually. We should look into automate this.

Yes, the automation is in the Makefile which is still under review: #2027 (make helm-doc-generate)

MonkeyCanCode · 2025-07-16T19:10:26Z

helm/polaris/tests/configmap_test.yaml

      - matchRegex: { path: 'data["application.properties"]', pattern: "polaris.tasks.max-concurrent-tasks=10" }
      - matchRegex: { path: 'data["application.properties"]', pattern: "polaris.tasks.max-queued-tasks=20" }
+
+  - it: should configure OIDC


Probably not in the scope of this PR. But we should add the doc for OIDC #1327 and have some docker compose with it (e.g. keycloak?)

We already have some docs for OIDC / external IDP:

https://github.com/apache/polaris/blob/45040bd5b80a6ab3dc5dc343182e8a6435d1e709/site/content/in-dev/unreleased/external-idp.md

* chore(deps): update dependency mypy to >=1.17, <=1.17.0 (apache#2114) * Spark 3.5.6 and Iceberg 1.9.1 (apache#1960) * Spark 3.5.6 and Iceberg 1.9.1 * Cleanup * Add `pathStyleAccess` to AwsStorageConfigInfo (apache#2012) * Add `pathStyleAccess` to AwsStorageConfigInfo This change allows configuring the "path-style" access mode in S3 clients (both in Polaris Servers and Iceberg REST Catalog API clients). This change is applicable both to AWS storage and to non-AWS S3-compatible storage (apache#1530). * Add TestFileIOFactory helper (apache#2105) * Add FileIOFactory.wrapExisting helper * fix(deps): update dependency gradle.plugin.org.jetbrains.gradle.plugin.idea-ext:gradle-idea-ext to v1.2 (apache#2125) * fix(deps): update dependency boto3 to v1.39.7 (apache#2124) * Abstract polaris-runtime-service tests for all persistence implementations (apache#2106) The NoSQL persistence implementation has to run the Iceberg table & view catalog plus the Polaris specific tests as well. Reusing existing tests is beneficial to avoid a lot of code duplcation. This change moves the actual tests to `Abstract*` classes and refactors the existing tests to extend those. The NoSQL persistence work extends the same `Abstract*` classes but runs with different Quarkus test profiles. * Add IMPLICIT authentication support to the CLI (apache#2121) PRs apache#1925 and apache#1912 were merged around the same time. This PR connects the two changes and enables the CLI to accept IMPLICIT authentication type. Since Hadoop federated catalogs rely purely on IMPLICIT authentication, the CLI parsing test has been updated to reflect the same. * feat(helm): Add support for external authentication (apache#2104) * fix(deps): update dependency org.apache.iceberg:iceberg-bom to v1.9.2 (apache#2126) * fix(deps): update quarkus platform and group to v3.24.4 (apache#2128) * fix(deps): update dependency boto3 to v1.39.8 (apache#2129) * fix(deps): update dependency io.smallrye.config:smallrye-config-core to v3.13.3 (apache#2130) * Add newIcebergCatalog helper (apache#2134) creation of `IcebergCatalog` instances was quite redundant as tests mostly use the same parameters most of the time. also remove an unused field in 2 other tests. * Add server and client support for the new generic table `baseLocation` field (apache#2122) * Use Makefile to simplify setup and commands (apache#2027) * Use Makefile to simplify setup and commands * Add targets for minikube state management * Add podman support and spark plugin build * Add version target * Update README.md for Makefile usage and relation to the project * Fix nit * Package polaris client as python package (apache#2049) * Package polaris client as python package * Package polaris client as python package * Change owner to spark when copying files from local into Dockerfile * CI: Address failure from accessing GH API (apache#2132) CI sometimes fails with this failure: ``` * What went wrong: Execution failed for task ':generatePomFileForMavenPublication'. > Unable to process url: https://api.github.com/repos/apache/polaris/contributors?per_page=1000 ``` The sometimes failing request fetches the list of contributors to be published in the "root" POM. Unauthorized GH API requests have an hourly(?) limit of 60 requests per source IP. Authorized requests have a much higher rate limit. We do have a GitHub token available in every CI run, which can be used in GH API requests. This change adds the `Authorization` header for the failing GH API request to leverage the higher rate limit and let CI not fail (that often). * fix(deps): update dependency com.nimbusds:nimbus-jose-jwt to v10.4 (apache#2139) * fix(deps): update dependency com.diffplug.spotless:spotless-plugin-gradle to v7.2.0 (apache#2142) * fix(deps): update dependency software.amazon.awssdk:bom to v2.32.4 (apache#2146) * fix(deps): update dependency org.xerial.snappy:snappy-java to v1.1.10.8 (apache#2138) * fix(deps): update dependency org.junit:junit-bom to v5.13.4 (apache#2147) * fix(deps): update dependency boto3 to v1.39.9 (apache#2137) * fix(deps): update dependency com.fasterxml.jackson:jackson-bom to v2.19.2 (apache#2136) * Python client: add support for endpoint, sts-endpoint, path-style-access (apache#2127) This change adds support for endpoint, sts-endpoint, path-style-access to the Polaris Python client. Amends apache#1913 and apache#2012 * Remove PolarisEntityManager.getCredentialCache (apache#2133) `PolarisEntityManager` itself is not using the `StorageCredentialCache` but just hands it out via `getCredentialCache`. the only caller of `getCredentialCache` is `FileIOUtil.refreshAccessConfig`, which in in turn is only called by `DefaultFileIOFactory` and `IcebergCatalog`. note that in a follow-up we will likely be able to remove `PolarisEntityManager` usage completely from `IcebergCatalog`. additional cleanups: - use `StorageCredentialCache` injection in tests (but we need to invalidate all entries on test start) - remove unused `UserSecretsManagerFactory` from `PolarisCallContextCatalogFactory` * chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.22-1.1752676419 (apache#2150) * fix(deps): update dependency com.diffplug.spotless:spotless-plugin-gradle to v7.2.1 (apache#2152) * fix(deps): update dependency boto3 to v1.39.10 (apache#2151) * chore: fix class reference in the javadoc of TableLikeEntity (apache#2157) * fix(deps): update dependency commons-codec:commons-codec to v1.19.0 (apache#2160) * fix(deps): update dependency boto3 to v1.39.11 (apache#2159) * Last merged commit 395459f --------- Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: Yong Zheng <yongzheng0809@gmail.com> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: Christopher Lambert <xn137@gmx.de> Co-authored-by: Pooja Nilangekar <poojan@umd.edu> Co-authored-by: Alexandre Dutra <adutra@apache.org> Co-authored-by: Yun Zou <yunzou.colostate@gmail.com>

github-project-automation bot added this to Basic Kanban Board Jul 15, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jul 15, 2025

adutra force-pushed the helm-auth branch 2 times, most recently from 6bbb635 to 062e54b Compare July 16, 2025 08:39

feat(helm): Add support for external authentication

7d018c2

adutra force-pushed the helm-auth branch from 062e54b to 7d018c2 Compare July 16, 2025 09:37

adutra commented Jul 16, 2025

View reviewed changes

adutra marked this pull request as ready for review July 16, 2025 12:00

adutra requested a review from MonkeyCanCode July 16, 2025 12:00

dimas-b reviewed Jul 16, 2025

View reviewed changes

review

8a1459e

dimas-b previously approved these changes Jul 16, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Jul 16, 2025

MonkeyCanCode reviewed Jul 16, 2025

View reviewed changes

MonkeyCanCode previously approved these changes Jul 16, 2025

View reviewed changes

MonkeyCanCode reviewed Jul 16, 2025

View reviewed changes

update helm.md

0683d92

adutra dismissed stale reviews from MonkeyCanCode and dimas-b via 0683d92 July 17, 2025 07:58

adutra requested review from MonkeyCanCode and dimas-b July 17, 2025 13:54

MonkeyCanCode approved these changes Jul 17, 2025

View reviewed changes

dimas-b approved these changes Jul 17, 2025

View reviewed changes

adutra merged commit 46a2a03 into apache:main Jul 17, 2025
12 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Jul 17, 2025

adutra deleted the helm-auth branch July 17, 2025 21:15

feat(helm): Add support for external authentication #2104

feat(helm): Add support for external authentication #2104

Uh oh!

Conversation

adutra commented Jul 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adutra Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MonkeyCanCode Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MonkeyCanCode Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adutra Jul 16, 2025 •

edited

Loading

MonkeyCanCode Jul 17, 2025 •

edited

Loading

MonkeyCanCode Jul 16, 2025 •

edited

Loading