Create a wrapper script to generate python client; regenerate the python client #1347

eric-maynard · 2025-04-09T20:51:38Z

As noted in #755 and elsewhere, the generated types in client/python are currently out of date. This introduces a script to regenerate them and a gradle task to run that script.

I've also run the script, which necessitated several things to get tests passing:

There were small nonfunctional spec changes needed in order to keep the Python client working
The CLI and its tests required a few fixes to work with the updated Python client
Many of the regtests required fixes to work with the updated Python client

eric-maynard · 2025-04-09T23:38:57Z

@snazy can you take a look at this draft? My impression is that I should be using the headers from codeconfig but I'd like to get your perspective on the best approach here. Additionally, this way of manually modifying files looks a little suspect to me but I could not find a way to easily augment the generator to do what we want.

eric-maynard · 2025-05-08T07:03:00Z

spec/iceberg-rest-catalog-open-api.yaml

@@ -260,6 +260,7 @@ paths:
      parameters:
        - $ref: '#/components/parameters/page-token'
        - $ref: '#/components/parameters/page-size'
+        - $ref: '#/components/parameters/prefix'


This change is needed due to an unfortunate bug in the python generator wherein parameters are overriden, not extended. Without this, the method list_namespaces doesn't have a prefix argument.

Shall we make a note somewhere (e.g. spec/README.md) that these changes are needed? Since iceberg-rest-catalog-open-api.yaml should be a copy-paste of upstream one, we may easily lose track of these additional changes the next time we copy-paste from the upstream.

Ideally, the CI will fail when a bad copy/paste happens. We need to wire up the CI to the new task but I would like to do that in a followup. For now, let me add this in the README!

Nit: it'd be nice to add the comment directly here like Non functional change required by Python generator. I believe this is the place where people may get confused every time pulling iceberg spec upstream.

HonahX

This looks great! Just to confirm, currently we expect to manually regenerate the client after a spec change so the CI should not run the regeneration yet.

HonahX · 2025-05-23T17:07:57Z

client/templates/regenerate.sh

+  "./polaris/management/__pycache__/"
+  "./polaris/management/models/__pycache__/"
+  "./polaris/management/api/__pycache__/"
+)


[non-blocker] Shall we add ./.DS_STORE to exclude_paths for mac users? This makes running regenerate.sh fails in my mac locally with

Re-applying license headers No header compatible with file ./.DS_Store

Suggested change

)

"./.DS_Store"

"./poetry.lock"

)

Also "poetry.lock" should be excluded

Synced with Eric offline, I believe we can solve this in a follow-up which automate the regeneration during test

HonahX

LGTM!

sfc-gh-ygu

LGTM

I ran these commands from main: ``` redocly bundle spec/polaris-catalog-service.yaml -o spec/generated/bundled-polaris-catalog-service.yaml ./gradlew regeneratePythonClient ``` I didn't realize before that some Python types are generated form the bundled spec, so some of the fixes from #1347 didn't get properly applied before.

It looks like this method lost the `prefix` and `namespace` parameters in #1347. They're re-introduced to the spec here, and then I've run: ``` redocly bundle spec/polaris-catalog-service.yaml -o spec/generated/bundled-polaris-catalog-service.yaml ./gradlew regeneratePythonClient ``` Then, some manual reverts: ``` alias gitrevert='git checkout upstream/main --' gitrevert client/python/.github/workflows/python.yml gitrevert client/python/.gitlab-ci.yml gitrevert client/python/pyproject.toml ``` I still hope to automate this process as part of CI soon; see #1675

* fix(nightly-CI): Do not publish snapshots from forks (apache#1635) Adopt the `Nightly Build` workflow to not (try to) publish every night from forks. * main: Update dependency io.smallrye.config:smallrye-config-core to v3.13.0 (apache#1637) * Use echo to print script errors (apache#1648) * [HOTFIX] QUICKSTART (apache#1646) The change adds the following to fix Quick start experience : [1] ENV variables required by common assets after apache#1522 [2] New configs required to enable FILE based sources apache#1649 Co-authored-by: singhpk234 <singhpk234@users.noreply.github.com> Co-authored-by: pjanuario <pjanuario@users.noreply.github.com> * main: Update dependency gradle to v8.14.1 (apache#1652) * main: Update dependency gradle to v8.14.1 * Re-adopt PR to the project's needs --------- Co-authored-by: Robert Stupp <snazy@snazy.de> * [Policy Store] Add policyTypeCode to Slice/Index for Future Filtering Support and Update Policy Persistence Method (apache#1628) This PR adds policyTypeCode to the in-memory tree map store's slice and the SQL index on policy_mapping_records (already done in JDBC in apache#1468). This prepares for future features that need to filter efficiently by policy type, like listing all entities with a data compaction policy. It also updates the loadAllTargetsOnPolicy method to accept policyTypeCode, enabling it to use the new index for better performance. * fix(test): Do not let some more tests spam `/tmp` (apache#1651) * fix(test): Do not let some more tests not spam `/tmp` * `PolarisRestCatalogViewFileIntegrationTest` * `FileIOExceptionsTest` * `PolarisRestCatalogViewFileIntegrationTest` Changes the tests to leverage JUnit's `@TempDir`. Simplifies `PolarisEclipseLinkMetaStoreManagerTest` * review: rename the (now) abstract class * fix(testing): Do not let PolarisOverlappingTableTest spam `/tmp` (apache#1641) Changes the test to leverage JUnit's `@TempDir`. * Add CATALOG_MANAGE_METADATA to super privilege set of policy attachment privileges (apache#1643) * Fix quickstart doc with docker compose (apache#1610) * main: Update dependency boto3 to v1.38.22 (apache#1657) * Refactor IcebergCatalog to isolate internal state (apache#1659) Following up on apache#1694 * Restore `private` scope on internal fields in `IcebergCatalog` * Use a test-only setter instead of sub-classing to manage injecting test FileIO implementations * Refactor: Use per-request STS credentials (apache#1629) * Refactor: Use per-request STS credentials No functional changes. This is mostly to allow more storage integration flexibility in downstream build. This might also be useful for non-AWS storage. * fix and enforce more errorprone checks (apache#1663) enforces the following checks: https://errorprone.info/bugpattern/ObjectsHashCodePrimitive https://errorprone.info/bugpattern/OptionalMapToOptional https://errorprone.info/bugpattern/StringCharset https://errorprone.info/bugpattern/VariableNameSameAsType * Create a wrapper script to generate python client; regenerate the python client (apache#1347) As noted in apache#755 and elsewhere, the generated types in client/python are currently out of date. This introduces a script to regenerate them and a gradle task to run that script. I've also run the script, which necessitated several things to get tests passing: 1. There were small nonfunctional spec changes needed in order to keep the Python client working 2. The CLI and its tests required a few fixes to work with the updated Python client 3. Many of the regtests required fixes to work with the updated Python client * [Python Client] CI for Python client (Continue PR#1096) (apache#1639) Adds CI for python client. It does not include caching poetry step for now since we do not have poetry.lock (it is in .gitignore), see relevant discussion in: apache#1102 (comment), apache#1096 (comment), we can add that later * main: Update actions/setup-python action to v5 (apache#1671) * main: Update actions/checkout action to v4 (apache#1670) * main: Update python Docker tag to v3.13 (apache#1669) * main: Update dependency pytest to ~=7.4.4 (apache#1668) * main: Update dependency software.amazon.awssdk:bom to v2.31.50 (apache#1677) * main: Update dependency boto3 to v1.38.23 (apache#1667) * feat(build): make archive builds reproducible (apache#1664) See https://docs.gradle.org/current/userguide/working_with_files.html#sec:reproducible_archives * main: Update dependency io.prometheus:prometheus-metrics-exporter-servlet-jakarta to v1.3.8 (apache#1679) * NoSQL: adapt to change on oss/main * INFO: Last merged commit: 6ef8b3e --------- Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: ModEtchFill <50123102+ModEtchFill@users.noreply.github.com> Co-authored-by: Prashant Singh <35593236+singhpk234@users.noreply.github.com> Co-authored-by: singhpk234 <singhpk234@users.noreply.github.com> Co-authored-by: pjanuario <pjanuario@users.noreply.github.com> Co-authored-by: Honah (Jonas) J. <honahx@apache.org> Co-authored-by: MonkeyCanCode <yongzheng0809@gmail.com> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@dremio.com> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: Christopher Lambert <xn137@gmx.de> Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com>

* main: Update dependency org.postgresql:postgresql to v42.7.6 (apache#1697) * main: Update helm/chart-testing-action action to v2.7.0 (apache#1700) * main: Update gradle/actions digest to 8379f6a (apache#1696) * main: Update medyagh/setup-minikube action to v0.0.19 (apache#1698) * feat(metrics): Mitigate potential performance issues with realm_id tag (apache#1662) As discussed in the ML, this PR introduces two flags to enable the inclusion of realm ID tags in API and HTTP metrics. They are both disabled by default. There is also a new safeguard: if the cardinality of realm IDs in HTTP metrics goes above a configurable threshold (100 by default), a warning is printed and no more HTTP metrics will be recorded. (Quarkus has a similar safeguard for URI tags in HTTP metrics.) * Site/contributing: add recommendations for working with PRs (apache#1625) This change updates the PR guidelines on the "Contributing" web page after [this discussion](https://lists.apache.org/thread/kfxo3cqmw3pgrpgtgqvqpwvn46yw8q7h). Also adopt `gradlew test` to `gradlew check` in README, following the intent (all tests, incl ITs) * main: Update dependency com.azure:azure-sdk-bom to v1.2.35 (apache#1703) * main: Update dependency com.adobe.testing:s3mock-testcontainers to v4.4.0 (apache#1705) * main: Update dependency boto3 to v1.38.24 (apache#1702) * Keep generated RSA-key-pair for JWT token broker on heap (apache#1661) Polaris allows using RSA key-paris for the JWT token broker. The recommended way is to [generate the RSA key pair](https://github.com/apache/polaris/blob/d8b862b13914d526ee147dc0e359bfc9c1e319ad/site/content/in-dev/unreleased/configuring-polaris-for-production.md?plain=1#L61-L66) and configure the location of the key files. However, if only `polaris.authentication.token-broker.type=rsa-key-pair` but not the `public/private-key-pair` options are configured, Polaris generates those and stores them in `/tmp` using random file names (using `Files.createTempFile()`) - this happens for each (matching) realm. Each Polaris startup generates new key-pairs for each of those realms. It's practically not possible to associate the files to a realm. There is already a [production readiness check](https://github.com/apache/polaris/blob/d8b862b13914d526ee147dc0e359bfc9c1e319ad/quarkus/service/src/main/java/org/apache/polaris/service/quarkus/config/ProductionReadinessChecks.java#L118-L166) to warn users about this behavior. Due to the issue that the files cannot be associated, those seem to be somewhat useless and bring no advantage over keeping these "ephemeral RSA key pairs" on heap. This PR changes the code to not write the key-pair to the file system and keeps these "ephemeral key pairs" on heap. Since the same code path is used for key-paris _provided_ by the user (via the `public/private-key-pair` config options), that code path now only reads those files once and not every time the private/public key is needed. * Merge JPA module with EclipseLink Module (apache#1718) * Create LICENSE and NOTICE for "single" distribution (apache#1694) * Fix a failing task with the release profile (apache#1693) * Remove unused adminDocs artifact (apache#1749) * Production readiness for Persistence (apache#1707) Production readiness for Persistence (apache#1707) * Fixes for direct usage of client_secret apache#1756 When the spec was upgraded and the python client regenerated from it, clientSecret was made a password, which means calling str on it directly yields a redacted string like ******. In the initial PR to change the python client and fix regtests, some existing usage of client_secret was not changed. * main: Update dependency org.junit:junit-bom to v5.13.0 (apache#1760) * main: Update dependency org.testcontainers:testcontainers-bom to v1.21.1 (apache#1748) * fix: Improve reliability of metrics tests (apache#1763) CI sometimes fails with errors like "http_server_requests_seconds not found" in the reported metrics. This looks like a race between the Quarkus metrics producer and the tests asking for these metrics. This change adds a time-limited retry loop until the expected metrics are available, before proceeding with other assertions. Note: in normal cases the loop finishes fast because the metrics are available. The two-minute timeout would apply only when the expected metrics fail to be produced at all. * Fix test_spark_credentials_s3_exception_on_metadata_file_deletion (apache#1759) * Regenerate bundled spec & Regenerate Python client (apache#1751) I ran these commands from main: ``` redocly bundle spec/polaris-catalog-service.yaml -o spec/generated/bundled-polaris-catalog-service.yaml ./gradlew regeneratePythonClient ``` I didn't realize before that some Python types are generated form the bundled spec, so some of the fixes from apache#1347 didn't get properly applied before. * main: Update dependency boto3 to v1.38.27 (apache#1714) * NoSQL: bump Weld/Junit5 (fixes a bug that surfaces w/ JUnit 5.13) * NoSQL: Let some more tests leverage Jandex * Info: Last merged commit b7aac72 --------- Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: Alexandre Dutra <adutra@users.noreply.github.com> Co-authored-by: Yufei Gu <yufei@apache.org> Co-authored-by: JB Onofré <jbonofre@apache.org> Co-authored-by: Prashant Singh <35593236+singhpk234@users.noreply.github.com> Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: gh-yzou <167037035+gh-yzou@users.noreply.github.com>

* Fix regression test docker setup for purge (apache#1768) * Use canonical catalog property names in tests (apache#1766) * In `PolarisPolicyServiceIntegrationTest` * In `PolarisRestCatalogIntegrationTest` Following up to apache#1557 * Unblock test `createViewWithCustomMetadataLocation` (apache#1320) * Add test for invalid custom metadata location * Add missing properties during table/view creation * main: Update docker.io/prom/prometheus Docker tag to v3.4.1 (apache#1767) * Testing: silence a bunch of harmless test warnings (apache#1773) * `OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended` * Hibernate Validator cannot instrument static methods (`Hibernate Validator does not support constraints on static methods yet. ...`) * ForkJoinPool test lifecycle warning * Couple of split-package warnings * Add unit test for legacy config lookup (apache#1774) Following up on apache#1766 * Handle RequestScoped instance injection gracefully for DefaultConfigurationStore (apache#1758) Although we do a check of !realmContextInstance.isUnsatisfied() it only checks if there is matching bean, however, it doesn't check if the context is active or not. Similarly isResolvable also don't check if the scope is active. Therefore if getConfiguration is called when there is no active scope, it will get Exception like ContextNotActiveException. This actually fails the TaskExecutor because it runs in a separate thread. In order to fix the problem, we introduces a new getConfiguration function to handle the background tasks, and also use isResolvable instead of isUnsatisfied to handle ambiguities. * Restructure the directory and package name for persistence modules (apache#1724) * Re-add missing parameters to create_table python API (apache#1778) It looks like this method lost the `prefix` and `namespace` parameters in apache#1347. They're re-introduced to the spec here, and then I've run: ``` redocly bundle spec/polaris-catalog-service.yaml -o spec/generated/bundled-polaris-catalog-service.yaml ./gradlew regeneratePythonClient ``` Then, some manual reverts: ``` alias gitrevert='git checkout upstream/main --' gitrevert client/python/.github/workflows/python.yml gitrevert client/python/.gitlab-ci.yml gitrevert client/python/pyproject.toml ``` I still hope to automate this process as part of CI soon; see apache#1675 * Replace getConfiguration usage with PolarisCallContext to use RealmContext (PART 1) (apache#1780) * JDBC: Fix getting started config (apache#1781) Fix typo in the JDBC config for getting started, the config should be max_duration_in_ms instead of max_delay_in_ms * main: Pin dependencies (apache#1701) * main: Update dependency pytest to v8 (apache#1710) * main: Update dependency boto3 to v1.38.28 (apache#1777) * Run renovatebot only on the main branch (apache#1786) * INFO: last merged commit a827d26 --------- Co-authored-by: gh-yzou <167037035+gh-yzou@users.noreply.github.com> Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: Liam Bao <liam.zw.bao@gmail.com> Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: Yufei Gu <yufei@apache.org> Co-authored-by: Eric Maynard <eric.maynard+oss@snowflake.com> Co-authored-by: Prashant Singh <35593236+singhpk234@users.noreply.github.com> Co-authored-by: JB Onofré <jbonofre@apache.org>

rebase

56477d4

github-project-automation bot added this to Basic Kanban Board Apr 9, 2025

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Apr 9, 2025

eric-maynard force-pushed the 2025-04-09-python branch from 592d2af to 56477d4 Compare April 9, 2025 23:40

eric-maynard added 2 commits April 9, 2025 23:56

add formatting

901cdd1

typofix

68ac664

eric-maynard force-pushed the 2025-04-09-python branch from 87accb7 to 68ac664 Compare April 10, 2025 06:56

eric-maynard added 2 commits April 9, 2025 23:58

no formatting

bb8ac06

attempt to update spec

7cca817

eric-maynard mentioned this pull request Apr 10, 2025

Generated files in regtests/client/python/polaris #755

Closed

fixes

53e8503

eric-maynard force-pushed the 2025-04-09-python branch from 592e296 to 53e8503 Compare April 10, 2025 21:10

eric-maynard marked this pull request as ready for review April 15, 2025 17:18

eric-maynard requested review from MonkeyCanCode, RussellSpitzer, adutra, ashvina, collado-mike, dennishuo, dimas-b, ebyhr, flyrain, jackye1995, jbonofre, snazy, takidau and vvcephei as code owners April 15, 2025 17:18

eric-maynard added 2 commits April 15, 2025 22:13

script fix

e652a10

dirname fix

021f771

eric-maynard added 2 commits May 8, 2025 00:00

small spec change, regen

20d9b05

regtests locally pass

4018286

eric-maynard commented May 8, 2025

View reviewed changes

eric-maynard added 8 commits May 8, 2025 00:36

fix listtables as well

e35ac90

fixes, regen again

628b12b

fixes, regen again

0db225f

pythonpath

898bd6f

relativize

8cd6575

revert client/python/pyproject.toml

7ddde9e

copy

af5d390

Merge branch 'main' into 2025-04-09-python

352a6ca

eric-maynard requested a review from singhpk234 as a code owner May 12, 2025 17:08

eric-maynard added the 1.0-blocker label May 20, 2025

readme line

bac1613

eric-maynard requested a review from ajantha-bhat as a code owner May 21, 2025 16:38

retest

249e9a6

eric-maynard mentioned this pull request May 23, 2025

[Python Client] CI for Python client (Continue PR#1096) #1639

Merged

HonahX reviewed May 23, 2025

View reviewed changes

HonahX approved these changes May 23, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board May 23, 2025

sfc-gh-ygu approved these changes May 23, 2025

View reviewed changes

eric-maynard merged commit af26732 into apache:main May 23, 2025
6 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board May 23, 2025

eric-maynard mentioned this pull request May 29, 2025

Regenerate bundled spec & Regenerate Python client #1751

Merged

eric-maynard mentioned this pull request Jun 2, 2025

Re-add missing parameters to create_table python API #1778

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Create a wrapper script to generate python client; regenerate the python client #1347

Create a wrapper script to generate python client; regenerate the python client #1347

Uh oh!

eric-maynard commented Apr 9, 2025 •

edited

Loading

Uh oh!

eric-maynard commented Apr 9, 2025

Uh oh!

eric-maynard May 8, 2025

Uh oh!

HonahX May 21, 2025

Uh oh!

eric-maynard May 21, 2025

Uh oh!

sfc-gh-ygu May 23, 2025

Uh oh!

HonahX left a comment

Uh oh!

HonahX May 23, 2025 •

edited

Loading

Uh oh!

HonahX May 23, 2025

Uh oh!

HonahX May 23, 2025

Uh oh!

HonahX left a comment

Uh oh!

sfc-gh-ygu left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

-)
+   "./.DS_Store"
+  "./poetry.lock"
+)

Create a wrapper script to generate python client; regenerate the python client #1347

Create a wrapper script to generate python client; regenerate the python client #1347

Uh oh!

Conversation

eric-maynard commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eric-maynard commented Apr 9, 2025

Uh oh!

eric-maynard May 8, 2025

Choose a reason for hiding this comment

Uh oh!

HonahX May 21, 2025

Choose a reason for hiding this comment

Uh oh!

eric-maynard May 21, 2025

Choose a reason for hiding this comment

Uh oh!

sfc-gh-ygu May 23, 2025

Choose a reason for hiding this comment

Uh oh!

HonahX left a comment

Choose a reason for hiding this comment

Uh oh!

HonahX May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HonahX May 23, 2025

Choose a reason for hiding this comment

Uh oh!

HonahX May 23, 2025

Choose a reason for hiding this comment

Uh oh!

HonahX left a comment

Choose a reason for hiding this comment

Uh oh!

sfc-gh-ygu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

eric-maynard commented Apr 9, 2025 •

edited

Loading

HonahX May 23, 2025 •

edited

Loading