Add idempotency adapter and E2E coverage by huaxingao · Pull Request #14773 · apache/iceberg

huaxingao · 2025-12-05T23:45:31Z

This is the 4th PR for Idempotency Key. Introduce test-only idempotency in RESTCatalogAdapter:

in-memory store keyed by Idempotency-Key,
cached-success replay
TTL expiry
optional “503 after success” simulation.

Add E2E tests:

testIdempotentDuplicateCreateReturnsCached
testIdempotencyKeyLifetimeExpiredTreatsAsNew
testIdempotentCreateReplayAfterSimulated503

First PR: #14649
SecondPR: #14700
Third PR: #14740

core/src/test/java/org/apache/iceberg/rest/RESTCatalogAdapter.java

nastra · 2025-12-10T14:01:05Z

core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java

+      long now = System.currentTimeMillis();
+      boolean expired =
+          existing.status == IdempotencyEntry.Status.FINALIZED
+              && (now - existing.firstSeenMillis) > idempotencyLifetimeMillis;


can we use Instant.now().minusMillis(existing.firstSeenMillis) here as that is more readable when adding isBefore / isAfter

you could also make this a function on the entry itself, so that you'd only call if(!entry.expired()) here

Fixed. Thanks!

nastra · 2025-12-10T14:06:17Z

core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java

+  }
+
+  // Test hooks/configuration for idempotency behavior
+  public static void simulate503OnFirstSuccessForKey(String key) {


I'm not really a fan of such a way to configure this. We should explore doing this similarly how remote scan planning sets the planning behavior for particular tests

Thanks for the suggestion. I removed the static hook from CatalogHandlers and mirrored the remote planning pattern:

Added a per-adapter IdempotencyBehavior in RESTCatalogAdapter and apply it post‑success in execute(...).

Tests configure behavior via the adapter (e.g., adapterForRESTServer.simulate503OnFirstSuccessForKey(key)), while routes still call CatalogHandlers.withIdempotency(...).

I don't think we should add this functionality at all to the adapter. Instead, what about just having a custom adapter that simulates this behavior for that particular test case. We already use custom adapters with custom behavior in a lot of other test methods.
My main concern is that we're trying to force "test conditions" into code that shouldn't have such test conditions, because RESTCatalogAdapter could be used in real usage scenarios and it shouldn't ideally contain test code. Additionally, this opens up a precedent for future code modifications where it might seem ok to add more and more testing conditions to the adapter or to CatalogHandlers

I guess one alternative would be to have a custom adapter in the testFixtures, similar to

iceberg/open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTServerCatalogAdapter.java

Line 30 in 9ea3b13

class RESTServerCatalogAdapter extends RESTCatalogAdapter {

where we have a custom adapter that performs credential vending

Thanks — agreed. I removed the test hook from CatalogHandlers and kept RESTCatalogAdapter free of test knobs.
The transient-503 simulation is now implemented only in a test-only adapter used by TestRESTCatalog (HeaderValidatingAdapter), so production-ish code paths don’t carry test conditions.

nastra · 2025-12-10T14:10:58Z

core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java

+    AuthSession httpSession = am.initSession(httpBase, conf);
+    RESTClient http = httpBase.withAuthSession(httpSession);
+
+    CreateTableRequest req =


seems a bit weird to create the request here. Can we not mock stuff as we do in a bunch of other tests and verify that the correct requests were sent?

I’ve added a reusable verify helper and now assert the request shape (method/path/headers, including Idempotency‑Key) after the POSTs. I’m keeping a real CreateTableRequest in the E2E tests because it’s needed to exercise serialization and replay.

core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java

huaxingao · 2025-12-15T18:36:37Z

@nastra @amogh-jahagirdar @singhpk234 Could you please take a look when you have a moment? Thanks!

singhpk234 · 2025-12-20T00:12:46Z

core/src/test/java/org/apache/iceberg/rest/RESTCatalogAdapter.java

+    simulate503OnFirstSuccessKeys.add(key);
+  }
+
  private static OAuthTokenResponse handleOAuthRequest(Object body) {


can we wrap this to plan-api endpoints too ?

Good point. Plan endpoints aren’t wrapped with withIdempotency yet. I’ll follow up with a separate PR to add Idempotency-Key support for the plan APIs (wrap handlers + add tests) and we can extend the simulation coverage there.

singhpk234 · 2025-12-20T00:14:55Z

core/src/test/java/org/apache/iceberg/rest/RESTCatalogAdapter.java

  }

+  /** Test helper to simulate a transient 503 after the first successful mutation for a key. */
+  public void simulate503OnFirstSuccessForKey(String key) {


nit : should we make it generic like simulate status code or something ?

Good suggestion. I kept it 503-specific since the test hook is only used to exercise retry behavior for transient 503/commit-state-unknown; happy to generalize later if we add tests needing other codes.

It would be great to not add test-specific code to the adapter/catalog handler. See my other comment on this: #14773 (comment)

core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java

core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java

singhpk234

Mostly LGTM, thanks @huaxingao left minor comments

core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java

core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java

core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java

core/src/test/java/org/apache/iceberg/rest/RESTCatalogAdapter.java

singhpk234

LGTM, thanks @huaxingao !

Lets give some time to @nastra @amogh-jahagirdar incase they have further feedbacks !

core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java

huaxingao · 2026-01-09T02:38:54Z

@nastra I removed the test hook from CatalogHandlers and kept RESTCatalogAdapter free of test knobs.
The transient-503 simulation is now implemented only in a test-only adapter used by TestRESTCatalog (HeaderValidatingAdapter). Could you please check if this looks OK to you? Thanks!

amogh-jahagirdar

sorry for the late review, overall I think it looks reasonable, just minor comments, will wait in case @nastra wanted to do another pass. Thanks @huaxingao

amogh-jahagirdar · 2026-01-09T13:11:44Z

core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java

+    private final Set<String> simulate503OnFirstSuccessKeys =
+        org.apache.iceberg.relocated.com.google.common.collect.Sets.newConcurrentHashSet();


Minor: Maybe this should be a mapping from the key to a generic throwable that's thrown, then we could add other failure cases and make sure idempotency returns or throws as expected.

Good idea. I generalized the test-only hook to a key -> RuntimeException map so tests can simulate different post-success transient failures while still keeping simulate503OnFirstSuccessForKey as a convenience wrapper for the current case.

amogh-jahagirdar · 2026-01-09T13:20:39Z

core/src/test/java/org/apache/iceberg/rest/RESTCatalogAdapter.java

        {
          if (PropertyUtil.propertyAsBoolean(vars, "purgeRequested", false)) {
-            CatalogHandlers.purgeTable(catalog, tableIdentFromPathVars(vars));
+            CatalogHandlers.withIdempotency(


Minor: I wonder if withIdempotency even needs to be exposed? Feels like it could just be invoked internally in each of the existing CatalogHandlers methods?

Good point. The idempotency behavior is driven by the HTTP request headers, so I’m keeping the wrapping at the routing/adapter layer rather than pushing HTTPRequest through every CatalogHandlers.* method.
That said, I agree it doesn’t need to be exposed publicly — I made withIdempotency(...) package-private and it’s only used internally by the REST adapter/routes.

amogh-jahagirdar · 2026-01-09T13:30:49Z

core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java

+    // replaces) the IN_PROGRESS entry for this Idempotency-Key. Only the leader executes the
+    // action and finalizes the entry; concurrent requests for the same key ("followers") wait on
+    // the latch and then replay the finalized result/error.
+    AtomicBoolean isLeader = new AtomicBoolean(false);


Minor: I'd probably just call this isFirst. Leader/follower makes it seem more than it really is.

Changed to isFirst. Thanks

amogh-jahagirdar · 2026-01-09T13:33:53Z

core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java

+    TableIdentifier ident = env.first();
+    Pair<RESTClient, Map<String, String>> httpAndHeaders = env.second();
+    RESTClient http = httpAndHeaders.first();
+    Map<String, String> headers = httpAndHeaders.second();
+    CreateTableRequest req = createReq(ident);


Nit: I think this bit is repeated across tests, any way to abstract some of this behind a helper?

I factored the repeated prepareIdempotentEnv unpacking into a small IdempotentEnv helper

huaxingao · 2026-01-13T19:21:10Z

Thanks @amogh-jahagirdar @geruh @nastra @singhpk234 for your review! Are there any more comments? If not, I plan to merge this PR in two days.

huaxingao · 2026-01-16T17:53:02Z

I merged this PR. If there are any more comments, I will address in follow-ups. Thank you all for reviewing this PR!

github-actions bot added the core label Dec 5, 2025

huaxingao requested review from nastra and singhpk234 December 6, 2025 01:28

nastra reviewed Dec 8, 2025

View reviewed changes

core/src/test/java/org/apache/iceberg/rest/RESTCatalogAdapter.java Outdated Show resolved Hide resolved

nastra reviewed Dec 10, 2025

View reviewed changes

core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java Outdated Show resolved Hide resolved

nastra requested a review from amogh-jahagirdar December 10, 2025 14:12

huaxingao force-pushed the idempotency_4 branch 2 times, most recently from 47aae84 to a8a0a16 Compare December 15, 2025 18:34

singhpk234 reviewed Dec 20, 2025

View reviewed changes

geruh reviewed Dec 23, 2025

View reviewed changes

core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java Outdated Show resolved Hide resolved

huaxingao force-pushed the idempotency_4 branch from a8a0a16 to 4a81530 Compare January 2, 2026 22:34

singhpk234 reviewed Jan 6, 2026

View reviewed changes

singhpk234 approved these changes Jan 7, 2026

View reviewed changes

core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java Outdated Show resolved Hide resolved

nastra reviewed Jan 7, 2026

View reviewed changes

core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java Outdated Show resolved Hide resolved

huaxingao force-pushed the idempotency_4 branch from c3e5460 to 44b467c Compare January 8, 2026 22:42

huaxingao added 5 commits January 8, 2026 17:18

rebase

9ee688c

rebase to upstream

d25e5c8

address comments

04586d8

address comments

4a9ec58

add back idempotency tests

76f6da0

huaxingao force-pushed the idempotency_4 branch from 76027b6 to 76f6da0 Compare January 9, 2026 01:18

amogh-jahagirdar approved these changes Jan 9, 2026

View reviewed changes

address comments

e401335

huaxingao merged commit 4f57687 into apache:main Jan 16, 2026
46 checks passed

		private final Set<String> simulate503OnFirstSuccessKeys =
		org.apache.iceberg.relocated.com.google.common.collect.Sets.newConcurrentHashSet();

Comments

Conversation

huaxingao commented Dec 5, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huaxingao Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

huaxingao commented Dec 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

singhpk234 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

singhpk234 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

huaxingao commented Jan 9, 2026

Uh oh!

amogh-jahagirdar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huaxingao commented Jan 13, 2026

Uh oh!

Uh oh!

huaxingao commented Jan 16, 2026

huaxingao Jan 8, 2026 •

edited

Loading

singhpk234 left a comment •

edited

Loading