[Kernel] [CatalogManaged] New Transaction and Committer etc. APIs #4814
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
scottsand-db
changed the title
Committer, CommitPayload, CommitResponse, CommitFailedException APIs
scottsand-db
requested review from
huan233usc,
sumeet-db,
dhruvarya-db,
nicklan,
raveeram,
zachschuermann,
raveeram-db,
vkorukanti,
scovich and
prakharjain09
and removed request for
raveeram
scottsand-db
commented
Jun 24, 2025
raveeram-db
reviewed
Jun 24, 2025
kernel/kernel-api/src/main/java/io/delta/kernel/commit/CommitFailedException.java
Outdated
Show resolved
Hide resolved
raveeram-db
reviewed
Jun 24, 2025
kernel/kernel-api/src/main/java/io/delta/kernel/commit/CommitPayload.java
Outdated
Show resolved
Hide resolved
vkorukanti
approved these changes
Jun 25, 2025
nicklan
reviewed
Jun 25, 2025
| * The {@link Protocol} that was read at the beginning of the commit. Empty if a new table is | ||
| * being created. | ||
| */ | ||
| public final Optional<Protocol> readProtocolOpt; |
There was a problem hiding this comment.
Can you remind me why we need the old/new p and ms?
There was a problem hiding this comment.
Good question! For example, to detect that the catalogManaged table feature has been enabled on the table.
There was a problem hiding this comment.
And, certainly, you want the new Protocol and Metadata so you can send that to the catalog, and then it always knows what's the latets P & M, and it can give that back on the next read request to efficiently construct a ResolvedTable without having to go to the file system to resolve the latest P & M
scottsand-db
force-pushed
the
stack/kernel_catalog_managed_writes_1
branch
from
b603c47 to
072ede5
Compare
raveeram-db
reviewed
Jun 26, 2025
kernel/kernel-api/src/main/java/io/delta/kernel/commit/CommitFailedException.java
Outdated
Show resolved
Hide resolved
scottsand-db
commented
Jun 27, 2025
kernel/kernel-api/src/main/java/io/delta/kernel/commit/CommitPayload.java
Outdated
Show resolved
Hide resolved
scottsand-db
commented
Jun 27, 2025
kernel/kernel-api/src/main/java/io/delta/kernel/commit/CommitPayload.java
Outdated
Show resolved
Hide resolved
scottsand-db
force-pushed
the
stack/kernel_catalog_managed_writes_1
branch
from
76c689f to
75bae78
Compare
scottsand-db
changed the title
Committer, CommitPayload, CommitResponse, CommitFailedException APIs
emkornfield
approved these changes
Jul 10, 2025
kernel/kernel-api/src/main/java/io/delta/kernel/commit/CommitFailedException.java
Show resolved
Hide resolved
kernel/kernel-api/src/main/java/io/delta/kernel/commit/CommitMetadata.java
Show resolved
Hide resolved
kernel/kernel-api/src/main/java/io/delta/kernel/commit/CommitMetadata.java
Show resolved
Hide resolved
kernel/kernel-api/src/main/java/io/delta/kernel/commit/CommitMetadata.java
Outdated
Show resolved
Hide resolved
kernel/kernel-api/src/main/java/io/delta/kernel/transaction/TransactionV2.java
Outdated
Show resolved
Hide resolved
Draft
5 tasks
emkornfield
approved these changes
Jul 10, 2025
huan233usc
reviewed
Jul 10, 2025
kernel/kernel-api/src/main/java/io/delta/kernel/commit/CommitMetadata.java
Show resolved
Hide resolved
huan233usc
approved these changes
Jul 10, 2025
scottsand-db
force-pushed
the
stack/kernel_catalog_managed_writes_1
branch
from
04565d9 to
230779b
Compare
5 tasks
raveeram-db
approved these changes
Jul 14, 2025
kernel/kernel-api/src/main/java/io/delta/kernel/transaction/TransactionV2.java
Outdated
Show resolved
Hide resolved
scottsand-db
added a commit
that referenced
this pull request
Jul 29, 2025
…DefaultCommitter skeleton (#4936) ## 🥞 Stacked PR Use this [link](https://github.com/delta-io/delta/pull/4936/files) to review incremental changes. - [**stack/kernel_catalog_managed_writes_get_committer**](#4936) [[Files changed](https://github.com/delta-io/delta/pull/4936/files)] --------- <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [X] Kernel - [ ] Other (fill in here) ## Description #4814 introduced new TransactionV2, Committer, CommitContext, etc APIs. This PR now adds the abilities for Engines to inject into Kernel the actual Committer. This PR also includes a no-op Default committer implementation that validates that it is not interacting with catalog-managed tables. We do this because: ResolvedTable does not return an Optional committer, but rather just a committer. So we need to give it _something_ in the case that no committer was provided to the table builder. ## How was this patch tested? New UTs. ## Does this PR introduce _any_ user-facing changes? New .withCommitter API
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🥞 Stacked PR
Use this link to review incremental changes.
Which Delta project/connector is this regarding?
Description
This PR adds new Transaction and Committer etc. APIs.
How was this patch tested?
Just new APIs.
Does this PR introduce any user-facing changes?
Yes.