Skip to content

Commit 793cf08

Browse files
authored
Feature: Rollback compaction on conflict (#1285)
Intention is make the catalog smarter, to revert the compaction commits in case of crunch to let the writers who are actually adding or removing the data to the table succeed. In a sense treating compaction as always a lower priority process. Presently the rest catalog client creates the snapshot and asks the Rest Server to apply the snapshot and gives this in a combination of requirement and update. Polaris could apply some basic inference and generate some updates to metadata given a property is enabled at a table level, by saying that It will revert back the commit which was created by compaction and let the write succeed. I had this PR in OSS, which was essentially doing this at the client end, but we think its best if we do this as server end. to support more such clients. How to use this Enable a catalog level configuration : polaris.config.rollback.compaction.on-conflicts.enabled when this is enabled polaris will apply the intelligence of rollbacking those REPLACE ops snapshot which have the property of polaris.internal.rollback.compaction.on-conflict in their snapshot summary to resolve conflicts at the server end ! a sample use case is there is a deployment of a Polaris where this config is enabled and there is auto compaction (maintenance job) which is updating the table state, it adds the snapshot summary that polaris.internal.rollback.compaction.on-conflict is true now when a backfill process running for 8 hours want to commit but can't because the compaction job committed before so in this case it will reach out to Polaris and Polaris will see if the snapshot of compation aka replace snapshot has this property if yes roll it back and let the writer succeed ! Devlist: https://lists.apache.org/thread/8k8t77dgk1vc124fnb61932bdp9kf1lc
1 parent 7bb0f93 commit 793cf08

File tree

3 files changed

+636
-23
lines changed

3 files changed

+636
-23
lines changed

polaris-core/src/main/java/org/apache/polaris/core/config/FeatureConfiguration.java

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -287,4 +287,15 @@ public static void enforceFeatureEnabledOrThrow(
287287
+ "This should only be set to 'true' for tests!")
288288
.defaultValue(false)
289289
.buildFeatureConfiguration();
290+
291+
public static final FeatureConfiguration<Boolean> ICEBERG_ROLLBACK_COMPACTION_ON_CONFLICTS =
292+
PolarisConfiguration.<Boolean>builder()
293+
.key("ICEBERG_ROLLBACK_COMPACTION_ON_CONFLICTS")
294+
.catalogConfig("polaris.config.rollback.compaction.on-conflicts.enabled")
295+
.description(
296+
"Rollback replace snapshots created by compaction which have "
297+
+ "polaris.internal.conflict-resolution.by-operation-type.replace property set to rollback "
298+
+ "in their snapshot summary")
299+
.defaultValue(false)
300+
.buildFeatureConfiguration();
290301
}

0 commit comments

Comments
 (0)