feat: use independent region manifest for MemWAL #5689

touch-of-grey · 2026-01-12T07:59:05Z

Based on discussion with @jackye1995 past few days, update the current MemWAL structure to introduce region specific manifest for independent update so that it can more efficiently run updates in streaming engines like Flink. This mostly comes from draft from @jackye1995, but separate out the MemWAL index part first with some my custom edits.

jackye1995 · 2026-01-12T08:00:36Z

Thanks for pushing progress on this, I will take a look tomorrow morning!

docs/src/format/table/mem_wal.md

- Mark as experimental - Restructure sections (architecture first, then components) - Use bloom filter instead of btree for PK existence - Fix vector index: use same IVF type/centroids/codebook as base - Add region pruning, sync/async ops summary - Update query plans for multi-region (no cross-region dedup) - Rename _memwal to _mem_wal - Remove redundant sections Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

codecov · 2026-01-13T08:53:56Z

Codecov Report

❌ Patch coverage is 68.08511% with 165 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/mem_wal.rs	24.68%	115 Missing and 4 partials ⚠️
rust/lance/src/io/commit/conflict_resolver.rs	89.49%	21 Missing and 12 partials ⚠️
rust/lance/src/dataset/transaction.rs	67.74%	9 Missing and 1 partial ⚠️
rust/lance/src/dataset/write/merge_insert.rs	50.00%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

- Replace RegionId struct with Uuid type alias for consistency - Rename snapshot_timestamp to snapshot_ts_millis per spec - Add conflict resolution between UpdateMemWalState and CreateIndex of MemWalIndex - Implement generation comparison: committed >= to_commit is non-retryable, committed < to_commit is retryable - Add rebase logic to merge merged_generations when CreateIndex rebases against UpdateMemWalState - Add comprehensive unit and integration tests for conflict scenarios Co-Authored-By: Jack Ye <yezhaoqin@gmail.com> Co-Authored-By: Claude <noreply@anthropic.com>

- Propagate load_mem_wal_index_details errors instead of silently returning Ok - Consume conflicting_mem_wal_merged_gens directly to avoid clone Co-Authored-By: Jack Ye <yezhaoqin@gmail.com> Co-Authored-By: Claude <noreply@anthropic.com>

jackye1995

thanks for working with me for all the comments! Looks good to me now, pending CI

Update Operation::Update field from mem_wal_to_merge to merged_generations. Co-Authored-By: Jack Ye <yezhaoqin@gmail.com> Co-Authored-By: Claude <noreply@anthropic.com>

- Add IndexCatchupProgress to track which merged generation each base table index has caught up to - Update MemWalIndexDetails proto and Rust implementation - Remove merged_generation from RegionManifest (single source of truth) - Add LSM Tree Merging section and Appendix 4 to spec - Improve Indexed Read documentation with better explanation Co-Authored-By: Jack Ye <yezhaoqin@gmail.com> Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

jackye1995 · 2026-01-13T20:12:44Z

I updated the spec to also talk about indexed read and index builder, let me know if it looks good to you!

jackye1995 · 2026-01-13T21:05:15Z

Thanks for going through my tedious comments! Let me know when you get the next PRs up I will review asap!

@jackye1995

Based on discussion with @jackye1995 past few days, update the current MemWAL structure to introduce region specific manifest for independent update so that it can more efficiently run updates in streaming engines like Flink. This mostly comes from draft from @jackye1995, but separate out the MemWAL index part first with some my custom edits. --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Jack Ye <yezhaoqin@gmail.com>

github-actions bot added the enhancement New feature or request label Jan 12, 2026

jackye1995 self-requested a review January 12, 2026 08:00