From efc78a20d56ac846d6d9ac4e685b72c8d63b41d6 Mon Sep 17 00:00:00 2001
From: hui lai <laihui@selectdb.com>
Date: Fri, 29 Aug 2025 15:34:12 +0800
Subject: [PATCH] [fix](move-memtable) fix segment number mismatch for
 erroneously skipped segments (#55092)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

### What problem does this PR solve?

Fix segment number mismatch caused by erroneously skipped segments
during concurrent incremental open on auto-partitioned table：

#### Problem
During concurrent incremental open on an auto-partitioned table, one
sink may incorrectly assume that stream opened by another sink have
already been opened and begin writing data while those segments are
still being opened. This leads to some segments being silently skipped
and results in a segment number mismatch. For example(two instances, 4
BEs: a, b, c, d):
| Time | Event |
| ---- |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
| t0 | `sink1` and `sink2` start incremental open for BEs **a, b, c,
d**. |
| t1 | `sink1` adds **a, b, c** to `_load_stream_map` and initiates
open. |
| t2 | `sink2` adds **d** to `_load_stream_map` and initiates open. |
| t3 | `sink1` completes open for **a** and **b**; **c** is still in
progress. |
| t4 | `sink2` successfully opens **d**, assumes **a, b, c** are **all**
ready, and starts writing. Because **c** is not yet fully open, its
segments are skipped, causing the mismatch. |

#### Expected behavior
A sink must wait until all stream it depends on are fully opened before
starting any write.

#### Proposed fix
All sinks open the full set of streams (a, b, c, d) instead of a partial
subset. Lock on each stream guarantees that:
- Duplicate open attempts are prevented:only the first sink performs the
actual open; subsequent sinks wait until the open is complete.
- Expected behavior is preserved:every sink waits until all streams are
fully opened before starting any write, eliminating skipped segments and
the resulting segment-number mismatch.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---
 be/src/vec/sink/writer/vtablet_writer_v2.cpp | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/be/src/vec/sink/writer/vtablet_writer_v2.cpp b/be/src/vec/sink/writer/vtablet_writer_v2.cpp
index 08a791457c015c..2125ad8726cdb5 100644
--- a/be/src/vec/sink/writer/vtablet_writer_v2.cpp
+++ b/be/src/vec/sink/writer/vtablet_writer_v2.cpp
@@ -99,9 +99,7 @@ Status VTabletWriterV2::_incremental_open_streams(
                     tablet.set_partition_id(partition->id);
                     tablet.set_index_id(index.index_id);
                     tablet.set_tablet_id(tablet_id);
-                    if (!_load_stream_map->contains(node)) {
-                        new_backends.insert(node);
-                    }
+                    new_backends.insert(node);
                     _tablets_for_node[node].emplace(tablet_id, tablet);
                     if (known_indexes.contains(index.index_id)) [[likely]] {
                         continue;