Introducing MergedSegmentWarmerFactory to support the extension of IndexWriter.IndexReaderWarmer #17881

guojialiang92 · 2025-04-10T16:23:53Z

Description

In order to reduce the visibility delay of replicas in Segment Replication, we propose the idea of merged segment pre-copy in 17528. After discussion, we believe that it can produce good results in both local segment replication and remote store situations.

This PR is the beginning of supporting merged segments pre-copy. We introduce MergedSegmentWarmerFactory for Local/Remote merged segment pre copy.

Related Issues

Resolves #[17528]

Experiment

Test1: Replica Lag Time

We enable pre-copy in local on-disk segment replication to test the optimization effect.
We used two 16C64GB nodes, nyc_taxis workload, 2 primary shard and 1 replica shard for testing. The test command is as follows：

nohup opensearch-benchmark execute-test --target-hosts [10.204.62.132:9425](http://10.204.62.132:9425/)  --workload=nyc_taxis --pipeline=benchmark-only --workload-params='{"bulk_indexing_clients":10,"bulk_size":5000,"index_settings":{"index.number_of_shards":2,"index.number_of_replicas":1,"index.replication.type":"SEGMENT","index.routing.allocation.total_shards_per_node":2}}' --offline --results-format=csv --results-file=/home/result.csv --kill-running-processes > nohup.out 2>&1 &

We use command _cat/segment_replication to collect indicator current_lag every 5 seconds and compare the replica lag time when closing and opening pre-copy feature.

Result1: segment replication without pre-copy

Result2: segment replication with pre-copy

The X-axis represents logical time, collected every 5 seconds. The Y-axis represents the indicator current_lag collected through command _cat/segment_replication. We can see that the maximum lag time has been reduced from 66s to 2.9s, with more than 20 times optimization.

Test2: CPU util and P99 Read Latency

At the same time, we conducted a write-search test and focused on CPU util and P99 Read Latency. We can see that there is no significant change in query latency and CPU util.

Result1: segment replication without pre-copy

Result2: segment replication with pre-copy

The analysis of the reasons is as follows. CPU-intensive tasks are performed before pre-copy. There will be no significant CPU overhead due to the introduction of pre-copy. At the same time, the time cost of merging in phase SegmentMerger#merge should be much greater than the time spent in phase IndexReaderWarmer#warm. The additional synchronization time introduced by pre-copy should not account for a high proportion in the entire merge process.

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

…pre copy Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

github-actions · 2025-04-10T17:14:50Z

❌ Gradle check result for e012362: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

ashking94 · 2025-04-10T18:19:40Z

The replication lag improvement looks promising. There appears to be up to 5-6% higher CPU utilization with pre-copy. Is this reproducible each time, or can this be considered as general variability between multiple runs?

guojialiang92 · 2025-04-11T02:49:48Z

Hi, @ashking94

The replication lag improvement looks promising.

Yeah, the improvementt is very obvious.

There appears to be up to 5-6% higher CPU utilization with pre-copy. Is this reproducible each time, or can this be considered as general variability between multiple runs?

Some explanation is needed here. Because the current value fluctuates, we generally care about the max and avg CPU utilization.

Taking node lf-byte-es-sr-dataplatform-two-data-0 as an example, we can see that when pre-copy is enabled, the maximum CPU utilization is reduced by 4.1% (from 67.3% to 63.2%), and the average CPU utilization is slightly higher by 0.6% (from 54.5% to 55.1%).

At the same time, we repeated the test of enabling pre-copy.

It can be seen that the average CPU utilization of lf-byte-es-sr-dataplatform-two-data-0 has decreased by 2.7% (from 54.5% to 51.8%).

In summary, merged segments pre copy hardly introduces additional CPU overhead.

server/src/main/java/org/opensearch/index/engine/InternalEngine.java

guojialiang92 · 2025-04-15T02:40:37Z

Thanks, @ashking94

LGTM, do address the unresolved comments if any.

All comments have been resolved, looking forward to merging the code.

ashking94 · 2025-04-15T04:56:34Z

@guojialiang92 can you resolve the conflicts?

…gment_warmer # Conflicts: # CHANGELOG.md # server/src/main/java/org/opensearch/common/util/FeatureFlags.java

github-actions · 2025-04-15T06:17:15Z

✅ Gradle check result for fa5481c: SUCCESS

guojialiang92 · 2025-04-15T06:26:36Z

@guojialiang92 can you resolve the conflicts?

@ashking94, It has been resolved.

…rWarmer (opensearch-project#17881) * support merged segment warmer Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * Introduce MergedSegmentWarmerFactory for Local/Remote merged segment pre copy Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * add FeatureFlags.MERGED_SEGMENT_WARMER_EXPERIMENTAL_FLAG Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * add an entry in CHANGELOG Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * add validation Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * modify changelog entry Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * add comment Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> * add comment Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> --------- Signed-off-by: guojialiang <guojialiang.2012@bytedance.com> Signed-off-by: Harsh Kothari <techarsh@amazon.com>

Bukhtawar · 2025-05-22T13:46:36Z

server/src/main/java/org/opensearch/index/engine/LocalMergedSegmentWarmer.java

+ *
+ * @opensearch.internal
+ */
+public class LocalMergedSegmentWarmer implements IndexWriter.IndexReaderWarmer {


Would have preferred this in a follow-up PR :)

mch2 · 2025-05-29T21:57:54Z

server/src/main/java/org/opensearch/index/engine/LocalMergedSegmentWarmer.java

+ */
+
+/*
+ * Licensed to Elasticsearch under one or more contributor


I'm very late to this pr, but we need to remove the es part of these licenses.

I'm very late to this pr, but we need to remove the es part of these licenses.

@mch2
Sorry for the late reply. I will open a PR to update licenses and invite you for review. :)

guojialiang92 added 2 commits April 10, 2025 17:33

support merged segment warmer

4ab033c

Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

Introduce MergedSegmentWarmerFactory for Local/Remote merged segment …

e012362

…pre copy Signed-off-by: guojialiang <guojialiang.2012@bytedance.com>

shourya035 reviewed Apr 11, 2025

View reviewed changes

server/src/main/java/org/opensearch/index/engine/InternalEngine.java Outdated Show resolved Hide resolved

Merge remote-tracking branch 'origin/main' into dev/support_merged_se…

fa5481c

…gment_warmer # Conflicts: # CHANGELOG.md # server/src/main/java/org/opensearch/common/util/FeatureFlags.java

ashking94 approved these changes Apr 15, 2025

View reviewed changes

ashking94 merged commit 80cb033 into opensearch-project:main Apr 15, 2025
31 checks passed

opensearch-ci-bot mentioned this pull request Apr 15, 2025

[AUTOCUT] Gradle Check Flaky Test Report for MultiMatchQueryBuilderTests #17934

Closed

opensearch-ci-bot mentioned this pull request Apr 28, 2025

[AUTOCUT] Gradle Check Flaky Test Report for IndexShardTests #14321

Open

opensearch-ci-bot mentioned this pull request May 6, 2025

[AUTOCUT] Gradle Check Flaky Test Report for IndexServiceTests #14407

Open

BrewTestBot mentioned this pull request May 6, 2025

opensearch 3.0.0 Homebrew/homebrew-core#222665

Merged

opensearch-ci-bot mentioned this pull request May 8, 2025

[AUTOCUT] Gradle Check Flaky Test Report for SearchServiceTests #18238

Closed

guojialiang92 mentioned this pull request May 12, 2025

[merged segment warmer] support local merged segment warmer #18255

Merged

3 tasks

Bukhtawar reviewed May 22, 2025

View reviewed changes

mch2 reviewed May 29, 2025

View reviewed changes

guojialiang92 mentioned this pull request Jun 26, 2025

[META] Merged segment pre-copy #18625

Open

8 tasks

This was referenced Jul 9, 2025

[AUTOCUT] Gradle Check Flaky Test Report for ReindexBasicTests #15826

Open

[AUTOCUT] Gradle Check Flaky Test Report for DeleteByQueryBasicTests #15820

Open

opensearch-ci-bot mentioned this pull request Aug 9, 2025

[AUTOCUT] Gradle Check Flaky Test Report for AdmissionControlSingleNodeTests #19004

Open

opensearch-ci-bot mentioned this pull request Mar 7, 2025

[AUTOCUT] Gradle Check Flaky Test Report for RemoteIndexShardTests #16116

Open

opensearch-ci-bot mentioned this pull request Oct 22, 2025

[AUTOCUT] Gradle Check Flaky Test Report for IndexFieldDataServiceTests #19730

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Introducing MergedSegmentWarmerFactory to support the extension of IndexWriter.IndexReaderWarmer #17881

Introducing MergedSegmentWarmerFactory to support the extension of IndexWriter.IndexReaderWarmer #17881

Uh oh!

guojialiang92 commented Apr 10, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Apr 10, 2025

Uh oh!

ashking94 commented Apr 10, 2025

Uh oh!

guojialiang92 commented Apr 11, 2025 •

edited

Loading

Uh oh!

Uh oh!

guojialiang92 commented Apr 15, 2025

Uh oh!

ashking94 commented Apr 15, 2025

Uh oh!

github-actions bot commented Apr 15, 2025

Uh oh!

guojialiang92 commented Apr 15, 2025

Uh oh!

Uh oh!

Bukhtawar May 22, 2025

Uh oh!

mch2 May 29, 2025

Uh oh!

guojialiang92 Jul 13, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Introducing MergedSegmentWarmerFactory to support the extension of IndexWriter.IndexReaderWarmer #17881

Introducing MergedSegmentWarmerFactory to support the extension of IndexWriter.IndexReaderWarmer #17881

Uh oh!

Conversation

guojialiang92 commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Experiment

Test1: Replica Lag Time

Test2: CPU util and P99 Read Latency

Check List

Uh oh!

github-actions bot commented Apr 10, 2025

Uh oh!

ashking94 commented Apr 10, 2025

Uh oh!

guojialiang92 commented Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

guojialiang92 commented Apr 15, 2025

Uh oh!

ashking94 commented Apr 15, 2025

Uh oh!

github-actions bot commented Apr 15, 2025

Uh oh!

guojialiang92 commented Apr 15, 2025

Uh oh!

Uh oh!

Bukhtawar May 22, 2025

Choose a reason for hiding this comment

Uh oh!

mch2 May 29, 2025

Choose a reason for hiding this comment

Uh oh!

guojialiang92 Jul 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

guojialiang92 commented Apr 10, 2025 •

edited

Loading

guojialiang92 commented Apr 11, 2025 •

edited

Loading

guojialiang92 Jul 13, 2025 •

edited

Loading