[BUG] [Segment Replication] Segment replication is broken for plugins using custom codecs #7781

dreamer-89 · 2023-05-26T20:40:54Z

What is the bug?
Segment replicaion event fails for plugins using custom codecs (e.g. kNN). The bug prevents replica shard from allocation resulting in forever unassigned replica.

Background

During peer recovery, for segment replication enabled indices a force segment sync is performed to keep the shard upto date from primary. Recently, we added a fix where to prevent segment replication events b/w primary and replica when they are using a different codec implementations. The fix had a bug where replica shard's default codec was used rather than the one on the engine config.

How can one reproduce the bug?
Steps to reproduce the behavior:

Create 2.7.0 OpenSearch cluster.
Install kNN plugin
Create index with kNN plugin with at least one replica

curl -X PUT "$host/test-index" -H 'Content-Type: application/json' -d '{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "replication.type": "SEGMENT",
      "knn": true
    }
  }
}'

See error in nodes on which replica shard recovery is attempted (enable trace logs to see primary shard codec)

org.opensearch.indices.recovery.RecoveryFailedException: [test-index][0]: Recovery failed from {ip-10-0-5-73.us-west-2.compute.internal}{KcxMm3vQRmOSQYCXAs8fNQ}{M8Xgt2zLS1CNSB0A6-P0FQ}{10.0.5.73}{10.0.5.73:9300}{di}{shard_indexing_pressure_enabled=true} into {ip-10-0-3-221.us-west-2.compute.internal}{moqR5mBCRbqL7ZVGZTB4ww}{M-c6kHBURUigjyyjzFnhbw}{10.0.3.221}{10.0.3.221:9300}{di}{shard_indexing_pressure_enabled=true} ([test-index][0]: Recovery failed from {ip-10-0-5-73.us-west-2.compute.internal}{KcxMm3vQRmOSQYCXAs8fNQ}{M8Xgt2zLS1CNSB0A6-P0FQ}{10.0.5.73}{10.0.5.73:9300}{di}{shard_indexing_pressure_enabled=true} into {ip-10-0-3-221.us-west-2.compute.internal}{moqR5mBCRbqL7ZVGZTB4ww}{M-c6kHBURUigjyyjzFnhbw}{10.0.3.221}{10.0.3.221:9300}{di}{shard_indexing_pressure_enabled=true})
        at org.opensearch.indices.recovery.RecoveryTarget.notifyListener(RecoveryTarget.java:134) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.indices.replication.common.ReplicationTarget.fail(ReplicationTarget.java:176) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.indices.replication.common.ReplicationCollection.fail(ReplicationCollection.java:192) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.indices.recovery.PeerRecoveryTargetService$RecoveryResponseHandler.onException(PeerRecoveryTargetService.java:743) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.indices.recovery.PeerRecoveryTargetService$RecoveryResponseHandler.handleException(PeerRecoveryTargetService.java:669) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1414) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:420) [opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [opensearch-2.7.0.jar:2.7.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.opensearch.indices.recovery.RecoveryFailedException: [test-index][0]: Recovery failed from {ip-10-0-5-73.us-west-2.compute.internal}{KcxMm3vQRmOSQYCXAs8fNQ}{M8Xgt2zLS1CNSB0A6-P0FQ}{10.0.5.73}{10.0.5.73:9300}{di}{shard_indexing_pressure_enabled=true} into {ip-10-0-3-221.us-west-2.compute.internal}{moqR5mBCRbqL7ZVGZTB4ww}{M-c6kHBURUigjyyjzFnhbw}{10.0.3.221}{10.0.3.221:9300}{di}{shard_indexing_pressure_enabled=true}
        ... 8 more
Caused by: org.opensearch.transport.RemoteTransportException: [ip-10-0-5-73.us-west-2.compute.internal][10.0.5.73:9300][internal:index/shard/recovery/start_recovery]
Caused by: org.opensearch.transport.RemoteTransportException: [ip-10-0-3-221.us-west-2.compute.internal][10.0.3.221:9300][internal:index/shard/replication/segments_sync]
Caused by: org.opensearch.indices.replication.common.ReplicationFailedException: [test-index][0]: Replication failed on
        at org.opensearch.indices.replication.SegmentReplicationTargetService$3.onFailure(SegmentReplicationTargetService.java:362) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.ActionListener$1.onFailure(ActionListener.java:88) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.ActionRunnable.onFailure(ActionRunnable.java:103) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:54) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:343) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:120) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:112) ~[opensearch-2.7.0.jar:2.7.0]
        at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
        at org.opensearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:112) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.BaseFuture.setException(BaseFuture.java:178) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.ListenableFuture.onFailure(ListenableFuture.java:149) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.StepListener.innerOnFailure(StepListener.java:82) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.NotifyOnceListener.onFailure(NotifyOnceListener.java:62) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.ActionListener$4.onFailure(ActionListener.java:190) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.ActionListener$6.onFailure(ActionListener.java:309) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.support.RetryableAction$RetryingListener.onFinalFailure(RetryableAction.java:218) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.support.RetryableAction$RetryingListener.onFailure(RetryableAction.java:210) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:74) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1414) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:420) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.7.0.jar:2.7.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.opensearch.common.util.CancellableThreads$ExecutionCancelledException: ParameterizedMessage[messagePattern=Requested unsupported codec version {}, stringArgs=[Lucene95], throwable=null]
        at org.opensearch.indices.replication.OngoingSegmentReplications.prepareForReplication(OngoingSegmentReplications.java:154) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.indices.replication.SegmentReplicationSourceService$CheckpointInfoRequestHandler.messageReceived(SegmentReplicationSourceService.java:138) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.indices.replication.SegmentReplicationSourceService$CheckpointInfoRequestHandler.messageReceived(SegmentReplicationSourceService.java:119) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:108) ~[?:?]
        at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:453) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) ~[opensearch-2.7.0.jar:2.7.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.7.0.jar:2.7.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]

Replica shards remain unassigned and will remain forever because all shard replica allocation tries goes through same code flow.

➜  opensearch-cluster-cdk git:(main) ✗ curl -s -X GET "$host/_cat/shards?v"
index                     shard prirep state      docs store ip         node
test-index                0     p      STARTED       0  230b 10.0.5.73  ip-10-0-5-73.us-west-2.compute.internal
test-index                0     r      UNASSIGNED                       
.opensearch-observability 0     r      STARTED       0  208b 10.0.5.73  ip-10-0-5-73.us-west-2.compute.internal
.opensearch-observability 0     r      STARTED       0  208b 10.0.3.221 ip-10-0-3-221.us-west-2.compute.internal
.opensearch-observability 0     p      STARTED       0  208b 10.0.3.241 seed
.kibana_1                 0     p      STARTED       1 5.1kb 10.0.3.241 seed
.kibana_1                 0     r      STARTED       1 5.1kb 10.0.4.116 ip-10-0-4-116.us-west-2.compute.internal

This bug is also manifested when an older lucene codec is used during index creation

curl -X PUT "$host/test-index-lucene94" -H 'Content-Type: application/json' -d '{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "replication.type": "SEGMENT",
      "codec": "Lucene94"
    }
  }
}'

What is the expected behavior?
Replica shards should be assigned for all type of plugins

What is your host/environment?

OS: linux
Version [e.g. 22]: 2.7.0
Plugins kNN

The text was updated successfully, but these errors were encountered:

dreamer-89 · 2023-05-26T23:17:54Z

This error is also manifested when using an older lucene codec implementation during index creation. One such example (verified on OS 2.7.0)

curl -X PUT "$host/test-index-lucene80" -H 'Content-Type: application/json' -d '{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "replication.type": "SEGMENT",
      "codec": "Lucene80"
    }
  }
}'

dreamer-89 · 2023-05-26T23:18:29Z

Created backports to 2.7 for removing string matching and use of default codec.

[Segment Replication] Use engine codec and replica shard ReplicationCheckpoint for replication events #7789
[Segment Replication] Remove codec name string match check for checkp… #7786

Bukhtawar · 2023-05-27T11:00:55Z

Thanks for raising this issue. I am curious to hear if an integration tests with plugins installed could have helped detect this earlier. @mch2 @andrross

dreamer-89 · 2023-05-30T22:29:38Z

After this fix backported, verified this bug is not present during OS upgrades from 2.7.1 -> 2.8.0 for kNN plugin

dreamer-89 · 2023-05-31T17:17:48Z

Closing this issue as the bug is fixed and backports PRs have been merged to 2.7 branch and will be released in 2.7.1

Verified with fix on 2.7 branch

Segrep does break for plugin integration providing custom codecs e.g. kNN (failure case 1)
Segrep does not break for indices using older codec e.g. Lucene80 (failure case 2)
Rolling upgrades from 2.7 to 2.8.0 for kNN index

Documentation change will be tracked in opensearch-project/documentation-website#4194

To prevent such future occurrence, taken up below action items.

Add tests in core (tracked here) Completed with [Segment Replication] Add lucene codec tests #7903
- Index using older codecs
- Index using random compression mode (best_speed, best_compression, default)
Basic sanity tests in kNN repo
- Tracked in [Segment Replication] Add tests for Segment replication feature k-NN#921

dreamer-89 added bug Something isn't working untriaged labels May 26, 2023

This was referenced May 26, 2023

[DOC] Add documentation for segment replication bug impacting plugins opensearch-project/documentation-website#4194

Closed

[Segment Replication] Add tests for Segment replication feature opensearch-project/k-NN#921

Open

anasalkouz added distributed framework and removed untriaged labels May 26, 2023

dreamer-89 added this to Segment Replication May 30, 2023

github-project-automation bot moved this to Todo in Segment Replication May 30, 2023

dreamer-89 moved this from Todo to Done in Segment Replication May 30, 2023

dreamer-89 closed this as completed May 31, 2023

dreamer-89 self-assigned this May 31, 2023

This was referenced Jun 2, 2023

[Segment Replication] Add lucene codec tests #7903

Merged

Add basic integration test for segment replication opensearch-project/k-NN#927

Closed

dreamer-89 mentioned this issue Jun 12, 2023

[Segment Replication] Create dummy test plugin to exercise custom codecs and add tests #8023

Open

dreamer-89 mentioned this issue Jun 22, 2023

[Meta] Validate plugins compatibility with segment replication #8211

Closed

37 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] [Segment Replication] Segment replication is broken for plugins using custom codecs #7781

[BUG] [Segment Replication] Segment replication is broken for plugins using custom codecs #7781

dreamer-89 commented May 26, 2023 •

edited

Loading

dreamer-89 commented May 26, 2023

dreamer-89 commented May 26, 2023 •

edited

Loading

Bukhtawar commented May 27, 2023

dreamer-89 commented May 30, 2023 •

edited

Loading

dreamer-89 commented May 31, 2023 •

edited

Loading

[BUG] [Segment Replication] Segment replication is broken for plugins using custom codecs #7781

[BUG] [Segment Replication] Segment replication is broken for plugins using custom codecs #7781

Comments

dreamer-89 commented May 26, 2023 • edited Loading

Background

dreamer-89 commented May 26, 2023

dreamer-89 commented May 26, 2023 • edited Loading

Bukhtawar commented May 27, 2023

dreamer-89 commented May 30, 2023 • edited Loading

dreamer-89 commented May 31, 2023 • edited Loading

dreamer-89 commented May 26, 2023 •

edited

Loading

dreamer-89 commented May 26, 2023 •

edited

Loading

dreamer-89 commented May 30, 2023 •

edited

Loading

dreamer-89 commented May 31, 2023 •

edited

Loading