[CI] ClusterDisruptionIT testAckedIndexing failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #10733 / windows-2025_checkpart1_platform-support-windows
• elasticsearch-pull-request #93737 / part-1
• elasticsearch-pull-request #93698 / part-1

Reproduction Line:

gradlew ":server:internalClusterTest" --tests "org.elasticsearch.discovery.ClusterDisruptionIT.testAckedIndexing" -Dtests.seed=4EFF5B1206673C42 -Dtests.locale=de-LU -Dtests.timezone=SystemV/PST8PDT -Druntime.java=25

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: timed out waiting for green state

Issue Reasons:

• [main] 3 failures in test testAckedIndexing (1.7% fail rate in 179 executions)
• [main] 2 failures in step part-1 (5.1% fail rate in 39 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (5.1% fail rate in 39 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testAckedIndexing (1.2% fail rate in 172 executions)

Build Scans:

• elasticsearch-periodic-platform-support #4919 / oraclelinux-7_platform-support-unix
• elasticsearch-intake #13071 / part1

[Tim-Brooks]

Concerning for [elasticsearch-periodic-platform-support #4919 / oraclelinux-7_platform-support-unix](https://gradle-enterprise.elastic.co/s/h3hx2chmhinck).

  2> java.lang.AssertionError: doc [14] exists [1] times in index
  2> 	at __randomizedtesting.SeedInfo.seed([7E3645A3E1F0C921]:0)
  2> 	at org.elasticsearch.index.engine.InternalEngine.assertDocDoesNotExist(InternalEngine.java:1586)
  2> 	at org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1415)
  2> 	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1219)
  2> 	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1084)
  2> 	at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1010)
  2> 	at org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:2006)
  2> 	at org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1993)
  2> 	at org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$4(RecoveryTarget.java:454)
  2> 	at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:356)
  2> 	at org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:429)
  2> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.performTranslogOps(PeerRecoveryTargetService.java:658)
  2> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.handleRequest(PeerRecoveryTargetService.java:605)
  2> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.handleRequest(PeerRecoveryTargetService.java:597)
  2> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRequestHandler.messageReceived(PeerRecoveryTargetService.java:691)
  2> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRequestHandler.messageReceived(PeerRecoveryTargetService.java:678)
  2> 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:76)
  2> 	at org.elasticsearch.transport.InboundHandler.doHandleRequest(InboundHandler.java:289)
  2> 	at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:302)
  2> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)
  2> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
  2> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  2> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  2> 	at java.base/java.lang.Thread.run(Thread.java:1575)

Less concerning error for [elasticsearch-intake #13071 / part1](https://gradle-enterprise.elastic.co/s/ixl2x4hz6v6sk)

  2> REPRODUCE WITH: ./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.discovery.ClusterDisruptionIT.testAckedIndexing" -Dtests.seed=13417668B863C4D3 -Dtests.locale=bho-IN -Dtests.timezone=Asia/Dili -Druntime.java=23
  2> java.lang.AssertionError: org.elasticsearch.cluster.coordination.ValidateJoinRequest/unset vs org.elasticsearch.transport.BytesTransportRequest/unset
        at __randomizedtesting.SeedInfo.seed([13417668B863C4D3]:0)
        at org.elasticsearch.test.transport.MockTransportService$3.sendRequest(MockTransportService.java:589)
        at org.elasticsearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:251)
        at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:982)
        at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:877)
        at org.elasticsearch.cluster.coordination.JoinValidationService$JoinValidation.doRun(JoinValidationService.java:322)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
        at org.elasticsearch.cluster.coordination.JoinValidationService.processNextItem(JoinValidationService.java:227)
        at org.elasticsearch.cluster.coordination.JoinValidationService$1.doRun(JoinValidationService.java:197)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1575)

[Tim-Brooks]

I unmuted this as I fixed one of the failure scenarios identified. We can mute again if it goes back to failing.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test testAckedIndexing (1.8% fail rate in 171 executions)

Build Scans:

• elasticsearch-periodic #4970 / openjdk23_checkpart1_java-matrix
• elasticsearch-periodic-platform-support #4919 / oraclelinux-7_platform-support-unix
• elasticsearch-intake #13071 / part1