[CI] FieldCapabilitiesIT testRelocation failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #19533 / part1
• elasticsearch-intake #19148 / part1

Reproduction Line:

./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.search.fieldcaps.FieldCapabilitiesIT.testRelocation" -Dtests.seed=BAB28790226049BF -Dtests.locale=bas -Dtests.timezone=Africa/Kigali -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: safeGet: listener was not completed within the timeout

Issue Reasons:

• [main] 2 failures in test testRelocation (0.2% fail rate in 915 executions)
• [main] 2 failures in step part1 (1.9% fail rate in 107 executions)
• [main] 2 failures in pipeline elasticsearch-intake (1.9% fail rate in 107 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testRelocation (0.2% fail rate in 915 executions)
• [main] 2 failures in step part1 (1.9% fail rate in 107 executions)
• [main] 2 failures in pipeline elasticsearch-intake (1.9% fail rate in 107 executions)

Build Scans:

• elasticsearch-intake #19533 / part1
• elasticsearch-intake #19148 / part1

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[kkrik-es]

@dnhatn any update on this?

[dnhatn]

    java.lang.AssertionError: safeGet: listener was completed exceptionally
        at __randomizedtesting.SeedInfo.seed([BDC4E09BF3DB816A]:0)
        at org.elasticsearch.test.ESTestCase.safeGet(ESTestCase.java:2497)
        at org.elasticsearch.action.admin.cluster.reroute.ClusterRerouteUtils.doReroute(ClusterRerouteUtils.java:47)
        at org.elasticsearch.action.admin.cluster.reroute.ClusterRerouteUtils.reroute(ClusterRerouteUtils.java:35)
        at org.elasticsearch.search.fieldcaps.FieldCapabilitiesIT.moveOrCloseShardsOnNodes(FieldCapabilitiesIT.java:624)
        at org.elasticsearch.search.fieldcaps.FieldCapabilitiesIT.lambda$testRelocation$2(FieldCapabilitiesIT.java:646)

        Caused by:
        java.util.concurrent.ExecutionException: org.elasticsearch.transport.RemoteTransportException: [node_s1][127.0.0.1:21401][cluster:admin/reroute]
            at org.elasticsearch.action.support.PlainActionFuture$Sync.getValue(PlainActionFuture.java:279)
            at org.elasticsearch.action.support.PlainActionFuture$Sync.get(PlainActionFuture.java:253)
            at org.elasticsearch.action.support.PlainActionFuture.get(PlainActionFuture.java:74)
            at org.elasticsearch.test.ESTestCase.safeGet(ESTestCase.java:2492)
            ... 4 more

            Caused by:
            org.elasticsearch.transport.RemoteTransportException: [node_s1][127.0.0.1:21401][cluster:admin/reroute]

                Caused by:
                java.lang.IllegalArgumentException: [move_allocation] can't move [new_index][8], shard is not started (state = INITIALIZING]
                    at org.elasticsearch.cluster.routing.allocation.command.MoveAllocationCommand.execute(MoveAllocationCommand.java:190)
                    at org.elasticsearch.cluster.routing.allocation.command.AllocationCommands.execute(AllocationCommands.java:75)
                    at org.elasticsearch.cluster.routing.allocation.allocator.ShardsAllocator.execute(ShardsAllocator.java:67)
                    at org.elasticsearch.cluster.routing.allocation.allocator.DesiredBalanceShardsAllocator.execute(DesiredBalanceShardsAllocator.java:300)
                    at org.elasticsearch.cluster.routing.allocation.AllocationService.reroute(AllocationService.java:425)
                    at org.elasticsearch.action.admin.cluster.reroute.TransportClusterRerouteAction$ClusterRerouteResponseAckedClusterStateUpdateTask.execute(TransportClusterRerouteAction.java:270)
                    at org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:580)
                    at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1076)
                    at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1039)
                    at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:246)
                    at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1692)
                    at org.elasticsearch.action.ActionListener.run(ActionListener.java:463)
                    at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1689)
                    at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1284)
                    at org.elasticsearch.action.ActionListener.run(ActionListener.java:463)
                    at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1263)
                    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
                    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
                    at java.lang.Thread.run(Thread.java:1575)

@kkrik-es working on a fix now.