[AUTOCUT] Gradle Check Flaky Test Report for MinimumClusterManagerNodesIT

[opensearch-ci-bot]

Flaky Test Report for MinimumClusterManagerNodesIT

Noticed the MinimumClusterManagerNodesIT has some flaky, failing tests that failed during post-merge actions.

Details

Git Reference	Merged Pull Request	Build Details	Test Name
6049587	14040	40080	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock org.opensearch.cluster.MinimumClusterManagerNodesIT.classMethod
9675c4f	14465	41398	org.opensearch.cluster.MinimumClusterManagerNodesIT.classMethod org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
0d780b6	15121	44058	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
2eb148c	15677	47308	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
3fa710b	15648	46708	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
50f411e	15582	46459	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
725ed36	15783	47574	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
9cd2635	15483	45625	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
a05d6d1	15905	47730	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
b35690c	14795	42953	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
c801270	15660	46762	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
d56d8c8	14489	41572	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
fc1bf2c	15759	47451	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
0fc94ca	13799	39263	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
1305002	14851	43091	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
56d0b76	14401	41153	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
acc4631	14587	42139	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
afa479b	14748	42455	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
c89a17c	13888	40047	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
017f7d4	15704	47021	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
107f0ce	15867	47672	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
1386a9b	13930	39885	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
2e13e9c	14107	40782	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
3a38a6c	14365	41154	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
5919409	13945	39654	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
67eceaa	15617	46561	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
9014894	15401	45300	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
dbdc151	15589	46329	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
f85a58f	14684	43162	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
fabf9bd	15293	44791	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
01c5e56	15750	47339	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
03b1306	15019	43633	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
06698dd	14922	43325	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
0c2ff03	14230	41005	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
11f8d79	14716	42338	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
1562100	15400	45193	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
1bee506	15227	44685	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
201c673	14458	41410	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
3a1be63	14639	41976	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
3ddb199	15586	46999	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
4038a3c	14074	40854	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
468f120	15724	47162	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
4c7d94c	14839	42902	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
57a597f	15494	45869	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
5bb2e28	15200	44343	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
64383dd	14561	41702	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
71d122b	15554	46063	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
7650e64	14345	41056	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
7e7e775	14864	43002	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
802f2e6	14424	41253	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
887698d	15132	44111	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
8e32ed7	14394	41336	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
903784b	14414	41239	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
913013b	13948	39666	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
a0a7098	14884	43148	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
a17aea5	14710	42244	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
a968790	15932	47815	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
afeddc2	14037	40910	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
b8c7819	12782	41391	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
bb013da	13717	39669	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
bde48a7	14133	40532	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
bf43678	13809	39614	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
c49eca4	13721	40576	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
cad81b0	15216	45855	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
d1cd7a2	15512	45861	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
d2bc9fc	15656	46780	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
e1a632f	14340	40973	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
e67ced7	13784	39156	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
edcbfd4	14923	43290	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
f9d15df	15715	47146	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
fef2003	15181	44308	org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock

The other pull requests, besides those involved in post-merge actions, that contain failing tests with the MinimumClusterManagerNodesIT class are:

• 14119
• 13655
• 15774
• 15295
• 13821
• 14774
• 14696
• 14853
• 15448
• 15569
• 15890
• 13772
• 13788
• 14652
• 15145
• 15343
• 13957
• 14069
• 14135
• 14977
• 15017
• 15239
• 15291
• 15327
• 15346
• 15372
• 15381
• 15570
• 15622
• 13703
• 14019
• 14115
• 14223
• 14266
• 14479
• 14614
• 14630
• 14708
• 15177
• 15210
• 15326
• 15410
• 15430
• 15614
• 15662
• 15681
• 15951
• 15980
• 12439
• 13636
• 13813
• 14166
• 14200
• 14399
• 14409
• 14515
• 14617
• 14738
• 14991
• 15370
• 15416
• 15533
• 15613
• 15621
• 15916
• 13374
• 13498
• 13590
• 13708
• 13785
• 13801
• 13807
• 13865
• 13897
• 13924
• 14025
• 14064
• 14167
• 14391
• 14432
• 14533
• 14565
• 14574
• 14655
• 14677
• 14735
• 14750
• 14759
• 14885
• 14925
• 14963
• 14972
• 15079
• 15153
• 15278
• 15290
• 15432
• 15442
• 15521
• 15526
• 15542
• 15664
• 15667
• 15668
• 15705
• 15972
• 15973
• 11573
• 12016
• 13131
• 13172
• 13315
• 13474
• 13525
• 13574
• 13632
• 13637
• 13678
• 13681
• 13684
• 13722
• 13759
• 13810
• 13885
• 13887
• 13895
• 13934
• 13944
• 13969
• 13997
• 14026
• 14095
• 14111
• 14140
• 14151
• 14158
• 14206
• 14238
• 14261
• 14369
• 14371
• 14397
• 14400
• 14402
• 14411
• 14415
• 14426
• 14437
• 14440
• 14454
• 14487
• 14512
• 14528
• 14540
• 14612
• 14613
• 14625
• 14635
• 14651
• 14659
• 14670
• 14673
• 14703
• 14715
• 14717
• 14725
• 14761
• 14792
• 14814
• 14817
• 14827
• 14855
• 14874
• 14876
• 14896
• 14905
• 14913
• 14949
• 14962
• 14967
• 14994
• 15012
• 15045
• 15188
• 15218
• 15230
• 15256
• 15305
• 15312
• 15320
• 15330
• 15336
• 15357
• 15374
• 15382
• 15395
• 15398
• 15409
• 15426
• 15428
• 15454
• 15465
• 15471
• 15476
• 15496
• 15508
• 15513
• 15523
• 15530
• 15565
• 15574
• 15578
• 15587
• 15591
• 15595
• 15598
• 15611
• 15615
• 15627
• 15636
• 15640
• 15643
• 15649
• 15654
• 15659
• 15671
• 15672
• 15679
• 15689
• 15690
• 15708
• 15740
• 15755
• 15765
• 15778
• 15859
• 15894
• 15940
• 15941
• 15949
• 15975
• 15997
• 16009

For more details on the failed tests refer to OpenSearch Gradle Check Metrics dashboard.

[andrross]

Adding the Storage:Remote label to this one because I believe it has been traced back to a commit related to that feature. From the original issue:

I believe I have traced this back to the commit that introduced the flakiness: 9119b6d (#9105)

The following command will reliably reproduce the failure for me:

./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock" -Dtests.iters=100

If I select the commit immediately preceding 9119b6d then it does not reproduce.

This is a bit concerning because the commit in question is related to the remote store feature but MinimumClusterManagerNodesIT does not do anything related to remote store, so it is possible there is a significant regression here.

[dbwiddis]

Mostly just adding some debugging logging statements.

1. We start out with 3 nodes, [node_t0, node_t1, node_t2]
2. We find the set of non-CM nodes, [node_t1, node_t0]
3. We shut down the non-CM nodes, leaving [node_t2]
4. We use the local path of the two nodes shut down to start up new nodes, they have the same UUID
5. Most of the time when the test passes, the new nodes are renamed, node_t0 -> node_t3 and node_t1 -> node_t4.
6. When the test fails, it's consistently because the (formerly CM) node still thinks it's in a cluster with node_t0 and node_t1 and its cluster state version is 2 versions behind the other two nodes.
7. The other two (new) nodes think that node_t2 is the cluster manager but it hasn't caught up yet.
8. The 2nd cluster state update is likely the cluster manager assignment, so the root cause is probably the first cluster state update that is failing on the (formerly cluster manager) node:

java.lang.AssertionError: a started primary with non-pending operation term must be in primary mode [test][1], node[ZdcgPV1JSmut1DojEIhCEw], [P], s[STARTED], a[id=Yl7dClDeQ0Ox4vlafvVO_A]
        at __randomizedtesting.SeedInfo.seed([D54CD0A4D377FB88]:0)
        at org.opensearch.index.shard.IndexShard.updateShardState(IndexShard.java:840)
        at org.opensearch.indices.cluster.IndicesClusterStateService.updateShard(IndicesClusterStateService.java:712)
        at org.opensearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:651)
        at org.opensearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:294)
        at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:626)
        at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:612)
        at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:580)
        at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:503)
        at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:205)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:923)
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:283)
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:246)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

[dbwiddis]

Adding a flush after the 2 nodes are randomly dropped seems effective in preventing the flakiness, but also takes a long time

client().admin().indices().prepareFlush().execute().actionGet();

Adding a refresh() fails at this point because there is no cluster manager.

[dbwiddis]

Placing a refresh() between the two node terminations seems to reduce, but not eliminate, the flakiness.

I'm about at the limit of what debug logging can tell me, but I'd suggest someone with knowledge of the linked PR investigate the interaction of that code with the cluster state.