Fix deadlock in SearchPhaseControllerTests cancellation tests (opensearch-project#19171)

The cancellation tests could deadlock when threads are delayed by OS
scheduling. If cancellation triggers before all threads start, late
threads may hit a code path where batchReduceSize causes the latch
callback to be deferred to a MergeTask. Under certain timing conditions,
these callbacks never execute, causing latch.await() to hang indefinitely.

Ensure latch.countDown() is always called by wrapping consumeResult in
try-catch. This guarantees test completion regardless of cancellation
timing or exceptions.

Signed-off-by: Atri Sharma <atri.jiit@gmail.com>
Signed-off-by: Ankit Jain <jainankitk@apache.org>

Author: atris

server/src/test/java/org/opensearch/action/search/SearchPhaseControllerTests.java

@@ -1826,7 +1826,13 @@ private static void consumeShardLevelQueryPhaseResultsAsync(int expectedNumResul
                 result.setShardIndex(index);
                 result.size(1);
 
-                consumer.consumeResult(result, latch::countDown);
+                try {
+                    consumer.consumeResult(result, latch::countDown);
+                } catch (Exception e) {
+                    // Ensure latch counts down even on cancellation
+                    latch.countDown();
+                    // Don't rethrow - let the thread complete normally
+                }
             });
             threads[index].start();
         }