[Concurrent Segment Search] Enable Search ITs with Concurrent Segment Search

[yigithub]

Currently Search ITs do not run with Concurrent Search feature. We need to identify and enable the suite of existing IT that needs to run with this feature.

[mch2]

@yigithub These look like sub-tasks with no description. Could you please link these back to a meta issue along with #7439, #7438, #7437 ?

[anasalkouz]

@yigithub Please make sure to add the details for this issues

[jed326]

Adding some more details --

As of #7356 there are now 3 ways to control Concurrent Segment Search: 1.) Feature Flag (Not Dynamic) 2.) Dynamic Cluster Setting 3.) Dynamic Index Setting

Currently, the Feature Flag is disabled for (almost) all java ITs so we do not have automated test coverage for concurrent search. The goal of this task is to come up with a way to automatically run the java ITs with concurrent segment search enabled.

Off the top of my head one solution is to enable the Concurrent Segment Search Feature Flag and then parameterize the tests using the dynamic cluster setting. However we would like to avoid adding the parameterization code to every single test if possible, so it would be good to come up with a solution where all of the controls are in one place.

[neetikasinghal]

I will be working on this.

[neetikasinghal]

Created an issue with EE team to ask about a custom jenkins job to run the following command that run the entire test suite with concurrent search enabled:
./gradlew :server:internalClusterTest --tests "org.opensearch.search.*" -Dopensearch.experimental.feature.concurrent_segment_search.enabled=true

[jed326]

Remaining tasks are tracked in #9443