Add SearchExtBuilders to SearchResponse

[austintlee]

Description

Adds support for SearchExtBuilders to SearchResponse so that search processors can include "ext" sections in the response with information specific to individual processors.

Related Issues

#9328

Check List

• [x ] New functionality includes testing.
  • [x ] All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• [x ] Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[austintlee]

Also, somewhat related to #6794.

[opensearch-trigger-bot]

Compatibility status:

Checks if related components are compatible with change 4871e25

Incompatible components

Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

• URL: https://build.ci.opensearch.org/job/gradle-check/22859/
• CommitID: 15857c3
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #9379 (8c88507) into main (012c4fa) will increase coverage by 0.37%.
Report is 2 commits behind head on main.
The diff coverage is 80.73%.

@@             Coverage Diff              @@
##               main    #9379      +/-   ##
============================================
+ Coverage     70.40%   70.77%   +0.37%     
- Complexity    56861    57177     +316     
============================================
  Files          4781     4782       +1     
  Lines        271231   271332     +101     
  Branches      39599    39614      +15     
============================================
+ Hits         190947   192039    +1092     
+ Misses        63974    62979     -995     
- Partials      16310    16314       +4     

Files Changed	Coverage Δ
...org/opensearch/search/GenericSearchExtBuilder.java	73.84% <73.84%> (ø)
...a/org/opensearch/action/search/SearchResponse.java	90.21% <75.00%> (-0.66%)	⬇️
.../org/opensearch/action/get/TransportGetAction.java	62.22% <100.00%> (-3.78%)	⬇️
...ensearch/action/search/SearchResponseSections.java	97.43% <100.00%> (+1.13%)	⬆️
...main/java/org/opensearch/cluster/ClusterState.java	97.87% <100.00%> (+0.02%)	⬆️
...search/search/internal/InternalSearchResponse.java	100.00% <100.00%> (ø)

... and 611 files with indirect coverage changes

[reta]

• URL: https://build.ci.opensearch.org/job/gradle-check/23723/

@austintlee I see a large number of failing tests that are not flaky, seems to be related to the change? Could you please take a look?

[austintlee]

@reta I started to go through the failing ones. I am not able to repro (the first two I picked both passed locally).

I do have one issue with running the BWC tests as they require a dependency that is not available for my Mac machine (darwin-arm64). I will need to find a Linux box to repro.

Project :qa:mixed-cluster declares a dependency from configuration 'opensearch_distro_extracted_testclusters-qa-mixed-cluster-v2.10.0-2-2.10.0-' to configuration 'expanded-darwin-arm64-tar' which is not declared in the descriptor for project :distribution:bwc:minor.

[reta]

@reta I started to go through the failing ones. I am not able to repro (the first two I picked both passed locally).

Let me help you out

[austintlee]

This one passed locally:

./gradlew ':server:internalClusterTest' --tests "org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockIsRemovedWhenAnyNodesNotExceedHighWatermarkWithAutoReleaseEnabled" -Dtests.seed=32C89B49D45BA12A -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-GT -Dtests.timezone=America/Lima -Druntime.java=20

Not sure if this is known to be flaky. If there is a list of known flaky tests, I can also cross check.

[austintlee]

This one passed as well:

./gradlew ':client:rest-high-level:asyncIntegTest' --tests "org.opensearch.client.PitIT.testDeleteAllAndListAllPits" -Dtests.seed=12C4A96B6731F79E -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=mt-MT -Dtests.timezone=Africa/Maseru -Druntime.java=20

[austintlee]

I am going to try the instructions given in opensearch-project/k-NN#580 to build that missing dependency locally.

[reta]

@austintlee please rebase with main (it seems like you are way behind), the tests should be back on track

[austintlee]

Does "Sync fork" on Github accomplish the same?

[reta]

Does "Sync fork" on Github accomplish the same?

I don't think so, you will sync main with main (afaik) but you have another branch (searchresponse-ext)

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/23735/
• CommitID: 8c88507
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[austintlee]

Down to one failure and this one does fail consistently locally as well.

./gradlew ':server:test' --tests "org.opensearch.index.shard.RemoteIndexShardTests.testNRTReplicaWithRemoteStorePromotedAsPrimaryCommitCommit" -Dtests.seed=A9EE596955D2AAD8 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=es-US -Dtests.timezone=Asia/Dushanbe -Druntime.java=20

[austintlee]

It passes in my IDE (Intellij)...

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      2 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.classMethod
      1 org.opensearch.index.shard.RemoteStoreRefreshListenerTests.testReplicaPromotion

• URL: https://build.ci.opensearch.org/job/gradle-check/23742/
• CommitID: 8c88507
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 8c88507

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9379-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 60787b869cb800eab53417e87a6399ed680de060
# Push it to GitHub
git push --set-upstream origin backport/backport-9379-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9379-to-2.x.

[reta]

@austintlee could you please send manual backport to 2.x branch? (please don't forget to change onOrAfter(3_0_0) to onOrAfter(2_10_0), thank you

[austintlee]

@austintlee could you please send manual backport to 2.x branch? (please don't forget to change onOrAfter(3_0_0) to onOrAfter(2_10_0), thank you

I'm on it!