Adding the SearchPhaseResultsProcessor interface in Search Pipeline #7283

navneet1v · 2023-04-24T00:51:16Z

Description

Add new search pipeline processor type, SearchPhaseResultsProcessor, that can modify the result of one search phase before starting the next phase.

Along with this, added the code to resolve the Search pipeline once and added new SearchRequest type PipelinedRequest.

Issues Resolved

RFC: opensearch-project/neural-search#152

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-04-24T00:58:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/14458/
CommitID: b70ef2e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-04-24T01:09:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/14457/
CommitID: 7778588
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/search/pipeline/SearchPipelineService.java

github-actions · 2023-06-28T20:32:52Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/18607/
CommitID: db8b3e0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

reta · 2023-06-28T20:33:49Z

@navneet1v needs rebase please

andrross · 2023-06-28T20:36:17Z

@navneet1v I just merged the latest from main, which should fix the tests now.

github-actions · 2023-06-28T21:09:15Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure
      1 org.opensearch.snapshots.RestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex
      1 org.opensearch.search.pit.DeletePitMultiNodeIT.testDeleteWhileSearch
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication
      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testBasicTaskResourceTracking

URL: https://build.ci.opensearch.org/job/gradle-check/18611/
CommitID: 81baeda
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

msfroh · 2023-06-28T22:56:16Z

I'm working on resolving the (many) conflicts that arose from #8053 getting merged.

I'll post the new commits shortly.

navneet1v · 2023-06-28T23:02:07Z

I'm working on resolving the (many) conflicts that arose from #8053 getting merged.

I'll post the new commits shortly.

Thanks @msfroh for picking it up. You can only resolve these conflicts. :)

msfroh · 2023-06-28T23:22:38Z

Ugh -- I think I clobbered @navneet1v's recent changes by force-pushing after rebase.

I'm going to see if I can recover them (or reapply them manually).

github-actions · 2023-06-28T23:43:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication
      1 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure
      1 org.opensearch.indices.replication.SegmentReplicationIT.testCancellation
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

URL: https://build.ci.opensearch.org/job/gradle-check/18624/
CommitID: 096417c
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-06-29T00:08:50Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.snapshots.RestoreSnapshotIT.testRestoreInSameRemoteStoreEnabledIndex
      1 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication

URL: https://build.ci.opensearch.org/job/gradle-check/18628/
CommitID: 81baeda
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Michael Froh <froh@amazon.com>

github-actions · 2023-06-29T00:57:27Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testNodeDropWithOngoingReplication
      1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocation

URL: https://build.ci.opensearch.org/job/gradle-check/18633/
CommitID: b35858d
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-06-29T01:02:07Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7283-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b33979a4d95f728ee3080eb6983d93e024fb7ba2
# Push it to GitHub
git push --set-upstream origin backport/backport-7283-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7283-to-2.x.

reta · 2023-06-29T01:04:19Z

@navneet1v needs manual backport to 2.x sadly, could you please take care of it? thank you.

navneet1v · 2023-06-29T01:54:43Z

@navneet1v needs manual backport to 2.x sadly, could you please take care of it? thank you.

Yes I will take care of that. Thanks everyone. This PR was pending for long time and took more effort than I expected.

…pensearch-project#7283) * Initial code for adding the SearchPhaseInjectorProcessor interface in Search Pipeline Signed-off-by: Navneet Verma <navneev@amazon.com> * Pass PipelinedRequest to SearchAsyncActions We should resolve a search pipeline once at the start of a search request and then propagate that pipeline through the async actions. When completing a search phase, we will then use that pipeline to inject behavior (if applicable). Signed-off-by: Michael Froh <froh@amazon.com> * Renamed SearchPhaseInjectorProcessor to SearchPhaseResultsProcessor and fixed the comments Signed-off-by: Navneet Verma <navneev@amazon.com> * Make PipelinedSearchRequest extend SearchRequest Rather than wrapping a SearchRequest in a PipelinedSearchRequest, changes are less intrusive if we say that a PipelinedSearchRequest "is a" SearchRequest. Signed-off-by: Michael Froh <froh@amazon.com> * Revert code change from merge conflict Signed-off-by: Michael Froh <froh@amazon.com> * Updated the changelog with more appropiate wording for the change. Signed-off-by: Navneet Verma <navneev@amazon.com> * Fixed Typos in the code Signed-off-by: Navneet Verma <navneev@amazon.com> * Fixing comments relating to return of SearchPhaseResults from processor Signed-off-by: Navneet Verma <navneev@amazon.com> * Moved SearchPhaseName enum in separate class and fixed comments. Signed-off-by: Navneet Verma <navneev@amazon.com> * Resolve remaining merge conflict Signed-off-by: Michael Froh <froh@amazon.com> --------- Signed-off-by: Navneet Verma <navneev@amazon.com> Signed-off-by: Michael Froh <froh@amazon.com> Co-authored-by: Michael Froh <froh@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com>

…7283) (#8512) Add new search pipeline processor type, SearchPhaseResultsProcessor, that can modify the result of one search phase before starting the next phase. Along with this, added the code to resolve the Search pipeline once and added new SearchRequest type PipelinedRequest. Backport of PR: #7283 --------- Signed-off-by: Navneet Verma <navneev@amazon.com> Co-authored-by: Michael Froh <froh@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com>

…pensearch-project#7283) * Initial code for adding the SearchPhaseInjectorProcessor interface in Search Pipeline Signed-off-by: Navneet Verma <navneev@amazon.com> * Pass PipelinedRequest to SearchAsyncActions We should resolve a search pipeline once at the start of a search request and then propagate that pipeline through the async actions. When completing a search phase, we will then use that pipeline to inject behavior (if applicable). Signed-off-by: Michael Froh <froh@amazon.com> * Renamed SearchPhaseInjectorProcessor to SearchPhaseResultsProcessor and fixed the comments Signed-off-by: Navneet Verma <navneev@amazon.com> * Make PipelinedSearchRequest extend SearchRequest Rather than wrapping a SearchRequest in a PipelinedSearchRequest, changes are less intrusive if we say that a PipelinedSearchRequest "is a" SearchRequest. Signed-off-by: Michael Froh <froh@amazon.com> * Revert code change from merge conflict Signed-off-by: Michael Froh <froh@amazon.com> * Updated the changelog with more appropiate wording for the change. Signed-off-by: Navneet Verma <navneev@amazon.com> * Fixed Typos in the code Signed-off-by: Navneet Verma <navneev@amazon.com> * Fixing comments relating to return of SearchPhaseResults from processor Signed-off-by: Navneet Verma <navneev@amazon.com> * Moved SearchPhaseName enum in separate class and fixed comments. Signed-off-by: Navneet Verma <navneev@amazon.com> * Resolve remaining merge conflict Signed-off-by: Michael Froh <froh@amazon.com> --------- Signed-off-by: Navneet Verma <navneev@amazon.com> Signed-off-by: Michael Froh <froh@amazon.com> Co-authored-by: Michael Froh <froh@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com>

…pensearch-project#7283) * Initial code for adding the SearchPhaseInjectorProcessor interface in Search Pipeline Signed-off-by: Navneet Verma <navneev@amazon.com> * Pass PipelinedRequest to SearchAsyncActions We should resolve a search pipeline once at the start of a search request and then propagate that pipeline through the async actions. When completing a search phase, we will then use that pipeline to inject behavior (if applicable). Signed-off-by: Michael Froh <froh@amazon.com> * Renamed SearchPhaseInjectorProcessor to SearchPhaseResultsProcessor and fixed the comments Signed-off-by: Navneet Verma <navneev@amazon.com> * Make PipelinedSearchRequest extend SearchRequest Rather than wrapping a SearchRequest in a PipelinedSearchRequest, changes are less intrusive if we say that a PipelinedSearchRequest "is a" SearchRequest. Signed-off-by: Michael Froh <froh@amazon.com> * Revert code change from merge conflict Signed-off-by: Michael Froh <froh@amazon.com> * Updated the changelog with more appropiate wording for the change. Signed-off-by: Navneet Verma <navneev@amazon.com> * Fixed Typos in the code Signed-off-by: Navneet Verma <navneev@amazon.com> * Fixing comments relating to return of SearchPhaseResults from processor Signed-off-by: Navneet Verma <navneev@amazon.com> * Moved SearchPhaseName enum in separate class and fixed comments. Signed-off-by: Navneet Verma <navneev@amazon.com> * Resolve remaining merge conflict Signed-off-by: Michael Froh <froh@amazon.com> --------- Signed-off-by: Navneet Verma <navneev@amazon.com> Signed-off-by: Michael Froh <froh@amazon.com> Co-authored-by: Michael Froh <froh@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

navneet1v requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng, VachaShah and xuezhou25 as code owners April 24, 2023 00:51

navneet1v requested a review from msfroh April 24, 2023 00:51

navneet1v force-pushed the searchPhaseInjectorProcessor branch from b70ef2e to 7945f01 Compare April 24, 2023 00:53

navneet1v changed the title ~~[]Initial code for adding the SearchPhaseInjectorProcessor interface in Search Pipeline~~ Initial code for adding the SearchPhaseInjectorProcessor interface in Search Pipeline Apr 24, 2023

navneet1v added backport 2.x Backport to 2.x branch Search Search query, autocomplete ...etc labels Apr 24, 2023

msfroh reviewed Apr 24, 2023

View reviewed changes

server/src/main/java/org/opensearch/search/pipeline/SearchPipelineService.java Outdated Show resolved Hide resolved

navneet1v force-pushed the searchPhaseInjectorProcessor branch from 7945f01 to 81f1a4a Compare April 24, 2023 18:51

navneet1v requested a review from reta June 28, 2023 19:58

reta approved these changes Jun 28, 2023

View reviewed changes

Merge branch 'main' into searchPhaseInjectorProcessor

81baeda

msfroh force-pushed the searchPhaseInjectorProcessor branch from 81baeda to 096417c Compare June 28, 2023 23:04

navneet1v force-pushed the searchPhaseInjectorProcessor branch from 096417c to 81baeda Compare June 28, 2023 23:29

Merge branch 'main' into searchPhaseInjectorProcessor

c01f4a8

Resolve remaining merge conflict

b35858d

Signed-off-by: Michael Froh <froh@amazon.com>

reta added the v2.9.0 'Issues and PRs related to version v2.9.0' label Jun 29, 2023

reta merged commit b33979a into opensearch-project:main Jun 29, 2023

navneet1v mentioned this pull request Jun 29, 2023

[RFC]: Search Phase Results Processor opensearch-project/neural-search#152

Closed

mingshl mentioned this pull request Jul 7, 2023

[Search pipelines] Pass "adhocness" flag to processor factories #8164

Merged

6 tasks

navneet1v mentioned this pull request Jul 7, 2023

[Backport 2.x] [Search Pipelines] Adding the SearchPhaseResultsProcessor interface in Search Pipeline #8512

Merged

6 tasks

martin-gaievski mentioned this pull request Jul 19, 2023

[FEATURE] Provide way of defining methods for score normalization and combination in scope of Hybrid search opensearch-project/neural-search#228

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding the SearchPhaseResultsProcessor interface in Search Pipeline #7283

Adding the SearchPhaseResultsProcessor interface in Search Pipeline #7283

navneet1v commented Apr 24, 2023 •

edited

Loading

github-actions bot commented Apr 24, 2023

github-actions bot commented Apr 24, 2023

github-actions bot commented Jun 28, 2023

reta commented Jun 28, 2023

andrross commented Jun 28, 2023

github-actions bot commented Jun 28, 2023

msfroh commented Jun 28, 2023

navneet1v commented Jun 28, 2023

msfroh commented Jun 28, 2023

github-actions bot commented Jun 28, 2023

github-actions bot commented Jun 29, 2023

github-actions bot commented Jun 29, 2023

opensearch-trigger-bot bot commented Jun 29, 2023

reta commented Jun 29, 2023

navneet1v commented Jun 29, 2023

Adding the SearchPhaseResultsProcessor interface in Search Pipeline #7283

Adding the SearchPhaseResultsProcessor interface in Search Pipeline #7283

Conversation

navneet1v commented Apr 24, 2023 • edited Loading

Description

Issues Resolved

Check List

github-actions bot commented Apr 24, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Apr 24, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 28, 2023

Gradle Check (Jenkins) Run Completed with:

reta commented Jun 28, 2023

andrross commented Jun 28, 2023

github-actions bot commented Jun 28, 2023

Gradle Check (Jenkins) Run Completed with:

msfroh commented Jun 28, 2023

navneet1v commented Jun 28, 2023

msfroh commented Jun 28, 2023

github-actions bot commented Jun 28, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 29, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 29, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Jun 29, 2023

reta commented Jun 29, 2023

navneet1v commented Jun 29, 2023

navneet1v commented Apr 24, 2023 •

edited

Loading