[META] Search Pipelines - GA

[macohen]

This issue tracks all work that must be included for us to make Search Pipelines generally available.

Search Pipelines will enable builders to configure an ordered list of processors that take queries, results, or both to enable transformations that typically occur in client applications either before queries are submitted to OpenSearch or after results are retrieved. This feature will make it easy to integrate and test result re-rankers, query re-writers, and other features to send data to other applications. We also want to enable features like running multiple pipelines per index, comparing results between pipelines, and setting a default pipeline for an index.

This will enable partners to more easily integrate their features with OpenSearch and empower customers to focus more energy on building their applications and less time integrating/building custom processors.

Describe alternatives you've considered
See opensearch-project/search-processor#80 for the RFC for the project. We want your feedback!

• [RFC] Search pipelines search-processor#80
• Prototype search pipelines search-processor#97
• Bug fixes and refactoring search-processor#38
• [RFC] Additional Reranker Implementations Requested search-processor#36
• Refactor metrics and logging search-processor#44
• [DOC] Search Processor (new feature) documentation-website#2560
• Search Pipelines - Process Requests, Responses, or Both search-processor#115
• Pipeline specified in a search request (ad hoc execution). #6717
• Index default pipelines #6719
• Search pipeline metrics (at least on par w/ ingestion pipeline metrics). #6723

[msfroh]

Commenting here, since I can't edit the issue to enumerate the tasks that need to be completed in time for a 2.7 release:

1. Prototype / initial implementation: Prototype search pipelines search-processor#97 / Initial search pipelines implementation #6587
2. Pipeline specified in a search request (ad hoc execution).
3. Index default pipelines
4. Processors that modify both request and response ("bracket processor").
5. Search pipeline metrics (at least on par w/ ingestion pipeline metrics).
6. Processor toolbox
   1. Filter query request processor (done as part of prototype)
   2. Rename field response processor
   3. Scripting (painless/mustache) request processor
   4. Scripting (painless/mustache) response processor
   5. Conditional request processor
   6. Conditional response processor
   7. TODO: Other ideas that come up

I'll create tasks for each of these, which we can link from the issue description.

[macohen]

Thinking that the META issue could be a long lived list of processors to create. Each issue linked to that could be assigned to a release as we prioritize. Feel free to reassign tasks.

[noCharger]

Some other feastures

• Add search pipeline support in OpenSearch JS client (explict parameter)
• Select Search Pipeline in Search Comparison Tool dashboards-search-relevance#182 (depend on 1)
• Search pipeline API support in search apps playground

[msfroh]

This issue tracks task that need to be completed to before search pipelines are generally available, like metrics, documentation, integration w/ OpenSearch Dashboards.

[macohen]

I'm removing a couple of line items
#6722
#6907
from the original issue list as they are not required for GA for 2.9.

@noCharger do you think that the three features here #6278 (comment) are critical for GA?

[noCharger]

I'm removing a couple of line items #6722 #6907 from the original issue list as they are not required for GA for 2.9.

@noCharger do you think that the three features here #6278 (comment) are critical for GA?

Nope, 3 is done. 1 and 2 are supportive features.

[macohen]

Search Pipelines is GA in 2.9