Add verbose pipeline parameter to output each processor's execution details #16843

junweid62 · 2024-12-13T00:00:40Z

Description

Related RFC : #16705

This PR introduces enhancements to OpenSearch's search pipeline functionality, focusing on improving the traceability and debugging of search request and response transformations. It addresses the increasing complexity of search pipeline processors by implementing verbose mode support, which provides detailed insights into processor execution.

Adds Verbose Mode for Search Pipelines:
- Introduced the verbose_pipeline parameter to search requests, default to false.
- Tracks each processor’s input, output, execution time, and status (success/failure).
- Provides detailed logs of the data flow through request and response processors.
Improves Pipeline Debugging:
- Captures step-by-step data transformations applied by each processor.
- Includes execution metadata (e.g., timestamps and elapsed time) in search responses for better analysis.
Supports All Pipeline Configurations:
- Works seamlessly with:
  - Default pipelines configured at the index level.
  - Pipelines explicitly specified in the search request.
  - Ad-hoc pipelines defined inline.
Test Framework Enhancements:
- Added tests for verbose mode to ensure correct functionality and compatibility.

Example output with request processor: filter_query response processor: rename_field and sort

{
    "took": 29,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 0.9116078,
        "hits": [
            {
                "_index": "my_index",
                "_id": "2",
                "_score": 0.9116078,
                "_source": {
                    "notification": "This is a public message 2",
                    "visibility": "public"
                }
            }
        ]
    },
    "processor_result": [
        {
            "processor_name": "filter_query",
            "duration_millis": 0,
            "input_data": {
                "query": {
                    "bool": {
                        "must": [
                            {
                                "match": {
                                    "message": {
                                        "query": "This is a public message 1",
                                        "operator": "OR",
                                        "prefix_length": 0,
                                        "max_expansions": 50,
                                        "fuzzy_transpositions": true,
                                        "lenient": false,
                                        "zero_terms_query": "NONE",
                                        "auto_generate_synonyms_phrase_query": true,
                                        "boost": 1.0
                                    }
                                }
                            }
                        ],
                        "adjust_pure_negative": true,
                        "boost": 1.0
                    }
                },
                "search_pipeline": "my_pipeline"
            },
            "output_data": {
                "query": {
                    "bool": {
                        "must": [
                            {
                                "bool": {
                                    "must": [
                                        {
                                            "match": {
                                                "message": {
                                                    "query": "This is a public message 1",
                                                    "operator": "OR",
                                                    "prefix_length": 0,
                                                    "max_expansions": 50,
                                                    "fuzzy_transpositions": true,
                                                    "lenient": false,
                                                    "zero_terms_query": "NONE",
                                                    "auto_generate_synonyms_phrase_query": true,
                                                    "boost": 1.0
                                                }
                                            }
                                        }
                                    ],
                                    "adjust_pure_negative": true,
                                    "boost": 1.0
                                }
                            }
                        ],
                        "filter": [
                            {
                                "term": {
                                    "visibility": {
                                        "value": "public",
                                        "boost": 1.0
                                    }
                                }
                            }
                        ],
                        "adjust_pure_negative": true,
                        "boost": 1.0
                    }
                },
                "search_pipeline": "my_pipeline"
            }
        },
        {
            "processor_name": "rename_field",
            "duration_millis": 0,
            "input_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "message": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ],
            "output_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ]
        },
        {
            "processor_name": "sort",
            "duration_millis": 0,
            "input_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ],
            "output_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ]
        }
    ]
}

Related Issues

Resolves #14745

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…etails Signed-off-by: Junwei Dai <junweid@amazon.com>

Signed-off-by: Junwei Dai <junweid@amazon.com>

junweid62 · 2024-12-13T00:49:43Z

@joshpalis Hey Josh, can you take a look when you have time? :)

Signed-off-by: Junwei Dai <junweid@amazon.com>

github-actions · 2024-12-13T01:41:41Z

❌ Gradle check result for 488377f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-12-13T04:10:48Z

❌ Gradle check result for 719fa1c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Junwei Dai <junweid@amazon.com>

joshpalis

Great work @junweid62 on implementing this feature. A few comments from my initial pass

joshpalis · 2024-12-13T17:24:10Z

server/src/main/java/org/opensearch/search/SearchHits.java

+     * @return a deep copy of the current SearchHits object
+     * @throws IOException if an I/O exception occurs during serialization or deserialization
+     */
+    public SearchHits deepCopy() throws IOException {


Im curious why this serialization/deserialization is needed

joshpalis · 2024-12-13T17:31:52Z

server/src/main/java/org/opensearch/search/internal/InternalSearchResponse.java

+
+    private static void writeProcessorResultOnOrAfter(StreamOutput out, List<ProcessorExecutionDetail> processorResult) throws IOException {
+        if (out.getVersion().onOrAfter(Version.V_2_18_0)) {
+            out.writeCollection(processorResult, (o, detail) -> detail.writeTo(o));


nit : we could use out.writeList() here

joshpalis · 2024-12-13T17:41:09Z

server/src/main/java/org/opensearch/search/pipeline/Pipeline.java

@@ -225,11 +241,19 @@ ActionListener<SearchResponse> transformResponseListener(
            final SearchResponseProcessor processor = searchResponseProcessors.get(i);

            responseListener = ActionListener.wrap(r -> {
+                ProcessorExecutionDetail detail = new ProcessorExecutionDetail(processor.getType());
+                detail.addInput(Arrays.asList(r.getHits().deepCopy().getHits()));


Trying to understand, what is the rationale for invoking deepCopy()? r.getHits() already returns an object of type SearchHit, this can just be written as detail.addInput(Arrays.asList(r.getHits().getHits()));

I initially thought the same:). However, when the Object receives this list, it holds a reference to the original SearchHit objects. Without a deepCopy, any modifications to the output will also reflect in the input, as they share the same reference.

By using deepCopy, we ensure that the input and output hold independent values, allowing us to clearly see the differences between the two at each step of the processor.

joshpalis · 2024-12-13T17:41:17Z

server/src/main/java/org/opensearch/search/pipeline/Pipeline.java

                beforeResponseProcessor(processor);
                final long start = relativeTimeSupplier.getAsLong();
                processor.processResponseAsync(request, r, requestContext, ActionListener.wrap(rr -> {
                    long took = TimeUnit.NANOSECONDS.toMillis(relativeTimeSupplier.getAsLong() - start);
                    afterResponseProcessor(processor, took);
+                    detail.addOutput(Arrays.asList(rr.getHits().deepCopy().getHits()));


github-actions · 2024-12-13T18:38:27Z

❌ Gradle check result for e4e30f5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions bot added enhancement Enhancement or improvement to existing feature or request Other Priority-High Search Search query, autocomplete ...etc v2.19.0 Issues and PRs related to version 2.19.0 labels Dec 13, 2024

junweid62 added the backport 2.x Backport to 2.x branch label Dec 13, 2024

Add verbose pipeline parameter to output each processor's execution d…

1c3b946

…etails Signed-off-by: Junwei Dai <junweid@amazon.com>

add change log

d931750

Signed-off-by: Junwei Dai <junweid@amazon.com>

junweid62 force-pushed the search-pipline-execution branch from d5a2c4c to d931750 Compare December 13, 2024 00:26

Add merge missing change log

1f879c1

Signed-off-by: Junwei Dai <junweid@amazon.com>

Add missing line

488377f

Signed-off-by: Junwei Dai <junweid@amazon.com>

opensearch-ci-bot mentioned this pull request Dec 13, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndexingIT #14302

Open

Refactor ProcessorExecutionDetail to improve field handling

e4e30f5

Signed-off-by: Junwei Dai <junweid@amazon.com>

junweid62 force-pushed the search-pipline-execution branch from 719fa1c to e4e30f5 Compare December 13, 2024 17:41

joshpalis reviewed Dec 13, 2024

View reviewed changes

opensearch-ci-bot mentioned this pull request Dec 14, 2024

[AUTOCUT] Gradle Check Flaky Test Report for SpecificClusterManagerNodesIT #15944

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add verbose pipeline parameter to output each processor's execution details #16843

Add verbose pipeline parameter to output each processor's execution details #16843

junweid62 commented Dec 13, 2024 •

edited

Loading

junweid62 commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

joshpalis left a comment

joshpalis Dec 13, 2024

joshpalis Dec 13, 2024

joshpalis Dec 13, 2024 •

edited

Loading

junweid62 Dec 13, 2024

joshpalis Dec 13, 2024

github-actions bot commented Dec 13, 2024

Add verbose pipeline parameter to output each processor's execution details #16843

Are you sure you want to change the base?

Add verbose pipeline parameter to output each processor's execution details #16843

Conversation

junweid62 commented Dec 13, 2024 • edited Loading

Description

Related Issues

Check List

junweid62 commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

joshpalis left a comment

Choose a reason for hiding this comment

joshpalis Dec 13, 2024

Choose a reason for hiding this comment

joshpalis Dec 13, 2024

Choose a reason for hiding this comment

joshpalis Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

junweid62 Dec 13, 2024

Choose a reason for hiding this comment

joshpalis Dec 13, 2024

Choose a reason for hiding this comment

github-actions bot commented Dec 13, 2024

junweid62 commented Dec 13, 2024 •

edited

Loading

joshpalis Dec 13, 2024 •

edited

Loading