Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Add dynamic variable substitution for Normalization processor parameters #1005

Open
martin-gaievski opened this issue Dec 6, 2024 · 1 comment

Comments

@martin-gaievski
Copy link
Member

martin-gaievski commented Dec 6, 2024

Is your feature request related to a problem?

The Normalization processor currently only accepts static parameters during pipeline creation. This limitation becomes problematic when parameter values need to be determined dynamically during search execution.

Proposed solution is to enable dynamic parameter substitution in the Normalization processor by:

  • allowing placeholder variables in pipeline definitions
  • supporting runtime value substitution during query execution

For example pipeline definition with normalization processor can look like this:

{
    "description": "Post processor for hybrid search",
    "phase_results_processors": [
        {
            "normalization-processor": {
                "normalization": {
                    "technique": "min_max"
                },
                "combination": {
                    "technique": "arithmetic_mean",
                    "parameters": {
                        "weights": [
                             ${keyword_weight},
                             ${neural_weight}
                         ]
                    }
                }
            }
        }
    ]
}

when one of the processors (e.g. ML Inference processor) set values in the pipeline context, the Normalization processor would substitute these variables with actual values.

What alternatives have you considered?

External parameter management: While possible to handle parameter values outside OpenSearch, this approach adds maintenance overhead and complexity to the system.

Do you have any additional context?

This is one of the building blocks for hybrid search optimizer that has been requested in opensearch-project/OpenSearch#16775

@wrigleyDan
Copy link

Great, that's very useful to overcome the "staticness" of hybrid search and open the door to dynamic query-based settings that are determined outside of the normalization process 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants