[BUG]match_all query filters applied despite courier:ignoreFilterIfFieldNotInIndex setting enabled

[ananzh]

Describe the bug
The match_all query filter is applied and included in the OpenSearch request when courier:ignoreFilterIfFieldNotInIndex is enabled. This setting is used to skip filters that apply to fields that don’t exist in the index for a visualization. However, we see filter applied even if the field the filter applies to does not exist in the index pattern of the visualization, which contradicts the expected behavior as per the setting's definition.

bug-rep.mov

To Reproduce

• Navigate to Stack Management > Advanced Settings > courier:ignoreFilterIfFieldNotInIndex.
• Enable the setting courier:ignoreFilterIfFieldNotInIndex and save the changes.
• Create a visualization of type DataTable and choose an index pattern.
• In Buckets, split rows, choose aggregation as "Terms", select a field of type string, and click Update.
• Add a normal filter (match_all query) that targets a non-existent field in the chosen index pattern.
• Click Inspect -> View: Requests -> Click request: Here the custom query DSL that we created is still sent in the request.

Expected behavior
with courier:ignoreFilterIfFieldNotInIndex Enabled
Filters that target fields not present in the index pattern should be skipped, and not included in the request. Both prefix query and match_all query should not send the request for non-existent fields.

OpenSearch Version
any

Dashboards Version
any

Plugins

na

[ananzh]

#5581

[anand3493]

@ananzh I believe you misunderstood my issue raised in the bug #5581

Re-explaining:
courier:ignoreFilterIfFieldNotInIndex is enabled now

Irrespective of the type of filter (either custom query DSL or normal filter), the filter should be applied if the field is present in the chosen index.

The filter should not be applied only when the field is not present on the index pattern

My issue is I have enabled the ignoreFilterIfFieldNotInIndex option and I create a visualization for a index pattern and I create a custom query DSL filter for a field which is present on the chosen index.
Expected Outcome: Fitler should be applied.
Current misbehavior: Filter is not sent in the request.

In the bug-rep video, first scenario where you try the prefix query dsl filter - that is the actual issue. I expect the filter should be applied there. However it is not sent in the request at all. You considered multiple indices where only one index has that prefix field.
Just think that you have enabled the ignoreFilterIfFieldNotInIndex option.
let's say we have index patterns: logstash*, metrics*, events*.
Now I create a visualization on top of logstash* index pattern, which has documents with field "user-agent".
So now when I try to create my custom query DSL filter with some prefix value on top logstash*, I expect the filter to be applied. However it is not sent.