feat: Implement apply_filter_policy and FilterPolicy.MERGE for the new filters

[vblagoje]

Why:

Implements proper merging of new filters.

• fixes: apply_filter_policy and FilterPolicy.MERGE don't work properly for the new filter syntax #7995

What:

• Implemented new utility functions to determine filter types (is_legacy, is_comparison_filter, is_logical_filter).
• Extended the apply_filter: policy function to support merging of init_filters with runtime_filters using a logical operator, facilitating complex filter scenarios.
• Added a comprehensive suite of tests to validate the functionality of applying filter policies including scenarios with no filters, comparison filters, logical operators, and user-defined logical operators.

How can it be used:

The enhancements enable complex filter logic to be applied seamlessly in document query operations, such as:

• Merging initial and runtime filters with support for legacy filter formats.
• Applying logical operators (AND, OR) when merging comparison or logical filters, allowing for a more nuanced filter logic that can accurately reflect user needs or query specifics.

Example usage of merging comparison and logical filters:

init_filter = {"field": "meta.type", "operator": "==", "value": "pdf"}
runtime_filter = {
    "operator": "AND",
    "conditions": [
        {"field": "meta.name", "operator": "==", "value": "John"},
        {"field": "meta.year", "operator": "==", "value": "2022"},
    ],
}
# Merging the above would result in runtime_filter including the init_filter as another condition under the same logical operator "AND".

How did you test it:

A series of unit tests were added covering various scenarios including:

• Merging filters under both MERGE and REPLACE policies with different combinations of comparison and logical filters.
• Ensuring that the correct logical operator is applied during the merge process.
• Testing the behavior with no filters provided, ensuring backward compatibility and robust error handling.

Notes for the reviewer:

• Special attention should be given to the logic involving merging different types of filters (comparison vs. logical) and ensuring the intended behavior aligns with real-world use cases.
• Review the test cases for apply_filter_policy to ensure all potential scenarios are covered and the expected behavior is clearly documented and verified.

[coveralls]

Pull Request Test Coverage Report for Build 10305030806

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 14 unchanged lines in 3 files lost coverage.
• Overall coverage decreased (-0.03%) to 90.11%

---

Files with Coverage Reduction	New Missed Lines	%
components/generators/utils.py	1	66.67%
components/preprocessors/document_cleaner.py	1	98.08%
document_stores/types/filter_policy.py	12	81.54%

Totals
Change from base Build 10266828865:	-0.03%
Covered Lines:	6952
Relevant Lines:	7715

---

💛 - Coveralls

[shadeMe]

There are also some logic that doesn't have enough test coverage (c.f coveralls report). Let's fix that as well.

[vblagoje]

This should be it now, see commits since c8a53f2 onward

[vblagoje]

Offline we agreed to eviscerate LogicalOperator enum and use Literal strings for operators. The change is in a1cc6f9