feat: Add filter_policy init parameter to in memory retrievers

[vblagoje]

A companion PR for deepset-ai/haystack-core-integrations#781 adding filter_policy parameter with options replace and merge for in memory retrievers.

Do not merge before we understand how to proceed with deepset-ai/haystack-core-integrations#781 and this PR in sync.

[coveralls]

Pull Request Test Coverage Report for Build 9364808613

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 10 unchanged lines in 3 files lost coverage.
• Overall coverage decreased (-0.03%) to 89.724%

---

Files with Coverage Reduction	New Missed Lines	%
components/retrievers/in_memory/embedding_retriever.py	2	93.48%
components/retrievers/in_memory/bm25_retriever.py	3	93.02%
document_stores/in_memory/document_store.py	5	98.23%

Totals
Change from base Build 9351549258:	-0.03%
Covered Lines:	6723
Relevant Lines:	7493

---

💛 - Coveralls

[vblagoje]

@Amnah199 have a look but I'll also ask @silvanocerza as well - he has more background on this task

[silvanocerza]

Would be great if we go the direction we're going with other Components and use an Enum instead of a Literal. This would be a good example.

haystack/haystack/utils/hf.py

Lines 62 to 92 in 74df8ed

	class HFEmbeddingAPIType(Enum):
	"""
	API type to use for Hugging Face API Embedders.
	"""
	# HF [Text Embeddings Inference (TEI)](https://github.com/huggingface/text-embeddings-inference).
	TEXT_EMBEDDINGS_INFERENCE = "text_embeddings_inference"
	# HF [Inference Endpoints](https://huggingface.co/inference-endpoints).
	INFERENCE_ENDPOINTS = "inference_endpoints"
	# HF [Serverless Inference API](https://huggingface.co/inference-api).
	SERVERLESS_INFERENCE_API = "serverless_inference_api"
	def __str__(self):
	return self.value
	@staticmethod
	def from_str(string: str) -> "HFEmbeddingAPIType":
	"""
	Convert a string to a HFEmbeddingAPIType enum.
	:param string:
	:return: The corresponding HFEmbeddingAPIType enum.
	"""
	enum_map = {e.value: e for e in HFEmbeddingAPIType}
	mode = enum_map.get(string)
	if mode is None:
	msg = f"Unknown Hugging Face API type '{string}'. Supported types are: {list(enum_map.keys())}"
	raise ValueError(msg)
	return mode

And is used here in the HuggingFaceAPIDocumentEmbedder.

haystack/haystack/components/embedders/hugging_face_api_document_embedder.py

Line 89 in 74df8ed

api_type: Union[HFEmbeddingAPIType, str],

We could define that in the haystack.document_store.types package, we already define the DuplicatePolicy in there.

The only downside is that we'd have to wait to update the integrations after this. If it's urgent we could support for Literals in the integration really quick, create a PR that actually uses the new Enum and merge it only after the next Haystack release.

[vblagoje]

Would be great if we go the direction we're going with other Components and use an Enum instead of a Literal. This would be a good example.

haystack/haystack/utils/hf.py

Lines 62 to 92 in 74df8ed

	class HFEmbeddingAPIType(Enum):
	"""
	API type to use for Hugging Face API Embedders.
	"""
	# HF [Text Embeddings Inference (TEI)](https://github.com/huggingface/text-embeddings-inference).
	TEXT_EMBEDDINGS_INFERENCE = "text_embeddings_inference"
	# HF [Inference Endpoints](https://huggingface.co/inference-endpoints).
	INFERENCE_ENDPOINTS = "inference_endpoints"
	# HF [Serverless Inference API](https://huggingface.co/inference-api).
	SERVERLESS_INFERENCE_API = "serverless_inference_api"
	def __str__(self):
	return self.value
	@staticmethod
	def from_str(string: str) -> "HFEmbeddingAPIType":
	"""
	Convert a string to a HFEmbeddingAPIType enum.
	:param string:
	:return: The corresponding HFEmbeddingAPIType enum.
	"""
	enum_map = {e.value: e for e in HFEmbeddingAPIType}
	mode = enum_map.get(string)
	if mode is None:
	msg = f"Unknown Hugging Face API type '{string}'. Supported types are: {list(enum_map.keys())}"
	raise ValueError(msg)
	return mode

And is used here in the HuggingFaceAPIDocumentEmbedder.

haystack/haystack/components/embedders/hugging_face_api_document_embedder.py

Line 89 in 74df8ed

api_type: Union[HFEmbeddingAPIType, str],

We could define that in the haystack.document_store.types package, we already define the DuplicatePolicy in there.

The only downside is that we'd have to wait to update the integrations after this. If it's urgent we could support for Literals in the integration really quick, create a PR that actually uses the new Enum and merge it only after the next Haystack release.

Sure let me do that 🙏

[vblagoje]

@silvanocerza why not add apply_filters to this Enum then?
Something like:

    def apply_filters(self, init_filters: Optional[Dict[str, Any]] = None,
                      runtime_filters: Optional[Dict[str, Any]] = None
                      ) -> Optional[Dict[str, Any]]:
        """
        Apply the filter policy to the given init and runtime filters.

        :param init_filters: The initial filters.
        :param runtime_filters: The runtime filters.
        :return: The resulting filters based on the policy.
        """
        if self == FilterPolicy.MERGE and runtime_filters:
            return {**(init_filters or {}), **runtime_filters}
        else:
            return runtime_filters or init_filters

[silvanocerza]

@vblagoje let's keep it simple and not do that. Might be a bit counter intuitive that way, and it also adds some indirection.

[silvanocerza]

Nice! 👍

[vblagoje]

Before we merge @silvanocerza - how do we proceed with deepset-ai/haystack-core-integrations#781 after this integration? We can't use this FilterPolicy abstraction until next minor/patch release? But that should be soon anyway...right?

[silvanocerza]

If it's not urgent let's update the PR in integration to uses the new Enum and keep it there as draft. When we release the new Haystack version with this PR we can update the integrations PR.

If it's urgent I would suggest changing a bit this PR, so that the retrievers support both str and the Enum and merge it. Then we merge the already existing one in the integrations that adds supports with str. And create a PR right away that updates all retrievers to use the Enum, when new Haystack is released we merge it and release the integrations. This way we don't make any breaking change.

[vblagoje]

No worries, this is not urgent by any means. I'll convert the other PR to draft and update it accordingly.