Feature/Add support for state-based metadata filter to Retriever Tool

[serhiy-matoffo]

Extended Retriever Tool to support metadata filter based on the object stored in agent state.

[HenryHengZJ]

Interesting! do you have an example use case?

[serhiy-matoffo]

Sure. We've found that our RAG flow often returns irrelevant documents. To be a bit more precise, result documents contain some content related to some words used by the user but are found in documents on a completely different subject. So, we've tasked one of our LLM Nodes to figure out the subject of the conversation and save that to state (at this point we can form a metadata filter object). Later down the line, we would like to take that subject and use it to narrow down the query to the vector store.

Is there a way to implement this flow using existing functionality? I couldn't find one and had to add this property to a Retriever Tool.

[HenryHengZJ]

I was thinking of how to make it more generic..

How bout this?

We can have an additional metadata filter, this adds to the existing filter from vector store if there is any. And users can filter using the existing flow variable such as $flow.state, $flow.chatId etc

[serhiy-matoffo]

This would not allow complex/nested filters, right?

[HenryHengZJ]

This would not allow complex/nested filters, right?

Do you have an example of that? Under the hood it's just a simple JSON that adds on top of existing filter

[serhiy-matoffo]

Do you have an example of that?

Here is an example from pinecone documentation: {"$and": [{"genre": "comedy"}, {"genre":"documentary"}]}

Would the functionality you suggested support this kind of syntax?

[HenryHengZJ]

yep definitely. If thats okay I can make the changes on this PR, you can then try it out

[serhiy-matoffo]

Sounds good!

[HenryHengZJ]

Updated, try and test see if it works @serhiy-matoffo

[serhiy-matoffo]

I can confirm that it works great, thanks!

Also, this trick allows using any dynamic object from the flow state:

I just prefer to form that object using a script instead of typing in all the properties through those boxes 😄

[HenryHengZJ]

merging

[sirsimonson]

so excited to try this out. As of now I'm trying to use metadata filter like so for postgres:

Somehow this does not work and returns an emptry list. is this a known issue or am I just forming it wrong? I'm looking for docs about this filter spec but can't find any. @HenryHengZJ @serhiy-matoffo

EDIT:

According to https://js.langchain.com/docs/integrations/vectorstores/pgvector/ filtering in pgvector is only supported via "in" or "arrayContains" operator in nodejs. It would be so cool to have exhaustive filter functionality as in the python langchain version of pgvector as can be found here: https://python.langchain.com/docs/integrations/vectorstores/pgvector/

We still plan to adopt this feature and dynamically filter from agent state. Unfortunately the most power lies in integrating it with pinecone.

[sirsimonson]

@HenryHengZJ Please advise .. Also the above filter mechanism (which had worked through "Test Retrieval Query" doesnt work for this new feature. Somehow the filter is not passed down to the postgres query.