[FEAT]: support faiss to enable efficient search

[lwaekfjlk]

Description

we should use faiss to accelerate

Additional Information

No response

[shreypandey]

Hi @lwaekfjlk,
Is FAISS integration required for accelerated retrieval of agent and paper retrieval?
If so, perhaps it could be achieved with an embedding database like Redis and creating appropriate ANN indexes on embeddings in that, as suggested in as mentioned in #430 .

Additionally, FAISS requires re-indexing whenever new embeddings are added, which could be managed asynchronously and more efficiently with an embedding database.

[lwaekfjlk]

yes, @shreypandey, good point! thanks for mentioning that, you can check #431 that we include redis support in our roadmap of the 0.0.2 version. Looking forward to futher discussion.

[shreypandey]

Hi @lwaekfjlk, I have proposed an implementation of redis integration on #428, Please review the proposed implementation.

[lwaekfjlk]

sure.

[lwaekfjlk]

this should be squashed together into #428. therefore, close.