FAISS indexing on SQL database

[mondoduk]

I have a large private SQL database (PostgreSQL) with text entries in it. I want to create a best-practices search engine to query these entries. I have been looking into the haystack FAISS document store (https://github.com/deepset-ai/haystack/blob/main/haystack/document_stores/faiss.py).

I am confused on how to put FAISS indexes into my private SQL database, using a FAISSDocumentStore. I provide my SQL database url in FAISSDocumentStore. But I don't know what do do next.

How do I select a table name and column to use with FAISSDocumentStore?

I only want to store FAISS indexes (not data - this would be redundant).

[mayankjobanputra]

Hi @mondoduk,

Thanks for your question. I am trying to understand your requirement and I want to confirm my understanding with you. I am assuming you already looked at FAISSDocumentStore documentation

If I understand you correctly then,

1. Your PostgreSQL data has some kind of id for each text.
2. You want to make your PostgreSQL data semantically searchable using FAISSDocumentStore.
3. To query, you would use query_by_embedding method from FAISSDocumentStore.

Am I understanding this correctly?

Although I don't think you can use your private SQL dataset with FAISSDocumentStore just to create a FAISS Index yet, I think I have a workaround for you :) I will talk to the team, confirm it, and then post it here. I will also mention this use-case and will try to see if we can prioritize supporting this for coming Haystack releases.

A question:

• How large is your dataset? 1M rows or above? This will help me recommend the right FAISS Index for your use case.

[jozgu]

Hi @mayankjobanputra,

I have a similar issue as @mondoduk. However I'm working with a MariaDB database (i'm still researching if this would be able to work with such database) that I would like to use with FAISS. The database have certain access authorization so that different companies can't view each others content (with is also why I'm very keen on keeping it).

Therefore I was wondering if you have an update on the workaround you mention?

[anakin87]

DocumentStore

As you can read in the docs, you can think of the DocumentStore as a database that stores your texts and meta data and provides them to the Retriever at query time.

FAISSDocumentStore

FAISSDocumentStore is internally composed of:

• a SQL database to store texts and metadata
• a FAISS index to store vectors

I think this internal structure of FAISSDocumentStore is an implementation detail and you probably shouldn't care much about it...

Integration with your data

You can't directly use your private SQL database in the FAISSDocumentStore.

Instead, I would consider the following steps (similar to this tutorial):

• initialize a new FAISSDocumentStore
• loop over your source database and write texts inFAISSDocumentStore, calling document_store.write_documents method
• initialize a retriever, such as EmbeddingRetriever (also consisting of an encoder to represent documents as vectors)
• compute and store vectors for your texts, by calling document_store.update_embeddings(retriever)
• save your FAISSDocumentStore
• repeat whenever your data changes...

[MarcSchluperAtIntel]

Do I understand it ("repeat whenever your data changes...") correctly that if just one document gets obsolete and needs to be deleted, I have to reload everything? What exactly needs to be repeated?

Also, the original question was about Postgres. Is there any tutorial about how to use postgres instead of sqlite?