Pinecone document store

[albertovilla]

Is your feature request related to a problem? Please describe.
I think it would be great to have a document store which makes use of Pinecone in a similar way to the Milvus document store already implemented.

Describe the solution you'd like
A seamless Pinecone document store which integrates with deepset, same functionalities than MilvusDocumentStore

Describe alternatives you've considered
An adhoc / separate implementation without deepset integration would be possible but a tighter integration would make things easier.

Additional context
N/A

[bogdankostic]

Hi @albertovilla, thanks for raising this feature request! As there exist dozens of different databases and we cannot support all of them, we would need to assess the advantages of a new database compared to the already supported ones. What would be, in your opinion, the advantages of Pinecone compared to Milvus?

[albertovilla]

Hi @bogdankostic, Pinecone is a fully managed service which simplifies infrastructure setup and maintenance even as you scale up. Pinecone also supports metadata filtering for better control over results, and hybrid storage.

I understand this may not be a priority for you. I was using Pinecone already for a use case that's why I have identified the need 😄.

Depending on how the use case progresses I may end up implementing this myself for better integration with haystack. I haven't yet got the results I want with haystack (I'm still learning to use it) so it will depend on how it goes 😄

[bogdankostic]

We will assess this internally and post our decision in this issue. If we decide to support Pinecone in Haystack, this will probably take some time, just so you know.

BTW, we are more than happy to help you getting familiar with Haystack and answering your questions in our community slack. :)

[gkogan]

@bogdankostic Hi, I'm from the Pinecone team. We'll be happy to help: greg@pinecone.io

[lalitpagaria]

@albertovilla It is very easy to integrate pinecone with Haystack, you may try on your own fork and see if this works then possibly raise PR if Deepset team agrees on it.

As I see Pinecone provides APIs to create, list and delete indexes as well as add, remove, and query records. Hence it is easier to integrate as done in Milvus or Weaviate case.

Pinecode API ref: https://www.pinecone.io/docs/api/operation/list_indexes
Haystack base class to integrate: https://github.com/deepset-ai/haystack/blob/master/haystack/document_stores/base.py

[bogdankostic]

PineconeDocumentStore was added in #2254, so we can close this issue.