How to generate sparse vectors (SPLADE model)

[DavidAdamczyk]

I am trying to find an efficient solution in the Anserini and Pyserini repositories for generating collections containing sparse vectors obtained from the SPLADE model. I would like to generate these collections from a large dataset compiled in the form of text files, where each line contains one record. I attempted to extract the source code from the Pyserini repository that could address this; however, vector retrieval on GPU is very slow and inefficient.

I would like to ask, please, how do you generate sparse vectors?

[lintool]

Is your question about generating the sparse vectors, i.e., encoding the corpus, or searching the sparse vectors?

For the latter, one of our reproduction docs would be a good starting point:
https://github.com/castorini/anserini/blob/master/docs/regressions/regressions-msmarco-passage-splade-pp-ed.md

[DavidAdamczyk]

Thank you for the response. I am familiar with this document, and I also know how to index documents and perform queries. Everything with MS Marco works great, but I would like to use my own dataset. The first step is to create sparse vectors. Unfortunately, my dataset is 1000 times larger than MS Marco, so I am looking for something to improve the performance of the process for creating vectors.

[lintool]

Encoding the documents scales linearly, so the only solutions are to procure more GPUs or better GPUs...