msmarco-v2 index & BM25: question about concatenated fields

[yixuan-qiao]

Hi,
Carefully read the Anserini: BM25 Baselines for the MS MARCO V2 Collections doc, i found that

each "document" in the index comprises the url, title, headings, and passage fields concatenated together.

By using the prebuilt Indexes msmarco-v2-passage-augmented, i found that the url, title, headings, and passage fields are not concatenated, but in a json format. This index can also be retrieved by BM25, just curious which part is involved in BM25, only the passage fields? or just use all elements internally?

Or maybe the index built by the Anserini: BM25 Baselines for the MS MARCO V2 Collections doc is not the same as prebuilt Indexes msmarco-v2-passage-augmented?

[lintool]

The fields are concatenated programmatically during indexing, see here: https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/collection/MsMarcoV2PassageCollection.java#L131

So this statement is accurate:

each "document" in the index comprises the url, title, headings, and passage fields concatenated together.

And it does correspond to the pre-built here: https://github.com/castorini/pyserini/#two-click-reproductions

[yixuan-qiao]

The instructions in https://github.com/castorini/pyserini/blob/master/pyserini/resources/index-metadata/lucene-index.msmarco-v2-passage-augmented.20220111.06fb4f.README.md, looks the same as Anserini: BM25 Baselines for the MS MARCO V2 Collections all using the MsMarcoV2PassageCollection. I downloaded the index from following,

"msmarco-v2-passage-augmented": {
"description": "Lucene index of the MS MARCO V2 augmented passage corpus.",
"filename": "lucene-index.msmarco-v2-passage-augmented.20220111.06fb4f.tar.gz",
"readme": "lucene-index.msmarco-v2-passage-augmented.20220111.06fb4f.README.md",
"urls": [
"https://rgw.cs.uwaterloo.ca/JIMMYLIN-bucket0/pyserini-indexes/lucene-index.msmarco-v2-passage-augmented.20220111.06fb4f.tar.gz"
],
"md5": "975f6be8d49238fe1d47e2895d26f99e",
"size compressed (bytes)": 65574361728,
"total_terms": 15272964956,
"documents": 138364198,
"unique_terms": 16579071,
"downloaded": False
}

When i check it by json.loads(searcher.doc(0).raw()) after LuceneSearcher, the results are still in json, all fileds are not concatenated together. Maybe the index i downloaded is not the index built by the docs?

[lintool]

The representation of the document is not what's indexed. We preserve the structure of the document so you can keep track of the fields separately, but the fields are concatenated during indexing time.

[yixuan-qiao]

Got it, much thanks!

Is there any way to show the specific index format by using some intrinsic attribute except digging into MsMarcoV2PassageCollection, just like .raw() to see the original representation of the document?

[lintool]

Sure, there are Lucene APIs for inspecting the index. From you Java end, you can directly access the APIs. What exactly do you want?

On the Python/Pyserini end, we'd need expose the hooks.

If you have a specific feature request, please file an issue directly and we'll prioritize.