Ex2 Part 2 msmarco_qrels.txt

[ThomasK1086]

Our group is struggling with implementing the evaluation of validation and test set, namely the files msmarco_tuples.validation.tsv and msmarco_tuples.test.tsv. It says in the task description to use the msmarco_qrels.txt file to evaluate. However, this file seems to contain all 1s as relevance levels (i assume the file format is queryId, hardcoded Q-0, documentId, relevance Level like in the previous tasks). Is this a mistake? Shouldt there be different labels like for example in fira-22.baseline-qrels.tsv?

[ThomasK1086]

After thinking about it some more, I think I can answer my own question (in case another group runs into this issue). I think that msmarco_qrels.txt only has binary relevance labels for query-document pairs and that non-relevant pairs (with relevance label 0) were simply dropped. Thus, all labels are 1 in this file, leading to my initial confusion. If a document does not appear for a certain query id, it can be assumed to be non-relevant, if it appears, it is relevant