Ex2: Part 1 query-document pairs for dbpedia

[waffelbaellchen]

Hello,
I noticed that some query-document pairs of dbpedia share almost the same name as if they were related. For example, there is a query-id db_q_<dbpedia:1564_in_Norway> ("events in the year 1564 in norway uk") and a document-id db_<dbpedia:1564_in_Norway> ("1564 in norway events in the year 1564 in norway.").

As I am unaware of how these pairs and IDs were generated, I wanted to ask whether it is fair to say that query-document pairs of dbpedia that share (almost) the same name should at least be topic-related. In the above example, the document text is a bit strange but seems to be related to the query.

[sophiaalthammer]

Hello,

it is true that the IDs of the query and document are related, this is because we generated with a generative language model queries for each document. The ID of the generated queries is then the document ID from which it was generated and the "q_" in front of it.

For the evaluation I would ask you, not to take this relation into account, but only consider the data from the annotation exercise: the relevance judgements, the time or the annotator.

Best,
Sophia