Methodology of the Annotaion Tasks

[ConBoe]

Hi,
I noticed that I got similar queries and texts (and formats of texts) in a row. So I wondered if that is on purpose or just by chance.
Just out of curiosity: would that not have an effect on the annotations and maybe introduce a bias into the data? Is there a reason behind that choice (if it is actually not randomized)?
If you do not want to answer this I understand.

Kind Regards,
CB

[sebastian-hofstaetter]

Hi,

Interesting, thanks for pointing this out. This was by chance (or a bug in the selection process) and not on purpose. We have a rather complicated priority based system that makes sure that we get multiple judgements on groups of pairs, before going to the next group (so that we don't run out of potential pairs and at the same time are guaranteed more than 1 judgement per pair) and we assume that ids are randomly distributed, however it might have been that one of the sources was not as random as we thought ...

It might introduce a bias, yes, but usually different annotators annotate different pairings, so it's unlikely that more than one annotator gets the same order to annotate (which should help reduce the bias, after aggregating the judgements).

Best,
Sebastian