Add WebFAQ bitext mining tasks

[michaeldinzinger]

Hi all,
this Pull Request follows up on a previous one regarding the WebFAQ Retrieval Dataset. This PR aims to add two Bitext Mining tasks. The source data remains the same (Question-Answer pairs extracted from FAQ pages), as described in the old PR and in the preprint of the dataset paper.

Overall, the Bitext Mining dataset consists of 1001 language pairs, with each at least 100 aligned QAs. If these are too many, I could also lower the number of language pairs to: 368 (all language pairs with at least 1000 samples) or 65 (all language pairs with at least 4000 samples)

The two Bitext Mining tasks are WebFAQBitextMiningQuestions and WebFAQBitextMiningQAs:

• For the first one, the sentences to be matched are only the questions, so usually one sentence. I marked this task as category s2s.
• For the second task, the sentences are concatenations of the questions with their corresponding answers. I marked it as category p2p.

Example

Input data:

Language	Question	Answer
deu	Akzeptiert das Empark Grand Hotel (Anhui) Hefei Kreditkarten?	Nein, das Hotel akzeptiert keine Kreditkarten.
eng	Does Empark Grand Hotel (Anhui) Hefei accept credit card payments?	No, the hotel does not accept credit card payment.

Task WebFAQBitextMiningQuestions:

• Sentence 1: Akzeptiert das Empark Grand Hotel (Anhui) Hefei Kreditkarten?
• Sentence 2: Does Empark Grand Hotel (Anhui) Hefei accept credit card payments?
• Average sentence length: 52 characters

Task WebFAQBitextMiningQAs:

• Sentence 1: Akzeptiert das Empark Grand Hotel (Anhui) Hefei Kreditkarten? Nein, das Hotel akzeptiert keine Kreditkarten.
• Sentence 2: Does Empark Grand Hotel (Anhui) Hefei accept credit card payments? No, the hotel does not accept credit card payment.
• Average sentence length: 361 characters

! Note that in this exemplary case, the question is longer than the answer. But in the majority of cases, it is the other way around: the answer is quite descriptive and therefore longer than the question.

Code Quality

• Code Formatted: Format the code using make lint to maintain consistent style.

Documentation

• Updated Documentation: Add or update documentation to reflect the changes introduced in this PR.

Testing

• New Tests Added: Write tests to cover new functionality. Validate with make test-with-coverage.
• Tests Passed: Run tests locally using make test or make test-with-coverage to ensure no existing functionality is broken.

Adding datasets checklist

Reason for dataset addition:

• Afaik, resource (FAQ) not utilized yet.

• Covers many different language pairs.

• Due to SotA process of bitext mining, the dataset yields high-quality translations (see preprint).

• I have run the following models on the task (adding the results to the pr). These can be run using the mteb -m {model_name} -t {task_name} command.

  • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
  • intfloat/multilingual-e5-small

• I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).

• If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()

• I have filled out the metadata object in the dataset file (find documentation on it here).

• Run tests locally to make sure nothing is broken using make test.

• Run the formatter to format the code using make lint.

[KennethEnevoldsen]

We probably need to take a look at the pairs.

I would be happy if we can lower the threshold (e.g. >250) and make the pairs more relevant (e.g. "ben-dan": ["ben-Beng", "dan-Latn"] is probably rarely used ;) )

[KennethEnevoldsen]

This is probably too many, and many are probably unreasonable.

We could consider Any to English. BibleNLPBitextMining does that. Flores uses Any to Any. However, it uses an efficient format that is only allowed because everything is paired up. I don't think this is the case here.

Alternatively, we could use Glottolog and do it within language families (any to eng and also, e.g., a connection between all the Scandinavian languages). For a similar approach to this, take a look at NTREXBitextMining (it might be that you can just use this one).

How long does it take to run?

[michaeldinzinger]

Now, the number of language pairs is reduced to 170. I kept Any2English, plus I tried to create some reasonable groups with respect to language families (e.g. grouping all languages from Caucasian, Indo-European (Baltic) and Indo-European (Slavic), and considering all combinations among the languages of these families). Besides, only those with at least 250 samples are considered; before, it was 100 samples.

Evaluation Time:

Model	Task	Time
multilingual-e5-small	WebFAQBitextMiningQuestions	68 mins
multilingual-e5-small	WebFAQBitextMiningQAs	78 mins
multilingual-e5-large	WebFAQBitextMiningQuestions	84 mins
multilingual-e5-large	WebFAQBitextMiningQAs	169 mins
paraphrase-multilingual-MiniLM-L12-v2	WebFAQBitextMiningQuestions	68 mins
paraphrase-multilingual-MiniLM-L12-v2	WebFAQBitextMiningQAs	80 mins

[KennethEnevoldsen]

Great very happy to merge this