fix Bug in BertScore calculation: pred target misalignment

[gxy-gxy]

fix Bug in BertScore calculation: pred target misalignment

Fixes bug in BertScore cal.
This pull request addresses a bug identified in the BertScore calculation within the TextDataset class in src/torchmetrics/functional/text/helper_embedding_metric.py.
The class is designed with a preprocess function automatically sorts input text by length to optimize batch encoding efficiency. However, this behavior introduces an issue during the BertScore calculation process, as predictions (preds) and targets (targets) are initialized in separate datasets. This results in a mismatched ordering of text pairs, which is problematic given the pairwise nature of BertScore's calculation. To ensure accurate scoring, it is critical to re-align the datasets to their original order before computing the scores. The proposed fix involves ensuring that the datasets for predictions and targets are processed in a way that maintains their original pairing throughout the calculation process.

Here is the fixed code:

preds_embeddings = preds_embeddings[preds_loader.dataset.sorting_indices]
target_embeddings = target_embeddings[target_loader.dataset.sorting_indices]

This change is essential for preserving the integrity of the BertScore evaluation, ensuring that each prediction is accurately compared against its corresponding target.

---

📚 Documentation preview 📚: https://torchmetrics--2347.org.readthedocs.build/en/2347/

[gxy-gxy]

Here is the test code:

from torchmetrics.text.bert import BERTScore

score_model = BERTScore(model_name_or_path='roberta-large', batch_size=2)

text1 = [ "Claim A from machine", "Claim A from machine"]
text2 = ["Claim A from machine", "Claim B"]
similarities = score_model(text1, text2)
print(similarities)

[Borda]

@stancld could you help here, pls?

[baskrahmer]

Good catch! I can reproduce the bug using your test code snippet. Although the actual metric values seem to be correct, the ordering is not always valid.

It might be nice to somehow integrate this case with the current test suite. E.g. an assertation that reversing the targets/preds also reverses the scores.

[Borda]

Here is the test code:

from torchmetrics.text.bert import BERTScore

score_model = BERTScore(model_name_or_path='roberta-large', batch_size=2)

text1 = [ "Claim A from machine", "Claim A from machine"]
text2 = ["Claim A from machine", "Claim B"]
similarities = score_model(text1, text2)
print(similarities)

@gxy-gxy can you pls add it as a test?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 39%. Comparing base (30b3fd5) to head (dc3b758).

❗ There is a different number of reports uploaded between BASE (30b3fd5) and HEAD (dc3b758). Click for more details.

HEAD has 7 uploads less than BASE

Flag	BASE (30b3fd5)	HEAD (dc3b758)
Windows	4	2
cpu	22	21
torch2.4.0+cpu	2	1
python3.11	7	5
torch2.3.0+cpu	3	2

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #2347     +/-   ##
========================================
- Coverage      69%     39%    -30%     
========================================
  Files         316     316             
  Lines       17878   17874      -4     
========================================
- Hits        12329    7030   -5299     
- Misses       5549   10844   +5295     

[Borda]

adding test - #2347 (comment)

[baskrahmer]

LGTM! Thanks for the addition. I can quickly add the test

[Borda]

@baskrahmer, would mind also adding an entry to the changelog?