SentenceTransformerTrainer compute_metrics

[Samoed]

Hi! I tried to train my model and evaluate it using compute_metrics, but I didn't get any metrics. Is this a bug, or is it not supposed to work?

Code for test:
https://colab.research.google.com/drive/11sml4nfhkVVoZy0fTsll6BLgpHYz-BWD

[ir2718]

Hi,

I think this is the expected behaviour. but I'm not 100% sure. The compute_metrics function usually takes in a list of predictions which you can then use to calculate the metrics. With triplet training, the difference is in the evaluation procedure, such that the evaluators take in a SentenceTransformer model used to calculate the anchor, positive, and negative embeddings, which are then used for calculating metrics.

Besides, there is no obvious way to define false positives and false negatives with triplet information. The only thing you can say when you've got triplet data is whether a positive is closer to the anchor than the negative, hence accuracy is only calculated.

[Samoed]

There's no information about this expected behavior or not. Maybe printing warning if the function is passed will be good.

[jacklanda]

+1

[tomaarsen]

Hello!

The compute_metrics argument is inherited from the transformers Trainer superclass, where it is primarily used here: https://github.com/huggingface/transformers/blob/b54109c7466f6e680156fbd30fa929e2e222d730/src/transformers/trainer.py#L4184-L4192

This happens during every evaluation, but Sentence Transformer models don't usually train with single inputs & single outputs, so the logits and labels are both None here. To be precise, they are None because the prediction_step calls compute_loss, which is implemented in the SentenceTransformerTrainer:

sentence-transformers/sentence_transformers/trainer.py

Lines 345 to 351 in 1802076

	if return_outputs:
	# During prediction/evaluation, `compute_loss` will be called with `return_outputs=True`.
	# However, Sentence Transformer losses do not return outputs, so we return an empty dictionary.
	# This does not result in any problems, as the SentenceTransformerTrainingArguments sets
	# `prediction_loss_only=True` which means that the output is not used.
	return loss, {}
	return loss

As a result, the compute_metrics function is never called over the EvalPrediction.

---

In Sentence Transformers, if you want to compute evaluations, it is recommended to use one of the Evaluators. There's some more information about them here.

If you want to create your own evaluator, you can subclass the SentenceEvaluator class. You can pass an evaluator to the STTrainer, or even a list if you have multiple.

• Tom Aarsen

[Samoed]

Thank you very much! Maybe add a warning about this? Because this is a bit unexpected behavior