The validation set performs very well, but the F1 score on the test set is even 0. What could be the reason for this?

[zcliu814]

Hello, I followed your Colab steps exactly without any modifications to the data, but the F1 score for FN is 0 on the test set, and FN0 performs particularly poorly. What could be the reason for this?

F1_PN F1_PN0 default mode
0 58.322 67.875 67.875 valid
1 0.000 28.557 28.557 test
2 0.000 28.557 28.557 test

This is my result in colab

[zcliu814]

Looking forward to your explanation. Is it because the test set doesn't have labels? Thank you!

[nicolay-r]

Hi @zcliu814 ! Thank you for experimenting on this!
By looking at your results, I can tell that everything goes well with the training, you got more-or-less expected resutls on valid set.
Yes, the labels for the test part of the RuSentNE-2023 dataset and the related competition are not publicly available to prevent the leakage and consistency of the benchmark 🔒
Thanks for poining it out though, I can agree it might be found confusing

[nicolay-r]

However, you're still have an option to officially test your model!

You have to prepare the zip 🗃️ archive with the answers onto the test set like this:
(it is just a list of labels per every input example)
https://github.com/dialogue-evaluation/RuSentNE-evaluation/blob/main/baseline_submission.zip
Simply just go to:
https://codalab.lisn.upsaclay.fr/competitions/9538
register your email.
and then go to the participate tab:

[zcliu814]

Thank you very much for your response, I will give it a try.

[nicolay-r]

Sure, keep me update in the case of other questions and new issues out there!