How to evaluate?

[byronBBL]

I see the evaluate script, but the test_***.py python file is missing.
Want to ask if it's open source, or provide implementation details

[YijuGuo]

Thank you for your question.

Regarding the helpfulness dimension, we conduct tests on MTBench.

For the honesty and harmlessness dimensions, our test data can be found at: src/CPSFT/cpsft/test_data_harmlessness and src/CPSFT/cpsft/test_data_honesty. The test code is located at: src/CPSFT/cpsft/test_harmlessness.py and src/CPSFT/cpsft/test_honesty.py.

I hope this information is helpful to you.

[CurryxIaoHu]

Thank you for your question.

Regarding the helpfulness dimension, we conduct tests on MTBench.

For the honesty and harmlessness dimensions, our test data can be found at: src/CPSFT/cpsft/test_data_harmlessness and src/CPSFT/cpsft/test_data_honesty. The test code is located at: src/CPSFT/cpsft/test_harmlessness.py and src/CPSFT/cpsft/test_honesty.py.

I hope this information is helpful to you.

It seems that the code provided for evaluating honesty only output answers from LLMs. There is no output metric. Can you provide the code for this?

[YijuGuo]

To rate the responses generated by GPT-4 regarding the templates for "helpfulness" and "honesty" in the file preference_templates.py from the repository https://github.com/OpenBMB/UltraFeedback/blob/main/src/data_annotation/preference_templates.py