Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Final changes to support the new LLM as Judges #1469

Open
3 tasks
yoavkatz opened this issue Jan 5, 2025 · 0 comments
Open
3 tasks

Final changes to support the new LLM as Judges #1469

yoavkatz opened this issue Jan 5, 2025 · 0 comments
Assignees

Comments

@yoavkatz
Copy link
Member

yoavkatz commented Jan 5, 2025

This issue documents what's still needs to be done:

  • Change the main LLM documentation . Make the new way the recommended away. Keep the old way only at the end highlighting this is the way to create a judge when you need full control of the input to the model.
  • Add all new examples to the examples documentation page. Remove most legacy examples, except one or two .
  • Finalize the "main" score of the the new llm as judge metrics.
    For direct: "llm_as_judge_score" - The mean score of all the instance judge scores.
    For pairwise: "1_win_rate" - The mean win_rate of the first system
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants