Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch eval and lm-eval update #3

Merged
merged 2 commits into from
Oct 23, 2024
Merged

Batch eval and lm-eval update #3

merged 2 commits into from
Oct 23, 2024

Conversation

lihaoxin2020
Copy link
Contributor

  • Add OpenAI batch API feature for cheaper LM judge implementation, along with a few tool scripts to manage batch jobs.
  • Upgrade lm-eval compatibility. Recent models, like Llama 3.2, are supported. --apply_chat_template option is available with generic lm-eval support.
  • Push template for Llama and Qwen model families. Note that --chat_template should be general if --apply_chat_template is enabled to prevent double templates.

Unverified

This user has not yet uploaded their public signing key.
@lihaoxin2020 lihaoxin2020 merged commit f1c7f1e into main Oct 23, 2024
1 check passed
@lihaoxin2020 lihaoxin2020 deleted the batch-eval branch October 23, 2024 06:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant