🚧 Work in progress.

Use humaneval.py --template to apply templates to humaneval interview suite turning them into prompts suitable for interview.

Note the format here is different, uses a single {{prompt}} variable as the input.

Vicuna-1p1-HumanEval.txt

Wizard-7B-HumanEval.txt

Llama-Humaneval.txt

Use humaneval.py --answers to convert any interview output into .jsonl format and use the upsteam evaluator: https://github.com/openai/human-eval