Performance Gap Between Reproduction and Paper Results on HumanEval and MBPP

[ZinYY]

Hi, thanks for sharing this impressive work. While evaluating Dream-org/Dream-Coder-v0-Instruct-7B on the HumanEval and MBPP benchmarks, I observed that the reproduced results do not match the performance reported in the paper.

For humaneval, I used a command aligned with the instructions in the Dream's official repository (https://github.com/DreamLM/Dream/blob/main/eval_instruct/eval.sh):

HF_ALLOW_CODE_EVAL=1 PYTHONPATH=. accelerate launch --main_process_port 12334 -m lm_eval \
    --model diffllm \
    --model_args pretrained=Dream-org/Dream-Coder-v0-Instruct-7B,trust_remote_code=True,max_new_tokens=768,diffusion_steps=768,dtype="bfloat16",temperature=0.1,top_p=0.9,alg="entropy" \
    --tasks humaneval_instruct \
    --device cuda \
    --batch_size 1 \
    --num_fewshot 0 \
    --output_path output_reproduce/humaneval \
    --log_samples --confirm_run_unsafe_code \
    --apply_chat_template

My reproduction yields a HumanEval score of 74.39:

diffllm (pretrained=Dream-org/Dream-Coder-v0-Instruct-7B,trust_remote_code=True,max_new_tokens=768,diffusion_steps=768,dtype=bfloat16,temperature=0.1,top_p=0.9,alg=entropy), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 1
|      Tasks       |Version|  Filter   |n-shot|Metric|   |Value |   |Stderr|
|------------------|------:|-----------|-----:|------|---|-----:|---|-----:|
|humaneval_instruct|      2|create_test|     0|pass@1|   |0.7439|±  |0.0342|

This is significantly below the 82.9 reported in the paper.

---

For MBPP, I used the command:

HF_ALLOW_CODE_EVAL=1 PYTHONPATH=. accelerate launch --main_process_port 12334 -m lm_eval \
    --model diffllm \
    --model_args pretrained=Dream-org/Dream-Coder-v0-Instruct-7B,trust_remote_code=True,max_new_tokens=1024,diffusion_steps=1024,dtype="bfloat16",temperature=0.1,top_p=0.9,alg="entropy" \
    --tasks mbpp_instruct \
    --device cuda \
    --batch_size 1 \
    --num_fewshot 0 \
    --output_path output_reproduce/mbpp \
    --log_samples --confirm_run_unsafe_code \
    --apply_chat_template

My reproduced MBPP score is 66.84, which is also lower than the 79.6 reported in the paper.

---

Could you clarify why these discrepancies occur or whether additional evaluation settings are required to match the results in the paper? Thank you!

[zhxieml]

Hi @ZinYY, thanks for the detailed report and for sharing your commands!

Why your scores differ from the paper

The main reason is that the paper reports results using EvalPlus, while your commands are using lm-evaluation-harness.

• EvalPlus reimplements HumanEval/MBPP and adds much more comprehensive test suites (HumanEval+, MBPP+), fixes buggy cases, and standardizes the evaluation harness.
• There are known cases where the same model differs by >10 points across HumanEval harnesses, so the gap you see is within the kind of shift we expect from changing evaluation suites.

Why we use EvalPlus (and not lm-eval) for Dream-Coder

For Dream-Coder’s reported results we use EvalPlus because:

• It is designed specifically for code benchmarks and provides extended, cleaned test sets and a unified harness.
• Recent strong code models (e.g., Qwen Coder series) also report on EvalPlus, so using it keeps Dream-Coder directly comparable to current open-source models.
• All baselines in the Dream-Coder paper (HumanEval, MBPP, and their “+” variants) were run with the same EvalPlus pipeline, ensuring apples-to-apples comparison.

The earlier Dream-7B release focused on general capabilities (math, reasoning, etc.), where lm-eval is very convenient because it offers a broad task zoo beyond code. We therefore continued using lm-eval for Dream-7B.

How to reproduce the paper numbers

To match the reported results, please use our EvalPlus-based evaluation suite.

[ZinYY]

Thank you for the quick response and the helpful instructions! I am now trying to evaluate using the evalplus package. However, I am wondering if some files might be missing in the instruct/code_eval/evalplus folder. It seems that the --backend dllm option doesn't work in evalplus.evaluate.

Thank you!

[zhxieml]

I am wondering if some files might be missing in the instruct/code_eval/evalplus folder

Thank you for pointing this out!

I've looked into it, and the core logic was indeed missing because its subfolder was accidentally included in the .gitignore file. This has been fixed in #12.