ValueError: please provide at least one prompt

[fe1ixxu]

After I followed SFT step,

accelerate launch --config_file=configs/zero2.yaml src/open_r1/sft.py \
    --model_name_or_path Qwen/Qwen2.5-Math-1.5B-Instruct \
    --dataset_name HuggingFaceH4/Bespoke-Stratos-17k \
    --learning_rate 2.0e-5 \
    --num_train_epochs 1 \
    --packing \
    --max_seq_length 4096 \
    --per_device_train_batch_size 4 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 4 \
    --gradient_checkpointing \
    --bf16 \
    --logging_steps 5 \
    --eval_strategy steps \
    --eval_steps 100 \
    --output_dir outputs/qwen-1.5B-distill

When I tried to evaluate,

MODEL=outputs/qwen-1.5B-distill
MODEL_ARGS="pretrained=$MODEL,dtype=float16,max_model_length=4096,gpu_memory_utilisation=0.8"
TASK=math_500 #aime24
OUTPUT_DIR=outputs/evals/qwen-1.5b-distill-$TASK

lighteval vllm $MODEL_ARGS "custom|$TASK|0|0" \
    --custom-tasks src/open_r1/evaluate.py \
    --use-chat-template \
    --system-prompt="Please reason step by step, and put your final answer within \boxed{}." \
    --output-dir $OUTPUT_DIR 

I got the error:

[rank0]: │ parse_and_batch_prompt                                                                           │
[rank0]: │                                                                                                  │
[rank0]: │    55 │   │   if is_list_of(prompt, list):                                                       │
[rank0]: │    56 │   │   │   prompt = cast(List[List[int]], prompt)                                         │
[rank0]: │    57 │   │   │   if len(prompt[0]) == 0:                                                        │
[rank0]: │ ❱  58 │   │   │   │   raise ValueError("please provide at least one prompt")                     │
[rank0]: │    59 │   │   │                                                                                  │
[rank0]: │    60 │   │   │   if is_list_of(prompt[0], int):                                                 │
[rank0]: │    61 │   │   │   │   # case 4: array of token arrays                                            │
[rank0]: │                                                                                                  │
[rank0]: │ ╭────────────────────────── locals ───────────────────────────╮                                  │
[rank0]: │ │ prompt = [[], [], [], [], [], [], [], [], [], [], ... +490] │                                  │
[rank0]: │ ╰─────────────────────────────────────────────────────────────╯                                  │
[rank0]: ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
[rank0]: ValueError: please provide at least one prompt

[fe1ixxu]

But very interesting thing is if I directly evaluate deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B, it runs without the error.

[Kartik14]

This is happening because the because the script is silently discarding inputs with length > 4096. One fix is to increase the ctx length to 32k in the model config file and run the eval with --max_seq_length 32k.

[fe1ixxu]

Thanks for the quick response! Do you mean < 4096? But may I ask any insights that why we are discarding these inputs?

[fe1ixxu]

@Kartik14 I tried to increase max_position_embeddings and model_max_length to 32768 in the model config and run the eval with

MODEL=outputs/qwen-1.5B-distill
MODEL_ARGS="pretrained=$MODEL,dtype=float16,max_model_length=32768,gpu_memory_utilisation=0.8"
TASK=math_500 #aime24
OUTPUT_DIR=outputs/evals/qwen-1.5b-distill-$TASK

lighteval vllm $MODEL_ARGS "custom|$TASK|0|0" \
    --custom-tasks src/open_r1/evaluate.py \
    --use-chat-template \
    --system-prompt="Please reason step by step, and put your final answer within \boxed{}." \
    --output-dir $OUTPUT_DIR 

But I still got the error

[fe1ixxu]

Oh It works for me when I set "max_position_embeddings": 131072, but still confused with this design of discarding short inputs.