Reward verification and evaluation fixes

[hynky1999]

• bumps math-verify to the newest version
• fixes aime evaluation by using the answer column directly (this would be problem for few-shot, but since we don't use it I think it's fine)
• Changes rewards evaluation:
  1 .The answer from model must be clearly preceded by answer is xxx or by be in \boxed env (with the boxed env being tried first).
  2. No latex malformed operators + nits are applied to the resulting latex, meaning model has to output correct latex.
  3. the gold must be in latex format, this ensures that the extracted gold is exactly what we want to compare against
  4. Removed the try/catch it's handled in math-verify now

cc @qgallouedec, I am not sure whether returning 1 if the gold is not found is the correct way to do this. It assumes that if rewards is 1 no gradient change is enforced is that correct?

[qgallouedec]

cc @qgallouedec, I am not sure whether returning 1 if the gold is not found is the correct way to do this. It assumes that if rewards is 1 no gradient change is enforced is that correct?

You can return any value here: the model generates a group of completions for one prompt, then these completions are rewarded, then normalised. If all completions get the same reward (because the common prompt is not parsable here), then the normalisation outputs a tensor of 0s. So no gradient.

[qgallouedec]

Nice!!
I approve the GRPO part

[pepinu]

one comment about abstracting away how reward is calculated from internals of the loop

[pepinu]

I'd move l46-l73 to another method named get_reward

def get_reward(content: str, solution: str) -> float:
    """
    Calculate reward by comparing the parsed content with the ground truth solution.
    
    Args:
        content (str): The model's output content to evaluate
        solution (str): The ground truth solution to compare against
        
    Returns:
        float: 1.0 if the content matches the solution, 0.0 otherwise
    """
    gold_parsed = parse(solution, extraction_mode="first_match", extraction_config=[LatexExtractionConfig()])
    
    if len(gold_parsed) == 0:
        return 1.0
        
    answer_parsed = parse(
        content,
        extraction_config=[
            LatexExtractionConfig(
                normalization_config=NormalizationConfig(
                    nits=False,
                    malformed_operators=False,
                    basic_latex=True,
                    equations=True,
                    boxed=True,
                    units=True,
                ),
                boxed_match_priority=0,
                try_extract_without_anchor=False,
            )
        ],
        extraction_mode="first_match",
    )
    
    return float(verify(answer_parsed, gold_parsed))

then the loop on line 45 changes to

def accuracy_reward(completions, solution, **kwargs):
    """Reward function that checks if the completion is the same as the ground truth."""
    contents = [completion[0]["content"] for completion in completions]
    rewards = []
    
    for content, sol in zip(contents, solution):
        reward = get_reward(content, sol)
        rewards.append(reward)
        
    return rewards

Which improves readability

[hynky1999]

Think it's fine for now, let's move it once we add stuff for code and other rewards.

[lewtun]

Thanks for the fix on AIME and improvement to GRPO! Both changes LGTM - feel free to merge with @qgallouedec's suggestion