Default temperature to None in the SDK

[enyst]

We would like to remove the default temperature=0, and if possible, the default temperature=1 for some reasoning models, and let it fallback to None.

I think maybe we want to make sure we detect and save the temperatures at which we eval, either by recording the default of a model, or by setting it always to 0. I’m not sure if we currently do that?

References:

• RFC: Always set temperature to default? software-agent-sdk#1913
• Set default temperature to None instead of 0.0 software-agent-sdk#1989

[juanmichelini]

Setting it to none is a good idea. Currently we are setting the temperature for each model used in evals in the software-agent-sdk here: https://github.com/OpenHands/software-agent-sdk/blob/main/.github/run-eval/resolve_model_config.py

We might want to check the few cases were we use temperature=0.0 in this repo, I think most are tests and legacy code though.

[enyst]

Thank you!

Currently we are setting the temperature for each model used in evals in the software-agent-sdk here: https://github.com/OpenHands/software-agent-sdk/blob/main/.github/run-eval/resolve_model_config.py

I'm sorry I'm dense, I see in the file we are setting temperature for several models, but not for all models in the file. For example Geminis don't have temperature explicitly set:

    "gemini-3-pro": {
        "id": "gemini-3-pro",
        "display_name": "Gemini 3 Pro",
        "llm_config": {"model": "litellm_proxy/gemini-3-pro-preview"},
    },
    "gemini-3-flash": {
        "id": "gemini-3-flash",
        "display_name": "Gemini 3 Flash",
        "llm_config": {"model": "litellm_proxy/gemini-3-flash-preview"},
    },

(line 70)

According to its docs, Gemini 3 series has a default of 1:

For Gemini 3, we strongly recommend keeping the temperature parameter at its default value of 1.0.
(source)

Minimax-m2, gpt models, and others, also don't have temperature set in that file?

To clarify, my concern is that we eval-ed them at an explicit temperature we knew (typically 0), becaue we enforced it hardcoded, but once we remove that default in the SDK, they will default to whatever their provider sets for temperature. For example, for Gemini 3 series and Minimax-m2 it seems it will be 1.

That may be a good thing for future evals, but it's not equivalent with the past evals, right? Please point out if I'm missing something!

[enyst]

I think you're right, actually, that we could record in that file the exact temperature we used.

What if we do this:

• set the temperature we actually used, in that file, explicitly, for all models
• check all hardcoded occurrences as you said, just in case
• (more of an issue for the other PR) do the same for reasoning models, where we may have hardcoded 1