RFC: Always set temperature to default?

[neubig]

Currently we set temperature to 0.0 for most models:

software-agent-sdk/openhands-sdk/openhands/sdk/llm/utils/model_features.py

Line 189 in 20291bd

def get_default_temperature(model: str) -> float:

This gets overridden by litellm in some cases, such as for gpt-5: https://github.com/BerriAI/litellm/blob/d2e113111f226f1e4d1c671b196beaf7248ea4e6/litellm/llms/openai/chat/gpt_5_transformation.py

And sometimes we get errors due to temperature being set to an invalid values.

I wonder if we should just always use the default temperature for all models?

[enyst]

For reasoning models or with reasoning effort set, we have to set 1, because only that is/was accepted by the API, or even drop the parameter.

FWIW, I seem to recall (older) Gemini models had different default temperatures. I don't have a reference, but I made a script with default hyperparameters at some point, and was surprised to see like 0.2 for one, 0.7 for another or so. The reasons weren't documented by google (I did search) 😅

Do you mean we could set the default set by the provider?

[neubig]

Yes, I mean we pass in no temperature, always use the default

[enyst]

I think maybe we may want to figure out what the default is, for the models on evals. To record it in the eval run logs.

Just in case the provider changes it later, even on models that should (tm) refer to the same model.

IMHO we could try it, it would simplify the code a bit, and idk, we may be surprised.

[all-hands-bot]

[Automatic Post]: This issue has been waiting for triage. @xingyaoww @neubig @enyst, could you please take a look and add the appropriate priority label when you have a chance?

[neubig]

@OpenHands

1. Open up an issue on the "OpenHands/benchmarks" repo reflecting enyst's comment.
2. Open up a draft PR fixing this issue, returning "None" as default instead of 0.0