Support Responses API

[enyst]

Related to

• Support reasoning content in Agent SDK #135

LLM providers have implemented ChatCompletions widely, with various wrinkles.
liteLLM offers an openai-compatible API to many providers.

Now we use litellm with chatcompletions in agent-sdk. On agent-sdk side, we want to continue using a simple, openai-compatible call to the LLM, which litellm will forward.

But relatively recently, OpenAI has released a different API: the Responses API. They send reasoning content only via Responses API, not the older ChatCompletions API. So at least for OpenAI models, we need to use Responses.

I can think of two options:

• we can implement Responses API directly in llm.py, by an adapter converting the messages on the fly
• or we can make a new llm_responses.py

The first option looks like this:

• Implement OpenAI Responses API adapter OpenHands#10451

[xingyaoww]

I just did an extensive overhaul refactor of llm.py, so i agree this is probably a good time to do this!

Seems like the conversion isn't too hard (at least it seems possible to do 1-1 translation of /chat/completions with /response:
https://platform.openai.com/docs/guides/migrate-to-responses?update-item-definitions
=chat-completions&update-multiturn=responses#migrating-from-chat-completions

And LiteLLM seems to already support it too https://docs.litellm.ai/docs/response_api.

I'm in favor of option 1 though, we can probably just write a new responses function next to completions (https://github.com/All-Hands-AI/agent-sdk/blob/47074e9b00c17e22048d36485bcdede023c380ac/openhands/sdk/llm/llm.py#L320-L401)

and maybe we can automatically convert completions request to responses API on the fly, at least for OpenAI to start?

[enyst]

Exactly, I think it’s doable. Let’s see how it looks like, maybe it doesn’t bring too much complexity.

@OpenHands Read all comments here and the links! Understand what we’re trying to achieve here. Then implement Xingyao’s suggestion.

[openhands-ai]

Pull Request Created

PR #142:

• feat: Add OpenAI Responses API support to LLM class #142

_{View full conversation}

[enyst]

Thoughts about this:

The Agent code now includes the former response_to_actions, and it depends on ChatCompletion litellm specific types.

• Responses API was created as a superset of Completions. That is, they're alike, but Responses contains Completions and some more, not the other way around.
  => Conversion:
  • from Responses API objects -> ChatCompletion (so use Responses transport)
    • it loses information
    • so we need to hack that in; otherwise it works (after PR 142), but it doesn't do actual benefit
  • from ChatCompletion -> Responses (using Completions transport)
    • it was intended to work
    • litellm actually has a bridge; it seems like the idea is that client code uses Responses, and for LLM providers that don't support it, it falls back to the "old" Completions.

IMHO we have three options, and some sub-categories of them:

1. Continue as we do now, graft extra Responses stuff on top of Completions

• use Completions ourselves, convert to Responses and back (so use Responses transport, with Completions tools/objects)
• PR 142

2. Support responses() and completion() fully - support both, as first class citizens: in llm.py and in the Agent, have two paths, completion() and responses();

• maybe break the Agent's dependency of ChatCompletion types by
  • restructuring the parsing of LLM responses on both APIs elsewhere again (e.g. a utility), or by
  • making a few wrappers around both APIs types, and do the parsing and conversion to events there

3. Make Responses API the first class citizen - support both, but make Responses API the first class citizen

• fallback / convert to Completions for most LLMs
• this appears to work in litellm
• it's an open question if we really can, because Anthropic and other LLMs have their own provider-specific content grafted onto Completions API

3.1 Wrap them ourselves, use our own types

• support both, make a wrapper around Responses API, including provider-specific Completions API content types
• use litellm's bridge to let most LLMs fall back to /chat/completion as they always did
• gives one path, with discriminator used only as necessary
• probably means that the Agent switches to .responses()

[xingyaoww]

I'm leaning towards (2), but keeping our event data structure unchanged, where it gives us the most control (only use responses when dealing with OpenAI model), although it might mean we need to do more work.

I think we can gradually transition to (3) if everyone starts to use it, OR let's have more robust support (but I'm sure there will be bugs..)

[enyst]

I do like (2)!

I think it will help us figure things out.

We might want to consider two extra things in the event structure though, no matter what path:

• reasoning summaries are a list of summaries, currently we mash them together (into a reasoning_content: string); not a big deal though because we're not sending it back
• more importantly, encrypted summary, will be necessary for using Responses API statelessly with reasoning sent back for the model to continue; it would be pointless to display that to the user, but I think we need it stored somewhere 🤔

[enyst]

To note, for the record and to keep things in one place:

In the review discussions on #142 we figured out that maybe keeping the conversion is still better, for now, because we may need it to also switch LLMs (a Responses API LLM <-> a Completions-only LLM)

On the other hand, litellm has a Responses-to-Completions bridge, which does the other way around than what ours is doing: client code can use responses(), and litellm converts undercover if the LLM doesn’t actually support responses(). That could help us with the switch LLM.