Open AI API Maturity

[Jack-Khuu]

🚀 The feature, motivation and pitch

The OpenAI API support in torchchat is actively in development and will be one of the main entry point for interacting with torchchat

https://github.com/pytorch/torchchat?tab=readme-ov-file#server

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

[vmpuri]

PR 995 addresses some initial concerns. Responses should now be formatted in JSON using the API dataclasses. Here are our functional gaps so far:

• system_fingerprint/seed implementation is incomplete. We need to figure out a good way to create and consume fingerprints from seeds. For now, the client cannot control reproducibility of the responses.
• model cannot be selected by the client. The model is selected when starting the server and cannot be changed without exiting the process. The server does not check to ensure the request/response models match
• Agentic behavior/toolcall functionality is unimplemented. We will focus on getting chat right first, then create entrypoints for tool usecases.

Behavior Issues

• Model responses may not respond in the expected manner. See Weird model behaviour on Server/Browser: Looks like it's not using the template #989

[ywang96]

Hello there! I'm curious if supporting OpenAI Vision API is also something you're interested in since it looks like the team does plan to support vision language models from the discussion in #988

If you do, happy to help!

[vmpuri]

Hey Roger, thanks for reaching out! We do plan on adding LLaVA support in the coming weeks, and we'd appreciate your help on the API/server components.

I'm still working on bringing the initial version of the API and server up to spec, but I'll keep this issue updated with my progress.

[vmpuri]

Landed pulls #1035 #1034 #1042 which prove that the basic completion API works as expected with the Python OpenAI API.