llms: Add support for using the whisper model to transcribe audio

[devalexandre]

PR Checklist

• Read the Contributing documentation.
• Read the Code of conduct documentation.
• Name your Pull Request title clearly, concisely, and prefixed with the name of the primarily affected package you changed according to Good commit messages (such as memory: add interfaces for X, Y or util: add whizzbang helpers).
• Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
• Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. Fixes #123).
• Describes the source of new concepts.
• References existing implementations as appropriate.
• Contains test coverage for new functions.
• Passes all golangci-lint checks.

---

[eliben]

Does it make sense to think of "transcribe audio" in the context of LLMs? AFAIU Whisper is a distinct model from LLMs like the GPT family.

Is this intended as a one-off method only for openai, or as some general audio transcription interface?

[devalexandre]

only for openai, but I think it's interesting to include it in the general context, because there are other models that do this, but at the moment it's only for openai

[corani]

I haven't tried it yet, but this seems interesting as a locally running alternative: https://github.com/JigsawStack/insanely-fast-whisper-api

It would be cool if we could support something like that, so you can combine it with a Ollama to build some local-only tools.

(So far I've been using https://github.com/Purfview/whisper-standalone-win locally, which is a single-binary wrapper around https://github.com/SYSTRAN/faster-whisper)

[devalexandre]

We can implement it in other LLMS, the problem is that so far I have only found the standalone version for Windows, but I am looking for other alternatives

[tmc]

I agree with @eliben's intuition here, I'm not sure if audio transcription as a concept fits right into our llm namespace. I'm open to exposing this and generalizing over providers but I think it belongs in a different namespace.

[devalexandre]

@tmc , @eliben

What would be the implementation idea for this functionality? maybe use openai.TranscribeAudio, leaving it only within the openai package and not in the LLM namespace?

I think in use how it, do a loader
https://js.langchain.com/docs/integrations/document_loaders/file_loaders/openai_whisper_audio

[devalexandre]

@tmc some update ?