Adding av adapters #144

jherr · 2025-12-13T01:17:37Z

🎯 Changes

This PR adds three new AI activities to @tanstack/ai: Video Generation, Text-to-Speech (TTS), and Audio Transcription.

New Activities

🎬 Video Generation

Core types: VideoOptions, VideoResult, VideoCreateResult, VideoStatusResult in types.ts
Adapter pattern: VideoAdapter interface with createVideo(), getVideoStatus(), getVideoUrl() methods supporting async job-based generation
Activity: videoActivity with kind: 'video' supporting polling workflows
OpenAI implementation: Full support via Sora API (sora-2.0-* models)

🗣️ Text-to-Speech (TTS)

Core types: TTSOptions, TTSResult with base64 audio output
Adapter pattern: TTSAdapter interface with generateSpeech() method
Activity: ttsActivity with kind: 'tts'
OpenAI implementation: tts-1, tts-1-hd models with voice selection (alloy, echo, fable, onyx, nova, shimmer)
Gemini implementation: Experimental support via gemini-2.5-flash-preview-tts

🎤 Audio Transcription

Core types: TranscriptionOptions, TranscriptionResult with segments/words timestamps
Adapter pattern: TranscriptionAdapter interface with transcribe() method
Activity: transcriptionActivity with kind: 'transcription'
OpenAI implementation: whisper-1, gpt-4o-transcribe models with language detection, timestamps

Package Changes

`@tanstack/ai`

New activity directories: activities/video/, activities/tts/, activities/transcription/
Extended ai() function to handle all new activity types
Updated AIAdapter, AIOptionsFor, AIResultFor union types

`@tanstack/ai-openai`

New adapters: OpenAIVideoAdapter, OpenAITTSAdapter, OpenAITranscriptionAdapter
New provider options: VideoProviderOptions, AudioProviderOptions, TranscribeProviderOptions
New model metadata for video, TTS, and transcription models

`@tanstack/ai-gemini`

New adapter: GeminiTTSAdapter (experimental)
Model metadata for TTS

Testing Panel (`testing/panel/`)

New pages: /video, /tts, /transcription with full UI for testing each activity
Chat enhancements:
- 🎤 Microphone button for voice input → transcription → text
- 🔊 Speaker button on assistant messages for TTS playback
New hooks: useAudioRecorder, useTTS in src/hooks/

Smoke Tests

New tests: TTS (text-to-speech), TRN (transcription)
Extended AdapterContext with ttsAdapter, transcriptionAdapter, ttsModel, transcriptionModel
Improved report format: removed model names from table, replaced emoji ellipsis with emdash

Documentation

New guides: text-to-speech.md, transcription.md, video-generation.md
Updated adapters: openai.md and gemini.md with TTS/Transcription/Video sections
Navigation: Added new guide entries to config.json

✅ Checklist

I have followed the steps in the Contributing guide.
I have tested this code locally with pnpm run test:pr.

🚀 Release Impact

This change affects published code, and I have generated a changeset.
This change is docs/CI/dev-only (no release).

coderabbitai · 2025-12-13T01:17:42Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch adding-av-adapters

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

nx-cloud · 2025-12-13T01:19:33Z

View your CI Pipeline Execution ↗ for commit ee99400

Command	Status	Duration	Result
`nx affected --targets=test:sherif,test:knip,tes...`	✅ Succeeded	26s	View ↗
`nx run-many --targets=build --exclude=examples/**`	✅ Succeeded	3s	View ↗

☁️ Nx Cloud last updated this comment at 2025-12-13 17:06:15 UTC

pkg-pr-new · 2025-12-13T01:21:19Z

Open in StackBlitz

@tanstack/ai

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai@144

@tanstack/ai-anthropic

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-anthropic@144

@tanstack/ai-client

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-client@144

@tanstack/ai-devtools-core

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-devtools-core@144

@tanstack/ai-gemini

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-gemini@144

@tanstack/ai-ollama

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-ollama@144

@tanstack/ai-openai

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-openai@144

@tanstack/ai-react

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-react@144

@tanstack/ai-react-ui

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-react-ui@144

@tanstack/ai-solid

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-solid@144

@tanstack/ai-solid-ui

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-solid-ui@144

@tanstack/ai-svelte

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-svelte@144

@tanstack/ai-vue

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-vue@144

@tanstack/ai-vue-ui

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-vue-ui@144

@tanstack/react-ai-devtools

npm i https://pkg.pr.new/TanStack/ai/@tanstack/react-ai-devtools@144

@tanstack/solid-ai-devtools

npm i https://pkg.pr.new/TanStack/ai/@tanstack/solid-ai-devtools@144

commit: ee99400

…ng-av-adapters

AlemTuzlak

Looks great, I can add some more typesafety and check the common and providerOptions to make sure our API is rock solid

AlemTuzlak · 2025-12-13T08:23:24Z

docs/guides/text-to-speech.md

+    text: 'Hello!',
+  })
+} catch (error) {
+  if (error.message.includes('exceeds maximum length')) {


A comment I saw on X about better-auth is that error handling sucks because you cant know what error is thrown and handle it easily, maybe we should talk about having an errormap for each ai method on all the error types and have them mapped internally, eg TooManyTokensError, TimedOutError

* video generation * text to speech and speech to text * adding some cool audio UI to the dev panel * small fixups * ci: apply automated fixes * client fixes on tool calls * few more client fixups * one last test fix --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

#137) * feat: split up adapters into functionalities and add generate function * ci: apply automated fixes * readme update * add structured outputs and stream flag * ci: apply automated fixes * image generation for openai and gemini * ci: apply automated fixes * structured outputs * lock fix * ci: apply automated fixes * Splitting made operational with extra tests (#142) * fix: refactoring ai for more activities * smoke tests passing * woot, all the test stuff is working * dev panel updates for images, summarization, one shot and structured * enhancing smoke tests * fixing tests * adding grok * last minute tests * Refactor imports in documentation and examples to use named imports for `ai` - Updated all instances of `import ai from "@tanstack/ai"` to `import { ai } from "@tanstack/ai"` across various documentation files, guides, and examples. - Ensured consistency in import statements for better clarity and adherence to best practices. * ci: apply automated fixes * fix typesafety on ai * ci: apply automated fixes * cleanup types * ci: apply automated fixes * remove grok * ci: apply automated fixes * fix provenence? * update deps * fix tests --------- Co-authored-by: Alem Tuzlak <t.zlak@hotmail.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Adding av adapters (#144) * video generation * text to speech and speech to text * adding some cool audio UI to the dev panel * small fixups * ci: apply automated fixes * client fixes on tool calls * few more client fixups * one last test fix --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Splitting adapters up fixes (#148) * fixing JSON-Schema conversion * summarize now really supports streaming * ci: apply automated fixes * linting fixes * feat: add createOptions helper and improve summarization streaming - Add createOptions() function for type-safe adapter option creation - Refactor OpenAI summarize adapter to use text adapter for streaming - Deprecate textOptions() in favor of createOptions() - Update examples to use createOptions pattern - Add runtime adapter switching documentation guide * massive overhaul to remove ai * ci: apply automated fixes * Going to openaiText * ci: apply automated fixes * doc fixes and removal of legacy exports * adding more options to gemini audio * doc updates * ci: apply automated fixes * add docs script --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Alem Tuzlak <t.zlak@hotmail.com> * migrate model into adapter (#157) * migrate model into adapter * ci: apply automated fixes * Feat/move model into adapter fix types and cleanup (#158) * checkpoint * checkpoint * ci: apply automated fixes * unrename * move to _types pattern * Remove commented code * Goodbye embeddings * move to ~types * checkpoint * ci: apply automated fixes * updated docs and fixed a small bug with types --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Alem Tuzlak <t.zlak@hotmail.com> --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Tanner Linsley <tannerlinsley@gmail.com> * type safety tests for chat * ci: apply automated fixes * image tests * ci: apply automated fixes * update format * ci: apply automated fixes * flatten options object * ci: apply automated fixes * testing fixups * fix: streaming functions and migration doc * few more adapter/model fixups * ci: apply automated fixes * small bug fixes * ci: apply automated fixes * fix up exports and example * tighten up types * ci: apply automated fixes * tighten up types * ci: apply automated fixes * fixes * ci: apply automated fixes * updating the changeset --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Jack Herrington <jherr@pobox.com> Co-authored-by: Tanner Linsley <tannerlinsley@gmail.com>

TanStack#137) * feat: split up adapters into functionalities and add generate function * ci: apply automated fixes * readme update * add structured outputs and stream flag * ci: apply automated fixes * image generation for openai and gemini * ci: apply automated fixes * structured outputs * lock fix * ci: apply automated fixes * Splitting made operational with extra tests (TanStack#142) * fix: refactoring ai for more activities * smoke tests passing * woot, all the test stuff is working * dev panel updates for images, summarization, one shot and structured * enhancing smoke tests * fixing tests * adding grok * last minute tests * Refactor imports in documentation and examples to use named imports for `ai` - Updated all instances of `import ai from "@tanstack/ai"` to `import { ai } from "@tanstack/ai"` across various documentation files, guides, and examples. - Ensured consistency in import statements for better clarity and adherence to best practices. * ci: apply automated fixes * fix typesafety on ai * ci: apply automated fixes * cleanup types * ci: apply automated fixes * remove grok * ci: apply automated fixes * fix provenence? * update deps * fix tests --------- Co-authored-by: Alem Tuzlak <t.zlak@hotmail.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Adding av adapters (TanStack#144) * video generation * text to speech and speech to text * adding some cool audio UI to the dev panel * small fixups * ci: apply automated fixes * client fixes on tool calls * few more client fixups * one last test fix --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Splitting adapters up fixes (TanStack#148) * fixing JSON-Schema conversion * summarize now really supports streaming * ci: apply automated fixes * linting fixes * feat: add createOptions helper and improve summarization streaming - Add createOptions() function for type-safe adapter option creation - Refactor OpenAI summarize adapter to use text adapter for streaming - Deprecate textOptions() in favor of createOptions() - Update examples to use createOptions pattern - Add runtime adapter switching documentation guide * massive overhaul to remove ai * ci: apply automated fixes * Going to openaiText * ci: apply automated fixes * doc fixes and removal of legacy exports * adding more options to gemini audio * doc updates * ci: apply automated fixes * add docs script --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Alem Tuzlak <t.zlak@hotmail.com> * migrate model into adapter (TanStack#157) * migrate model into adapter * ci: apply automated fixes * Feat/move model into adapter fix types and cleanup (TanStack#158) * checkpoint * checkpoint * ci: apply automated fixes * unrename * move to _types pattern * Remove commented code * Goodbye embeddings * move to ~types * checkpoint * ci: apply automated fixes * updated docs and fixed a small bug with types --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Alem Tuzlak <t.zlak@hotmail.com> --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Tanner Linsley <tannerlinsley@gmail.com> * type safety tests for chat * ci: apply automated fixes * image tests * ci: apply automated fixes * update format * ci: apply automated fixes * flatten options object * ci: apply automated fixes * testing fixups * fix: streaming functions and migration doc * few more adapter/model fixups * ci: apply automated fixes * small bug fixes * ci: apply automated fixes * fix up exports and example * tighten up types * ci: apply automated fixes * tighten up types * ci: apply automated fixes * fixes * ci: apply automated fixes * updating the changeset --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Jack Herrington <jherr@pobox.com> Co-authored-by: Tanner Linsley <tannerlinsley@gmail.com>

jherr added 4 commits December 12, 2025 10:42

video generation

deb4220

text to speech and speech to text

29bec2b

adding some cool audio UI to the dev panel

6bacae8

small fixups

28f3593

ci: apply automated fixes

7add83c

jherr added 2 commits December 12, 2025 21:40

client fixes on tool calls

29292f0

Merge branch 'adding-av-adapters' of github.com:TanStack/ai into addi…

c5dc5bb

…ng-av-adapters

AlemTuzlak approved these changes Dec 13, 2025

View reviewed changes

jherr added 2 commits December 13, 2025 08:53

few more client fixups

410e1ec

one last test fix

ee99400

jherr merged commit 2b56f66 into feat/splitting-adapters-up Dec 13, 2025
6 checks passed

jherr deleted the adding-av-adapters branch December 13, 2025 17:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Adding av adapters #144

Adding av adapters #144

Uh oh!

jherr commented Dec 13, 2025

Uh oh!

coderabbitai bot commented Dec 13, 2025

Review skipped

Uh oh!

nx-cloud bot commented Dec 13, 2025 •

edited

Loading

Uh oh!

pkg-pr-new bot commented Dec 13, 2025 •

edited

Loading

Uh oh!

AlemTuzlak left a comment

Uh oh!

AlemTuzlak Dec 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Adding av adapters #144

Adding av adapters #144

Uh oh!

Conversation

jherr commented Dec 13, 2025

🎯 Changes

New Activities

🎬 Video Generation

🗣️ Text-to-Speech (TTS)

🎤 Audio Transcription

Package Changes

@tanstack/ai

@tanstack/ai-openai

@tanstack/ai-gemini

Testing Panel (testing/panel/)

Smoke Tests

Documentation

✅ Checklist

🚀 Release Impact

Uh oh!

coderabbitai bot commented Dec 13, 2025

Review skipped

Uh oh!

nx-cloud bot commented Dec 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pkg-pr-new bot commented Dec 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlemTuzlak left a comment

Choose a reason for hiding this comment

Uh oh!

AlemTuzlak Dec 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

`@tanstack/ai`

`@tanstack/ai-openai`

`@tanstack/ai-gemini`

Testing Panel (`testing/panel/`)

nx-cloud bot commented Dec 13, 2025 •

edited

Loading

pkg-pr-new bot commented Dec 13, 2025 •

edited

Loading