Speaker hint support

[yujonglee]

No description provided.

[coderabbitai]

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Walkthrough

Adds speaker hints functionality to track per-word speaker information across the system. Updates include database schema with new speaker_hints table, state management integration via TinyBase and Zustand for persistence, a new SegmentHeader component for visualization, and utilities for converting speaker hints between storage and runtime formats.

Changes

Cohort / File(s)	Summary
Database Migration & Schema packages/db/drizzle/0001_speaker_hints.sql, packages/db/drizzle/meta/0001_snapshot.json, packages/db/drizzle/meta/_journal.json	Creates new speaker_hints table with foreign keys to profiles, transcripts, and words; adds RLS policies; updates drizzle metadata snapshot and journal.
Database Schema Definition packages/db/src/schema.ts	Adds speakerHints pgTable, speakerHintSchema, and new providerSpeakerIndexSchema with corresponding type exports.
State Management - TinyBase apps/desktop/src/store/tinybase/main.ts, apps/desktop/src/store/tinybase/schema-external.ts	Introduces speakerHintToTranscript and speakerHintToWord relationships, adds speakerHintsByTranscript and speakerHintsByWord indexes, exports SpeakerHint and SpeakerHintStorage types.
State Management - Zustand apps/desktop/src/store/zustand/listener/transcript.ts, apps/desktop/src/store/zustand/listener/transcript.test.ts	Updates HandlePersistCallback signature to include hints parameter, adds partialHints field to TranscriptState, extends transformWords to return both words and hints, adds fixSpacingForWords helper function.
Task Configuration apps/desktop/src/store/zustand/ai-task/task-configs/enhance.ts	Extends getTranscriptSegments to collect speaker hints, maps word IDs to indices, converts storage hints to runtime format.
Utilities - Segment Building apps/desktop/src/utils/segment.ts, apps/desktop/src/utils/segment.test.ts	Adds SpeakerHintData and RuntimeSpeakerHint types, updates buildSegments and createSpeakerTurns to accept optional speakerHints parameter for speaker-based segmentation.
Utilities - Speaker Hints Conversion apps/desktop/src/utils/speaker-hints.ts	Introduces parseProviderSpeakerIndex and convertStorageHintsToRuntime functions for parsing and transforming speaker hints.
Hooks apps/desktop/src/hooks/useStartListening.ts	Extends handlePersist to accept hints parameter, persists speaker hints with word IDs and provider speaker indices when STT provider is "deepgram".
AudioPlayer Context apps/desktop/src/contexts/audio-player/provider.tsx	Adds sessionId prop to AudioPlayerProvider, fetches audio existence via useQuery, exposes audioExists in context value.
Transcript Components apps/desktop/src/components/main/body/sessions/index.tsx, apps/desktop/src/components/main/body/sessions/note-input/transcript/viewer/index.tsx	Updates AudioPlayer.Provider usage with sessionId, refactors transcript viewer to use partialHints, replaces audioExist query check with hints flow, adds seek-and-play on word click.
New Component apps/desktop/src/components/main/body/sessions/note-input/transcript/viewer/segment-header.tsx	Introduces SegmentHeader component with sticky header displaying colored speaker label and timestamp range using chroma-js for color generation.
Dependencies apps/desktop/package.json	Adds @hypr/codemirror (workspace:^), chroma-js (^3.1.2), and @types/chroma-js (^3.1.2).
Rust Template crates/template/Cargo.toml, crates/template/src/filters.rs, crates/template/src/lib.ts	Adds htmd and reqwest dependencies; introduces new url filter function that fetches and converts HTML to Markdown.
Web Configuration apps/web/netlify.toml	Updates image remote pattern from glob to regex format for broader remote image matching.

Sequence Diagram(s)

sequenceDiagram
    participant STT as STT Provider (Deepgram)
    participant useStartListening as useStartListening Hook
    participant transformWords as transformWords
    participant handlePersist as handlePersist
    participant DB as Database

    STT->>useStartListening: words + speaker_index data
    useStartListening->>transformWords: alternative with speaker hints
    transformWords-->>transformWords: Build WordLike[] + RuntimeSpeakerHint[]
    useStartListening->>handlePersist: (words, hints)
    handlePersist->>DB: Persist words with word_id
    handlePersist->>DB: Persist speaker_hints with word_id + provider_speaker_index
    DB-->>handlePersist: Success
    handlePersist-->>useStartListening: Complete

Loading

sequenceDiagram
    participant Store as Zustand Store
    participant buildSegments as buildSegments
    participant createSpeakerTurns as createSpeakerTurns
    participant UI as Segment UI

    Store->>buildSegments: finalWords + partialHints
    buildSegments->>createSpeakerTurns: words + speakerHints
    createSpeakerTurns-->>createSpeakerTurns: Map word_id to speaker_index<br/>Create ChannelSpeaker keys
    createSpeakerTurns-->>buildSegments: Segments grouped by speaker
    buildSegments-->>Store: Segments
    Store->>UI: Render SegmentHeader + words
    UI-->>UI: Color label by speaker<br/>Display timestamp range

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Areas requiring extra attention:

• Speaker hints persistence logic (apps/desktop/src/store/zustand/listener/transcript.ts): Verify correct mapping of word indices to speaker hints, ensure hints are properly indexed and filtered during partial/final word transitions
• Segment building with speaker hints (apps/desktop/src/utils/segment.ts): Review speaker index extraction, ChannelSpeaker key creation, and speaker turn boundary detection
• AudioPlayer provider changes (apps/desktop/src/contexts/audio-player/provider.tsx): Validate audio existence query integration with sessionId, ensure proper context propagation
• Database migration consistency (packages/db/drizzle/0001_speaker_hints.sql, packages/db/src/schema.ts): Confirm RLS policies, foreign key constraints, and schema alignment
• State management integration (apps/desktop/src/store/zustand/ai-task/task-configs/enhance.ts, apps/desktop/src/store/tinybase/main.ts): Verify TinyBase relationships and index definitions work correctly with persisted hints

Possibly related PRs

• Audio Player initial design #1563: Modifies AudioPlayer provider API and its integration in sessions component, directly related to the sessionId and audioExists context changes in this PR.
• transcript-editor-ui #850: Updates TinyBase integration; this PR adds speaker hints schema, relationships, and indexes to TinyBase that may require coordination.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch speaker-hint-support

---

📜 Recent review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 4da9398 and b367e08.

⛔ Files ignored due to path filters (2)

• Cargo.lock is excluded by !**/*.lock
• pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml

📒 Files selected for processing (22)

• apps/desktop/package.json (3 hunks)
• apps/desktop/src/components/main/body/sessions/index.tsx (1 hunks)
• apps/desktop/src/components/main/body/sessions/note-input/transcript/viewer/index.tsx (10 hunks)
• apps/desktop/src/components/main/body/sessions/note-input/transcript/viewer/segment-header.tsx (1 hunks)
• apps/desktop/src/contexts/audio-player/provider.tsx (5 hunks)
• apps/desktop/src/hooks/useStartListening.ts (3 hunks)
• apps/desktop/src/store/tinybase/main.ts (4 hunks)
• apps/desktop/src/store/tinybase/schema-external.ts (5 hunks)
• apps/desktop/src/store/zustand/ai-task/task-configs/enhance.ts (4 hunks)
• apps/desktop/src/store/zustand/listener/transcript.test.ts (1 hunks)
• apps/desktop/src/store/zustand/listener/transcript.ts (5 hunks)
• apps/desktop/src/utils/segment.test.ts (2 hunks)
• apps/desktop/src/utils/segment.ts (4 hunks)
• apps/desktop/src/utils/speaker-hints.ts (1 hunks)
• apps/web/netlify.toml (1 hunks)
• crates/template/Cargo.toml (1 hunks)
• crates/template/src/filters.rs (2 hunks)
• crates/template/src/lib.rs (1 hunks)
• packages/db/drizzle/0001_speaker_hints.sql (1 hunks)
• packages/db/drizzle/meta/0001_snapshot.json (1 hunks)
• packages/db/drizzle/meta/_journal.json (1 hunks)
• packages/db/src/schema.ts (3 hunks)

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}