Skip to content

Mixed audio #1471

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yujonglee merged 7 commits into from
Sep 17, 2025
Merged

Mixed audio #1471

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
ab83ce2
add headphone info for macos input device
yujonglee Sep 15, 2025
e7dd87b
add mixed source
yujonglee Sep 15, 2025
f236e3d
wip
yujonglee Sep 15, 2025
a24cec6
more progress
yujonglee Sep 16, 2025
f16ae61
workaround
yujonglee Sep 16, 2025
b317639
fix
yujonglee Sep 16, 2025
d4ef0ac
fix compile error
yujonglee Sep 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 5 additions & 22 deletions crates/audio/src/device_monitor.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ use std::thread::JoinHandle;

#[derive(Debug, Clone)]
pub enum DeviceEvent {
DefaultInputChanged { headphone: bool },
DefaultInputChanged,
DefaultOutputChanged { headphone: bool },
}

Expand Down Expand Up @@ -63,7 +63,8 @@ impl DeviceMonitor {
#[cfg(target_os = "macos")]
mod macos {
use super::*;
use cidre::{core_audio as ca, io, ns, os};
use crate::utils::macos::is_headphone_from_default_output_device;
use cidre::{core_audio as ca, ns, os};

extern "C-unwind" fn listener(
_obj_id: ca::Obj,
Expand All @@ -77,11 +78,10 @@ mod macos {
for addr in addresses {
match addr.selector {
ca::PropSelector::HW_DEFAULT_INPUT_DEVICE => {
let headphone = detect_headphones(ca::System::default_input_device().ok());
let _ = event_tx.send(DeviceEvent::DefaultInputChanged { headphone });
let _ = event_tx.send(DeviceEvent::DefaultInputChanged);
}
ca::PropSelector::HW_DEFAULT_OUTPUT_DEVICE => {
let headphone = detect_headphones(ca::System::default_output_device().ok());
let headphone = is_headphone_from_default_output_device();
let _ = event_tx.send(DeviceEvent::DefaultOutputChanged { headphone });
}
_ => {}
Expand All @@ -90,23 +90,6 @@ mod macos {
os::Status::NO_ERR
}

fn detect_headphones(device: Option<ca::Device>) -> bool {
match device {
Some(device) => match device.streams() {
Ok(streams) => streams.iter().any(|s| {
if let Ok(term_type) = s.terminal_type() {
term_type.0 == io::audio::output_term::HEADPHONES
|| term_type == ca::StreamTerminalType::HEADPHONES
} else {
false
}
}),
Err(_) => false,
},
None => false,
}
}

pub(super) fn monitor(event_tx: mpsc::Sender<DeviceEvent>, stop_rx: mpsc::Receiver<()>) {
let selectors = [
ca::PropSelector::HW_DEFAULT_INPUT_DEVICE,
Expand Down
87 changes: 80 additions & 7 deletions crates/audio/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,20 @@
mod device_monitor;
mod errors;
mod mic;
mod mixed;
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

🧩 Analysis chain

Cross‑platform MixedInput availability: verify non‑mac builds compile

MixedInput is referenced in the public struct and re‑exported unconditionally. Please confirm mixed defines a cross‑platform symbol (stub or cfg‑guarded module exposing MixedInput/MixedStream) so Linux/Windows builds don’t break.

Run to check symbol presence and cfg guards:

Also applies to: 13-13, 77-77

🏁 Script executed:

#!/bin/bash
# Expect: at least one unconditional `pub struct MixedInput` (or a re-export)
# or lib.rs fields gated accordingly; otherwise non-mac will fail.
rg -nP '^\s*(pub\s+)?struct\s+MixedInput\b|type\s+MixedInput\b|pub\s+use\s+.*MixedInput' -C2
rg -nP 'cfg\s*\(\s*target_os\s*=\s*"macos"\s*\)' crates/audio/src/mixed -n
# Verify lib.rs doesn't reference MixedInput behind cfg on non-mac
rg -nP 'MixedInput' crates/audio/src/lib.rs -n -C2

Length of output: 783

Unconditional MixedInput reference breaks non‑mac builds — fix required

mixed/macos.rs defines pub struct MixedInput (crates/audio/src/mixed/macos.rs:17) and crates/audio/src/mixed/mod.rs is #[cfg(target_os = "macos")] (crates/audio/src/mixed/mod.rs:1), but lib.rs references MixedInput unguarded (crates/audio/src/lib.rs:75–77: mixed: Option<MixedInput>). This will fail on Linux/Windows.

Actionable fixes (choose one):

  • Gate the module/uses: #[cfg(target_os = "macos")] mod mixed; and #[cfg(target_os = "macos")] on the field/funcs that use MixedInput; or
  • Provide a non‑mac stub/re‑export in mixed/mod.rs (e.g. #[cfg(not(target_os = "macos"))] pub struct MixedInput; or a no‑op reexport) so MixedInput exists cross‑platform.
🤖 Prompt for AI Agents 
In crates/audio/src/lib.rs (mod declaration at line 4 and usages around lines
75–77) the code references MixedInput unconditionally even though mixed/mod.rs
is only compiled on macOS, which breaks non‑mac builds; fix by either gating the
module and all declarations/uses with #[cfg(target_os = "macos")] (e.g.
#[cfg(target_os = "macos")] mod mixed; and #[cfg(target_os = "macos")] on the
MixedInput field and any functions that touch it) or by adding a non‑mac
stub/reexport in mixed/mod.rs (e.g. behind #[cfg(not(target_os = "macos"))]
declare a pub struct MixedInput; or a no‑op reexport) so MixedInput is defined
on all platforms; pick one approach and make the corresponding cfg changes
consistently for the module, field, and any functions that reference MixedInput.

mod norm;
mod resampler;
mod speaker;
mod utils;

pub use device_monitor::*;
pub use errors::*;
pub use mic::*;
pub use mixed::*;
pub use norm::*;
pub use resampler::*;
pub use speaker::*;
pub use utils::*;

pub use cpal;
use cpal::traits::{DeviceTrait, HostTrait};
Expand Down Expand Up @@ -62,21 +66,42 @@ impl AudioOutput {
pub enum AudioSource {
RealtimeMic,
RealtimeSpeaker,
RealtimeMixed,
Recorded,
}

pub struct AudioInput {
source: AudioSource,
mic: Option<MicInput>,
speaker: Option<SpeakerInput>,
mixed: Option<MixedInput>,
data: Option<Vec<u8>>,
}

impl AudioInput {
pub fn get_default_mic_device_name() -> String {
let host = cpal::default_host();
let device = host.default_input_device().unwrap();
device.name().unwrap_or("Unknown Microphone".to_string())
pub fn get_default_mic_name() -> String {
let name = {
let host = cpal::default_host();
let device = host.default_input_device().unwrap();
device.name().unwrap_or("Unknown Microphone".to_string())
};

name
}
Comment on lines +82 to +90
Copy link
Contributor

@coderabbitai coderabbitai bot Sep 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

🧩 Analysis chain

Potential panic when no default input device.

default_input_device().unwrap() will panic on systems without an input device; verify this is acceptable for the environments you target.

🏁 Script executed:

#!/bin/bash
# Locate all call sites of get_default_mic_name to ensure it's not used in headless/CI paths.
rg -nP '\bget_default_mic_name\s*\(' --type=rs

Length of output: 26

🏁 Script executed:

#!/bin/bash
set -euo pipefail
echo "Rust file count:"
find . -type f -name '*.rs' | wc -l
echo
echo "Searching for occurrences of get_default_mic_name..."
rg -nP '\bget_default_mic_name\s*\(' -S -g '**/*.rs' || true
rg -nP '\bget_default_mic_name\b' -S -g '**/*.rs' || true
echo
echo "Show crates/audio/src/lib.rs (lines 1-200) if exists:"
if [ -f crates/audio/src/lib.rs ]; then
  sed -n '1,200p' crates/audio/src/lib.rs
else
  echo "crates/audio/src/lib.rs not found"
fi

Length of output: 6061

Avoid panic: handle missing default input device in get_default_mic_name

default_input_device().unwrap() will panic if no input device; change get_default_mic_name (crates/audio/src/lib.rs, ~lines 82–90) to avoid unwrap — return Option or use .default_input_device().and_then(|d| d.name().ok()).unwrap_or_else(|| "Unknown Microphone".to_string()). Call sites to review: plugins/listener/src/actors/source.rs:69, plugins/listener/src/actors/source.rs:78.

🤖 Prompt for AI Agents 
In crates/audio/src/lib.rs around lines 82–90, get_default_mic_name currently
calls default_input_device().unwrap() which will panic when no input device;
change it to avoid unwrap by either (A) changing the signature to pub fn
get_default_mic_name() -> Option<String> and return
host.default_input_device().and_then(|d| d.name().ok()), or (B) keep returning
String but use host.default_input_device().and_then(|d|
d.name().ok()).unwrap_or_else(|| "Unknown Microphone".to_string()) so it never
panics. Update call sites in plugins/listener/src/actors/source.rs (around lines
69 and 78) to handle the new Option<String> if you choose (A), or leave as-is if
you choose (B); ensure all .unwrap() usages are removed and replace with safe
handling.


pub fn is_using_headphone() -> bool {
let headphone = {
#[cfg(target_os = "macos")]
{
utils::macos::is_headphone_from_default_output_device()
}
#[cfg(not(target_os = "macos"))]
{
false
}
};

headphone
}

pub fn list_mic_devices() -> Vec<String> {
Expand All @@ -101,6 +126,7 @@ impl AudioInput {
source: AudioSource::RealtimeMic,
mic: Some(mic),
speaker: None,
mixed: None,
data: None,
})
}
Expand All @@ -110,24 +136,40 @@ impl AudioInput {
source: AudioSource::RealtimeSpeaker,
mic: None,
speaker: Some(SpeakerInput::new().unwrap()),
mixed: None,
data: None,
}
}

#[cfg(target_os = "macos")]
pub fn from_mixed() -> Result<Self, crate::Error> {
let mixed = MixedInput::new().unwrap();

Ok(Self {
source: AudioSource::RealtimeMixed,
mic: None,
speaker: None,
mixed: Some(mixed),
data: None,
})
}

pub fn from_recording(data: Vec<u8>) -> Self {
Self {
source: AudioSource::Recorded,
mic: None,
speaker: None,
mixed: None,
data: Some(data),
}
}

pub fn device_name(&self) -> String {
match &self.source {
AudioSource::RealtimeMic => self.mic.as_ref().unwrap().device_name(),
AudioSource::RealtimeSpeaker => "TODO".to_string(),
AudioSource::Recorded => "TODO".to_string(),
AudioSource::RealtimeSpeaker => "RealtimeSpeaker".to_string(),
AudioSource::RealtimeMixed => "Mixed Audio".to_string(),
AudioSource::Recorded => "Recorded".to_string(),
}
}

Expand All @@ -139,6 +181,9 @@ impl AudioInput {
AudioSource::RealtimeSpeaker => AudioStream::RealtimeSpeaker {
speaker: self.speaker.take().unwrap().stream().unwrap(),
},
AudioSource::RealtimeMixed => AudioStream::RealtimeMixed {
mixed: self.mixed.take().unwrap().stream().unwrap(),
},
AudioSource::Recorded => AudioStream::Recorded {
data: self.data.as_ref().unwrap().clone(),
position: 0,
Expand All @@ -150,6 +195,7 @@ impl AudioInput {
pub enum AudioStream {
RealtimeMic { mic: MicStream },
RealtimeSpeaker { speaker: SpeakerStream },
RealtimeMixed { mixed: MixedStream },
Recorded { data: Vec<u8>, position: usize },
}

Expand All @@ -166,7 +212,7 @@ impl Stream for AudioStream {
match &mut *self {
AudioStream::RealtimeMic { mic } => mic.poll_next_unpin(cx),
AudioStream::RealtimeSpeaker { speaker } => speaker.poll_next_unpin(cx),
// assume pcm_s16le, without WAV header
AudioStream::RealtimeMixed { mixed } => mixed.poll_next_unpin(cx),
AudioStream::Recorded { data, position } => {
if *position + 2 <= data.len() {
let bytes = [data[*position], data[*position + 1]];
Expand All @@ -192,7 +238,34 @@ impl kalosm_sound::AsyncSource for AudioStream {
match self {
AudioStream::RealtimeMic { mic } => mic.sample_rate(),
AudioStream::RealtimeSpeaker { speaker } => speaker.sample_rate(),
AudioStream::RealtimeMixed { mixed } => mixed.sample_rate(),
AudioStream::Recorded { .. } => 16000,
}
}
}

#[cfg(test)]
pub(crate) fn play_sine_for_sec(seconds: u64) -> std::thread::JoinHandle<()> {
use rodio::{
cpal::SampleRate,
source::{Function::Sine, SignalGenerator, Source},
OutputStream,
};
use std::{
thread::{sleep, spawn},
time::Duration,
};

spawn(move || {
let (_stream, stream_handle) = OutputStream::try_default().unwrap();
let source = SignalGenerator::new(SampleRate(44100), 440.0, Sine);

let source = source
.convert_samples()
.take_duration(Duration::from_secs(seconds))
.amplify(0.01);

stream_handle.play_raw(source).unwrap();
sleep(Duration::from_secs(seconds));
})
}
Loading
Loading