How to record audio and call openai's whisper recognition

[ching123]

Is anyone know: how to record audio and call openai's whisper recognition. I don't found any example.
Thanks.

[Nsk246]

Did u find the solution ? I am also looking for the same thing .

[Alek99]

we have a solution for this will try and post something tomorrow

[Nsk246]

Thanks... Waiting for it

[tgberkeley]

So we do not currently have an audio recording specific component, but we do have a webcam component: https://github.com/tgberkeley/reflex-webcam

There are instructions on how to use it in the ReadMe. For your use case I would recommend using the Advanced Usage (Video recording) section and then you will get a video.webm as an output that already has audio. You can then run some python functionality (maybe this: https://www.thepythoncode.com/article/extract-audio-from-video-in-python) to extract the audio.

This definitely isn't the optimal solution and you could wrap your own audio recording component such as https://www.npmjs.com/package/react-audio-voice-recorder, which might make things easier. You can follow the docs on how to wrap a component here: https://reflex.dev/docs/custom-components/overview/

Let us know if you have any questions about either of these approaches.

[Nsk246]

Thanks . I prefer to use the 2nd method use suggested by wrapping react components. I tried to do it but I actually can't figure out how to do it. I am new to this framework, can you help please 😅

[tgberkeley]

Hi Nsk246,
Are you on the Reflex discord, if not here is the link: https://discord.gg/T5WSbC2YtQ

If you jump on here then you can message me tomg7418 then I can help you more with it and get the team involved to help as well.

[tgberkeley]

Or feel free to book in a call with me here: https://calendly.com/tom-vkq/30min

[TimChild]

I've just been trying to do the same thing, but I can't quite figure out how the webcam example maps over to the audio recorder.

@tgberkeley, I'll see if I can reach you on discord (I'm new to it). And I'll come back to this to post an update once it's figured out!

Thanks

[TimChild]

NOTE: This is not yet an answer, but a compilation of relevant sources of information that seem related to getting to an answer. Hopefully, a full answer can be posted as a reply shortly after!

I also have basically zero javascript experience, so I may be overlooking something very obvious!

Aims:

• Implement an audio recorder that streams data back to server
• Do something with that audio, like transcribe using a third party API

Steps:

• Wrap a react component so that it can be used in reflex
• Implement event handler to handle audio processing on some event

Progress:

Wrapping react component:

Seems like the npm react-audio-voice-recorder would be a nice fit for this (not sure about streaming data, but at least for recording a full audio clip).

Related reflex documentation:

• https://reflex.dev/docs/wrapping-react/overview/
• https://reflex.dev/docs/wrapping-react/imports/
• https://reflex.dev/docs/wrapping-react/logic/

Example of react webcam component in reflex (has some similarities to audio recording)

• npm -- https://www.npmjs.com/package/react-webcam
• reflex implementation -- https://github.com/tgberkeley/reflex-webcam

The example usage of react-audio-voice-recorder on npm:

import React from "react";
import ReactDOM from "react-dom/client";
import { AudioRecorder } from 'react-audio-voice-recorder';

const addAudioElement = (blob) => {
  const url = URL.createObjectURL(blob);
  const audio = document.createElement("audio");
  audio.src = url;
  audio.controls = true;
  document.body.appendChild(audio);
};

ReactDOM.createRoot(document.getElementById("root")).render(
  <React.StrictMode>
    <AudioRecorder 
      onRecordingComplete={addAudioElement}
      audioTrackConstraints={{
        noiseSuppression: true,
        echoCancellation: true,
      }} 
      downloadOnSavePress={true}
      downloadFileExtension="webm"
    />
  </React.StrictMode>
);

Code so far:

from typing import Dict, Any
import reflex as rx
from reflex.vars import Var


class AudioRecorder(rx.Component):
    """Wrapper for react-audio-voice-recorder component."""
    
    # The React library to wrap.
    library = "react-audio-voice-recorder"
    
    # The React component tag.
    tag = "AudioRecorder"
    
    # If the tag is the default export from the module, you can set is_default = True.
    # This is normally used when components don't have curly braces around them when importing.
    is_default = False
    
    # The props of the React component.
    # Note: when Reflex compiles the component to Javascript,
    # `snake_case` property names are automatically formatted as `camelCase`.
    # The prop names may be defined in `camelCase` as well.
    
    # Show waveform while recording
    show_visualizer: Var[bool] = True
	    
    # Download audio client-side on save press
    download_on_save_press: Var[bool] = False
    
    def get_event_triggers(self) -> Dict[str, Any]:
        return {
            **super().get_event_triggers(),
            "on_recording_complete": lambda e0: [e0],
        }

audio_recorder = AudioRecorder.create

And then to use it:

class AudioState(rx.State):
    audio_recorded: str = ""
    
    def update_recorded_audio(self, audio_data: str):
        self.audio_recorded = str(audio_data)
    
def audio_component() -> rx.Component:
    return rx.fragment(
        audio_recorder.audio_recorder(
            id="audio_recorder",
            on_recording_complete=AudioState.update_recorded_audio,
			download_on_save_press=True,
        ),
        rx.cond(
            AudioState.audio_recorded,
            rx.text(AudioState.audio_recorded),
            rx.text("Click to record audio."),
        ),
    )

Problem:

This almost works... The component shows up, you can click it to start recording, and on saving, the file is downloaded client-side (so audio was recorded) and the update_recorded_audio event is triggered, however, the audio_data in the event is just {}, so I'm not sure how to actually get the audio data server-side.

Thoughts:

There are a few things I am unsure about:

• What data format I would even expect to receive the audio, and whether it is important that I get that right in the AudioState to even have a chance of receiving it (i.e. does something about types get transpiled, and maybe that is why I don't receive any data?)
• In the example on npm, the onRecordingComplete seems to call the addAudioElement function. I don't understand the relevance of what document.createElement("audio") is doing in there, but maybe it is important... My understanding is that a blob gets passed to that function when called, and I would expect to receive that same blob in my event handler (but I don't).
• In the webcam example, there is a getScreenshot method that seems to be the key to getting the data back to the server. Could an equivalent function be written in javascript for the audio component?
• I feel like I should be defining how an rx.EventHandler is called or used somehow, or maybe rx.call_script or rx.client_side?
• Or maybe the _get_custom_code method should be defined on the AudioRecorder?

I think the problem is my lack of understanding of how javascript works, so I don't understand what parts are signficant when trying to wrap it with reflex. I feel like the answer should be pretty simple in the end, and that all of the parts are there, I'm just not sure how to piece them together.

Audio processing

Once I can get the audio to the server, I think the rest is reasonable simple... Something like extending the usage to:

class AudioState(rx.State):
    audio_recorded: str = ""
    transcribed_audio: str = ""
    
    def update_recorded_audio(self, audio_data: str):
        self.audio_recorded = str(audio_data)
        return AudioState.transcribe_audio
        
    async def transcribe_audio(self):
        transcription = await external_transcription_service(self.audio_recorded)
        self.transcribed_audio = transcription
        
    async def stream_transcribe_audio(self, audio_chunk):
        transcription = await external_transcription_service(audio_chunk)
        self.transcribed_audio += transcription

    
def audio_component() -> rx.Component:
    return rx.fragment(
        audio_recorder.audio_recorder(
            id="audio_recorder",
            on_recording_complete=AudioState.update_recorded_audio,
            on_audio_chunk=AudioState.stream_transcribe_audio,  # << Ideally would actually use something like this
        ),
        rx.cond(
            AudioState.audio_recorded,
            rx.text(f'Transcription: {AudioState.transcribed_audio}'),
            rx.text("Click to record audio."),
        ),
    )

As noted, ideally I would like to be able to stream the audio and transcribe. But, I'm not sure whether it is easily achievable to add a new event like on_audio_chunk. Maybe it's just some simple javascript, or maybe it requires deep changes to the react component itself?

Closing thoughts:

I hope this provides a good starting point for discussion of what is missing, what is wrong, what needs to be changed, and what the logic behind the changes is.

[TimChild]

Possible that this discussion #2845 also demonstrates some of the necessary steps:

• demonstrating _get_custom_code
• Setting up an event trigger that takes additional arguments generated by the custom code

Still not quite sure how to apply this...

[TimChild]

Some progress...

def _on_recording_complete_signature(blob: Any) -> list[Any]:
    return [
        rx.Var.create_safe(f"extract_audio({blob})"),
    ]

clientside_audio_url_script = dedent("""\
            const extract_audio = (blob) => {
            if (!(blob instanceof Blob)) {
                console.error('Invalid argument type:', typeof blob);
                throw new TypeError('Argument must be a Blob');
            }
            console.log('Extracting audio from type:', typeof blob);
            console.log('instance of Blob:', blob instanceof Blob);
            const url = URL.createObjectURL(blob);
            return url;
            };
            """)

class AudioRecorder(rx.Component):
    """Wrapper for react-audio-voice-recorder component."""

    # The React library to wrap.
    library = "react-audio-voice-recorder"

    # The React component tag.
    tag = "AudioRecorder"

    is_default = False

    def get_event_triggers(self) -> Dict[str, Any]:
        return {
            **super().get_event_triggers(),
            "on_recording_complete": _on_recording_complete_signature,
        }

    def _get_custom_code(self) -> str:
        return clientside_audio_url_script

With this, it is possible to play recorded audio using the rx.audio component with src updated from the returned url.

However, this still leaves the audio data clientside only, the only thing that gets sent back to the server is the url from the URL.createObjectURL(blob) line...

For someone with more javascript knowledge, there may be a more obvious next step from here... I tried using a FileReader inside a Promise, but I got pretty lost so didn't manage to get that working.

Getting closer though...

[PlutarcoII]

I was curious about your code… I hope this help.

from textwrap import dedent
import reflex as rx

from typing import Dict, Any, List
import reflex as rx
from reflex.vars import Var
import base64

class AudioRecorder(rx.Component):
    """Wrapper for react-audio-voice-recorder component."""
    
    # The React library to wrap.
    library = "react-audio-voice-recorder"
    
    # The React component tag.
    tag = "AudioRecorder"

    is_default = False

    def get_event_triggers(self) -> dict[str, Any]:
        return {
            **super().get_event_triggers(),
            "on_recording_complete": _on_recording_complete_signature,
        }

audio_recorder = AudioRecorder.create

def _on_recording_complete_signature(blob: Any) :
    return [
        rx.Var.create_safe(f"extract_audio({blob})")
    ]

class AudioState(rx.State):
    
    def recording_complete_callback(self, b64_str):
        decodedData = base64.b64decode(b64_str.split(',')[1])
        webmfile = (rx.get_upload_dir() / "audio.webm")
        with open(webmfile, 'wb') as file:
            file.write(decodedData)
        
    def update_recorded_audio(self, url: any):
        return [
            rx.call_script(f"""download_blob("{url}")""", callback=AudioState.recording_complete_callback)
        ]


def index() -> rx.Component:
    return rx.center(
        rx.script("""function download_blob(url) {
                    console.log(url);
                    var xhr = new XMLHttpRequest();
                    xhr.open('GET', url, true);
                    xhr.responseType = 'blob';
                    return new Promise((resolve, reject) => {
                        xhr.onload = function(e) {
                            if (this.status == 200) {
                                var blob = this.response;
                                // console.log(blob);
                                // blob is now the blob that the object URL pointed to.
                                var reader = new window.FileReader();
                                reader.readAsDataURL(blob); 
                                reader.onloadend = function() {
                                    var base64 = reader.result;
                                    // base64 = base64.split(',')[1];
                                    // console.log(base64);
                                    resolve(base64);
                                }
                            }
                        };
                        xhr.send();
                    });
                  };
                """),
        
        rx.script("""function extract_audio(blob) {
                if (!(blob instanceof Blob)) {
                    console.error('Invalid argument type:', typeof blob);
                    throw new TypeError('Argument must be a Blob');
                }
                console.log('Extracting audio from type:', typeof blob);
                console.log('instance of Blob:', blob instanceof Blob);
                const url = URL.createObjectURL(blob);
                return url;
            };
            """),
         audio_recorder(
            id="audio_recorder",
            on_recording_complete=AudioState.update_recorded_audio,
			download_on_save_press=True,
        ),
    )    
    
app = rx.App()
app.add_page(index)

[Nsk246]

It's working. Thank you very much !!!

[TimChild]

@PlutarcoII Thanks for that! It works great! I see now that I had not realised rx.call_script could be used like that... I think that was one major piece of the puzzle, although the rest of the javascript was also pretty foreign to me. I think I understand it now though.

I feel like there should be a nicer way to keep the additional scripts required linked more directly to the component.
I would have thought that was what the _get_custom_code method was for, but it doesn't seem to work if placed there...
Or it would have been nice to use the Promise in the rx.Var.create_safe directly.

Looks like I somehow overlooked this page of the docs https://reflex.dev/docs/api-reference/browser-javascript/#external-scripts
Or maybe I just didn't realise the significance of it the first time through... I think I'll read through that again in the morning to see if I can properly understand the _get_hooks and _get_custom_code methods and whether they should apply in this situation.

@masenf I'd still be very interested to see an implementation that allows for streaming the data as you mentioned in discord.

[TimChild]

I wonder if using rx.utils.format.format_event_chain would enable putting all the script into just the rx.Component...

[TimChild]

Another very nice version implemented by @masenf here: https://pypi.org/project/reflex-audio-capture/
Also a demo of it in action here: https://audio-capture-demo.reflex.run/

This implementation can also stream and select which microphone to use etc.

[TimChild]

Looks like the example doesn't work any more. I tried using the example in the component docstring:

    ```python
    def index() -> rx.Component:
        capture = AudioRecorderPolyfill.create(
            id="my_audio_recorder",
            on_data_available=State.on_data_available,
            on_error=State.on_error,
            timeslice=State.timeslice,
        )
        return rx.vstack(
            capture,
            rx.cond(
                capture.is_recording,
                rx.button("Stop Recording", on_click=capture.stop),
                rx.button("Start Recording", on_click=capture.start),
            ),
        )
    ```

With an rx.State defined like this:

class State(rx.State):
    timeslice: int = 0

    # on_data_available
    @rx.event()
    def on_data_available(self, data):
        print(data)

    @rx.event()
    def on_error(self, error):
        print(error)

But I got this error at build time:

reflex.utils.exceptions.EventFnArgMismatch: Event on_click only provides 0 arguments, but <bound method AudioRecorderPolyfill.stop of {}> requires at least 1 arguments to be passed to the event handler.
See https://reflex.dev/docs/events/event-arguments/

Quite a bit has changed in reflex recently (particularly with the change to 0.6.0+), so that's likely why... I don't understand well enough how it should be adjusted for new reflex versions. Hopefully @masenf has some time to answer soon.

[masenf]

The example is working for me with 0.6.5.

1. create the component: https://github.com/masenf/reflex-audio-capture/blob/47c032d393a96bfc34a7ac6d2aba3027c932c063/audio_capture_demo/audio_capture_demo/audio_capture_demo.py#L61-L68
2. use it as a component (don't call it again): https://github.com/masenf/reflex-audio-capture/blob/47c032d393a96bfc34a7ac6d2aba3027c932c063/audio_capture_demo/audio_capture_demo/audio_capture_demo.py#L106
3. @TimChild you need to use capture.start() and capture.stop() with parentheses https://github.com/masenf/reflex-audio-capture/blob/47c032d393a96bfc34a7ac6d2aba3027c932c063/audio_capture_demo/audio_capture_demo/audio_capture_demo.py#L110-L114

[david-thrower]

@masenf , @TimChild , thank you for the quick reply. The Reflex community is amazing.

[TimChild]

Thanks @masenf , silly mistake on my end!
@david-thrower , hopefully next time my response will be a bit more helpful!

[TimChild]

Just realised I also had an outdated version of reflex-audio-capture -- updating to 0.0.4 fixed some additional issues I had with the example above...