Bluetooth

[Krupskis]

This is work in progress changes for bluetooth client on ios and android, this will not work on web and you will need a physical device to use this.

Changes:

• Changed the sample rate to 8000, bit depth to 16, as BLE (Bluetooth Low Energy) has limited bandwidth. It should work with 16000 sample rate, but I didn't notice any difference in transcriptions.
• Changed the process-audio function to take in base64 encoded byte arrays. I couldn't get Capacitor's http client to send raw bytes. (this is a breaking change).

I left some other comments in the code for discussion

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
ad-deus	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 19, 2024 5:52am

[adamcohenhillel]

I guess this is related to the the issue you described of not being able to send the raw bytes?

Are you able to see / log the raw bytes locally on the phone?

Meaning, can you work with the bytes locally, just can't send them, or you can't even collect them as raw bytes?

[Krupskis]

You are right.

I get raw bytes from esp32, so I can get them as raw bytes and populate an ArrayBuffer, but sending just the array buffer over HTTP Capacitor client did not work, as in no request was made.

So instead I encoded that array buffer as base64 and sent it as part of a JSON.

[adamcohenhillel]

I see - so the header can still be managed by the client side, and just base64 the entire bytes of the wav file?

To make the breaking change smaller, or even not exists

You can add an if statement that checks if it is base64, do x else y

[Krupskis]

I added several if statements, so now it no longer is a breaking change, it will still work with form requests and binary bodies.

    if (contentType.includes('multipart/form-data')) {
        const form = await multiParser(req);
        if (!form || !form.files || !form.files.file) {
            return new Response('File not found in form', {
                status: 400,
                headers: corsHeaders,
            });
        }
        console.log('Form:', form);
        const file = form.files.file;
        arrayBuffer = file.content.buffer;
        filenameTimestamp = file.filename || filenameTimestamp;
    } else if (contentType.includes('application/json')) {
      const { data } = await req.json();
      const audioData = decodeBase64(data);

      console.log('Audio data:', audioData.length);
      // 1 Channel, 8000 sample rate, 16 bit depth
      const wavHeader = createWavHeader(audioData.length, 16000, 1, 16);
      const wavBytes = new Uint8Array(wavHeader.length + audioData.length);
      wavBytes.set(wavHeader, 0);
      wavBytes.set(audioData, wavHeader.length);
      arrayBuffer = wavBytes.buffer;
    } else {
        arrayBuffer = await req.arrayBuffer();
    }
    ```

[adamcohenhillel]

Adding @Jacksonmills as a reviewer to check out the frontend changes!

(also, there's a merge conflict)

[Jacksonmills]

Looking decent, I think we could definitely clean up the connect page and make it more of a fleshed out page

[Jacksonmills]

Lets make sure to use next/link Link for these. You can use asChild on Button so it renders just as the <a></a> tag and not <button><a></a></button>

e.g.

<Button
  ...
  + asChild
  - onClick={() => router.push('/')}
>
  + <Link href="/connect">
      <Bluetooth size={20} />
  + </Link>
</Button>

[Jacksonmills]

oh and then remove the useRouter if we arent needing it anymore

[Krupskis]

I think we could definitely clean up the connect page and make it more of a fleshed out page
totally agree. ESP32 is still fairly unusable - high power consumption, requires soldering a mic on it, want to push out nrf PR and then polish the connect page.

[Jacksonmills]

same as other link

[Jacksonmills]

supabaseClient! It's best practice to check for a null supabaseClient at the beginning of your component's render function and handle it by returning null. This prevents the rest of the function from executing with an invalid client, which is more efficient and makes the code easier to read and maintain.

  if (!supabaseClient) {
    return <div>Supabase client not found</div>;
  }

[Krupskis]

@Jacksonmills can you take a look again?