Jarvis in JavaScript

This is a fun side project to build a custom voice assistant in JavaScript that sounds like Jarvis from Iron Man.

It's been tested on Mac and Raspberry Pi and should run on pretty much any platform if installed properly. It uses the latest OpenAI Realtime speech-to-speech API via Node.js + WebSocket, and uses OpenWakeWord and Python for wake word detection.

OpenAI has no JavaScript SDK for this, so I thought it might be helpful to show how simple it is for JavaScript developers to build this without an SDK. Rather than use an NPM module for speaker/microphone, I found it far better to use sox via child process and stdin/stdout.

Let's connect! Reach me at gaberogan.com.

Demo

Disclaimer

This is a fun side project not built for production. You may encounter several bugs.

You may have to troubleshoot if you decide to follow the README. Using an AI editor like Cline or Cursor can help a lot with troubleshooting.

Quickstart

1. Create .env with OPENAI_API_KEY and (optional) GOOGLE_API_KEY for Google Search
2. Install sox i.e. brew install sox
3. Install NVM + Node.js 22 (other versions untested)
4. Install pyenv + Python 3.12 (other versions untested)
5. Run pip install -r requirements.txt
6. Run python scripts/download_models.py
7. If using Raspberry Pi, download missing audio libraries
8. Run npm i
9. Run npm start
10. Say "Hey Jarvis, how are you?"
11. (optional) For more wake words, see https://github.com/fwartner/home-assistant-wakewords-collection

Todo

1. Clean up Quickstart steps
2. Implement a memory tool to save things like location to memory
3. Use "Jarvis" wake word instead, may need TFLite
4. Fix AEC with PulseAudio on Raspberry Pi