Video Analyzer

Inspired by Tracking Football Players with YOLOv5 + ByteTrack, we try to implement a video analyzer some with web-based techniques.

The following techniques are used for now:

• WebCodecs Decode and encode audio & video frames.
• Web Audio API
• WebGPU Render VideoFrame and other HUDs on OffscreenCanvas off main thread.
• tensorflow.js Run pre-trained models with WebGPU backend in webworker.
• Some UIs come from:
  • react-timeline-editor
  • Player

Getting started

Start a vite dev server.

$ pnpm install
$ pnpm dev

WebCodecs

With WebCodecs it's possible to decode and encode videos frame & audios using hardware acceleration. It is preferable to move handling of individual frames and encoded chunks into a web worker.

FFmpeg is used for muxing/demuxing videos and for some codecs that are not supported by WebCodecs. Since we only plan to support .mp4, MP4Box.js is suitable to implement a pull-based demuxer.

For audio & video tracks in .mp4:

• AudioDecoder decodes chunks of audio from MP4AudioTrack and store these chunks in a RingBuffer, which will be used for playing with Web Audio API later.
• VideoDecoder can output VideoFrame, which is transferable and can be easily handled with Canvas2D / WebGL / WebGPU API later.

Web Audio API

An AudioContext controls both the creation of the nodes it contains and the execution of the audio processing, or decoding. It has a member AudioWorklet which is used to supply custom audio processing scripts that execute in a separate thread to provide very low latency audio processing.

First, we need to define a custom AudioWorkletProcessor named AutoSink. Next, in our main script file we'll load the processor, create an instance of AudioWorkletNode passing it the name of the processor, and connect the node to an audio graph. It will consume the RingBuffer stored in the previous step.

WebGPU

With HTMLCanvasElement.transferControlToOffscreen even rendering can be done off the main thread.

https://developer.chrome.com/blog/from-webgl-to-webgpu/#video-frame-processing

Processing video frames using JavaScript and WebAssembly has some drawbacks: the cost of copying the data from GPU memory to CPU memory, and the limited parallelism that can be achieved with workers and CPU threads. WebGPU does not have those limitations, making it a great fit for processing video frames thanks to its tight integration with the WebCodecs API.

tensorflow.js

Since Web Neural Network API (WebNN) is in development, tf.js is our only choice in browser for now. By the way, its polyfill is based on tf.js either.

We use WebGPU backend in WebWorker to run pre-trained models including:

• YOLO for object detection.
• ByteTrack for object tracking.

Inspired by:

• Serving YOLOv8 in browser using tensorflow.js with webgl backend.
• Track an object as it moves in a video with no training.