ImageProcessing

Processing images from the webcam is the cornerstone of SlouchyJS

Webcam Set-Up (Classic Way)

1. Class constructor

// webcam.js
class WebCam {
  constructor(videoElmt, canvas) {
    this.videoElmt = videoElmt; // html video tag
    this.canvas = canvas; // hidden canvas
    this.hasWebcamAccess = false;
  }
...

source code

2. Stream from the webcam

getMediaStream() {
    navigator.getUserMedia = ( navigator.getUserMedia ||
                     navigator.webkitGetUserMedia ||
                     navigator.mozGetUserMedia ||
                     navigator.msGetUserMedia);

    return window.navigator.mediaDevices.getUserMedia({video: true})
     .then(function(mediaStream)
     {
        this.hasWebcamAccess = true;

        // transmitting footage from the webcam to the html video
        this.videoElmt.srcObject = mediaStream;

        this.mediaStreamTrack = mediaStream.getVideoTracks()[0];
        // Allow us to extract image from the video stream
        this.imageCapture = new ImageCapture(this.mediaStreamTrack);

     }.bind(this))
...

source code

3. Taking picture

We use the ImageCapture Object from above but only Chrome seems to fully support it

// webcam.js
takePicture(img_size) {
    return this.imageCapture.grabFrame().then(imageBitmap => ...

source code

We need to convert the ImageBitmap to a matrix of pixels to then use tensorflow

4. Conversion (tricky way)

The simple way I found was to use a canvas

// webcam.js
ctx.drawImage(imageBitmap, 0,0, img_size, img_size);
let x = [];

for(let i = 0; i < img_size; i++){
 x[i] = [];
 for(let j = 0; j < img_size; j++){
   let pixel = ctx.getImageData(j,i,1,1).data;

   // rgb is inverted for Keras pre trained model ?
   x[i][j] = [pixel[2]/255.0, pixel[1]/255.0, pixel[0]/255.0];
 }
}
return x;

source code

• Conlusion

It works fine on Chrome but requires a hidden canvas on the html page.

The Tensorflow Way

Inspired by tfjs-example.

• Capturing images

capture() {
    return tf.tidy(() => {
      // Reads the image as a Tensor from the webcam <video> element.
      const webcamImage = tf.fromPixels(this.webcamElement);

      // Crop the image so we're using the center square of the rectangular
      // webcam.
      const croppedImage = this.cropImage(webcamImage);

      // Expand the outer most dimension so we have a batch size of 1.
      const batchedImage = croppedImage.expandDims(0);

      // Normalize the image between -1 and 1. The image comes in between 0-255,
      // so we divide by 127 and subtract 1.
      return batchedImage.toFloat().div(tf.scalar(127)).sub(tf.scalar(1));
    });
}

Other in-browser deep learning applications like tfjs-yolo-tiny tend to prefer the tensorflow way to capture image from the webcam since it is straightforward and works on many modern browsers.

However, Tensorflow JS being in an early stage of its development maybe we should keep using ImageCapture while waiting for a major version.