Live Image Processing with getUserMedia() and Web Workers

Demo of our internal tools using Web Workers & getUserMedia() to create image effects.

At Aviary, we are constantly exploring new technologies including the latest (and not-yet-fully supported) HTML5 features. Over the past few months, we’ve begun using Web Workers for a number of our internal tools. Web Workers allow us to perform heavy image processing tasks as a background process to avoid freezing the UI. We hope to be able to roll these benefits into our product in the near future while providing a fallback to unsupported browsers.

getUserMedia() is another exciting new feature in HTML5. It allows web applications to access video and audio from a user’s camera and other media device. In this demo, I'm piping webcam video data into a canvas element. I’ve also built a UI that allows me to control Aviary’s JavaScript Image Processors in real-time.

Since image processing is CPU intensive and can freeze our UI, we are also going to leverage the aforementioned Web Workers to perform the pixel manipulation as a background process. As I mentioned before, this will prevent the UI from freezing while the image is processing. No more need for the setTimeout( …, 0) trick.

Here’s how to get the video stream:

#!javascript // create video element (attach to DOM if you’d like to // view the stream but not necessary here) var video = document.createElement('video');

// default to vendor prefixed method for getUserMedia navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia || navigator.mozGetUserMedia || navigator.msGetUserMedia;

// acquire the video stream navigator.getUserMedia({video:true}, function(stream){ video.src = URL.createObjectURL(stream);;

// setup interval to getImageData every X seconds: setInterval(render, 10); }, function(error){ console.log('error', error); });

In our render() method we first draw the video stream onto a canvas and use getImageData() for the pixels. Then we pass the pixels to our Web Worker for processing. (See example code link at bottom for more detail)

#!javascript var render = function(){ ctx.drawImage(video, 0, 0, w, h); var srcData = ctx.getImageData(0,0,w,h);

// pass image data to web worker processor.postMessage({ imageData: srcData }); };

After that, we use Web Workers to increase the Red channel, then use postMessage() to pass the resulting pixel data back.

#!javascript // message receiver onmessage = function(event) { var imageData =, dst =;

/* Image Processing will go here */ for (var i=0; i < dst.length; i += 4) { dst[i] += 70; // increase red channel }

postMessage({ dstData: imageData // pass result back }); };

Once the Worker is finished we will listen for the result and draw it onto our canvas:

#!javascript processor.onmessage = function(event){ ctxEffects.putImageData(, 0, 0); };

Check out the full example code and a working demo below.

Demo: (you'll need the latest Chrome or Firefox)