This project is an experiment for compositing user-provided content and syncing it with live-action video footage in the browser. It was used in a viral marketing site for Mad Decent's 2014 Holiday album for an Elf Yourself inspired parody.
The Ember app that is in this repo is a stripped down version of the site that ran with Mad Decent, with 4 of the videos from the original campaign for demonstration purposes. The app does run in a mobile browser with decently performing playback. Although, the site is currently not responsive as certain parts of the process relies on precise pixel sizes, however I'm working on improving that.
I also plan to continue exploring how <canvas> and in-browser javascript can be used to combine user-provided content and video to create interactive (and hopefully in the future more immersive) experiences.
Demo: http://convan.me/decent-video/
- User selects a headshot and crops/rotates their photo within a head-shaped mask
- An image Blob is generated in-browser and then uploaded to Cloudinary (so I can generate a shareable URL)
- User can watch a video that super-imposes their headshot on top of a Mad Decent Twerkin' dancer
The crux of this project is the decent-player component, which combines pre-calcuated head-tracking data, a png image sequence (more on that later), an audio track, and the user's headshot to create a simple video player. It is a somewhat complex process but can be broken down into a few parts:
There is a frames.json file for each scene in the public/scenes/ folder. Each frames.json file contains an array of frame objects that specify the scale, position and rotation for our "head" image we will be overlaying on each frame.
These files were generated by copying and pasting tracking data from After Effects, and then formatted by hand.
Each scene is exported as a PNG sequence, each PNG frame ran through tinyPNG, then all bundled up inside of a single uncompressed zip file for each scene. The browser then downloads the zip file, extracts it using zip.js (gildas-lormeau/zip.js) which gives us a Blob reader that we can use to read out data-uris for our image frames in order. The decent-player component renders out these image frames into the <canvas> element.
Why an image sequence? Because the "other" method of using a <video> element and drawing each frame from the <video> on to the canvas results in mixed-performance on mobile and there is no way to guarantee frame-accurate animations, as you can not determine what frame number a <video> is on, you can only "guess" by running a clock and doing the math. Also, PNGs allow for transparency and we can easily draw the video frames on-top of our headshot, which results in a cleaner cut-out.
Why a zip file? Why uncompressed? Fewer network requests. Uncompressing a compressed zip file is very taxing on a browser, would take too much time to uncompress, and since the PNGs have already been run through tinyPNG, zip compression didn't result in any smaller of a file.
Super-imposing the overlay image now is rather simple - we can simply draw it out on the <canvas> using the tracking data from the JSON file to position. We also need to make sure the positioning data is properly transformed to consider the current scale of the <canvas> compared to the scale that the JSON positioning data is based on.
Also, technically I am underlaying the image - it is being drawn onto the <canvas> before the video frame, so that we don't have to worry about cleanly masking out the headshot image.
- This project needs tests
- The project that launched with Mad Decent's XMAS campaign included a server-side
Node.jsprocess that generated on-demand image previews that was used for social sharing and also generated H.264 MP4 video files of their custom video that users could download. This process ran in an on-demand worker queue on Heroku (only costing roughly $0.0005 per video!) and used most of the same code that is currently in thedecent-videocomponent. I also plan on eventually releasing portions of that code (however it's a complete mess right now).
You will need the following things properly installed on your computer.
git clone <repository-url>this repository- change into the new directory
npm installbower install
ember server- Visit your app at http://localhost:4200.
Make use of the many generators for code, try ember help generate for more details
ember testember test --server
ember build(development)ember build --environment production(production)
Specify what it takes to deploy your app.