Displaying multiple frames of a video element simultaneously

Question

Some background: graphics newbie here, have just dipped my toes into the world of 3D in the browser with mrdoob's excellent three.js. I intend to go through all the tuts at http://learningwebgl.com/ soon :)

I'd like to know how one would roughly go about re-creating something similar to: http://yooouuutuuube.com/v/?width=192&height=120&vm=29755443&flux=0&direction=rand

My naive understanding of how yooouuutuuube works is as follows:

Create a massive BitmapData (larger than any reasonable browser window size).
Determine the number of required rows / columns (across the entire BitmapData plane, not just the visible area) based on the width/height of the target video frame
Copy pixels from the most recent video frame to a position on the BitmapData (based on the direction of movement)
Iterate through every cell in the BitmapData, copying pixels from the cell that precedes it
Scroll the entire BitmapData in the opposite direction to create the illusion of movement, with a Zoetrope-type effect

I'd like to do this in WebGL as opposed to using Canvas so I can take advantage of post-processing using shaders (noise and color channel separation to mimic chromatic aberration).

Here's a screenshot what I have so far:

enter image description here

Three videos (same video, but separated into R, G and B channels) are drawn to a canvas 2D context. Each video is slightly offset in order to fake that chromatic aberration look.
A texture is created in Three.JS which references this canvas. This texture is updated every draw cycle.
A shader material is created in Three.JS which is linked to a fragment shader (which creates noise / scanlines)
This material is then applied to a number of 3D Planes.

This works just fine for showing single frames of video, but I'd like to see if I could show multiple frames at once without needing to add additional geometry.

What would be the optimal way of going about such a task? Are there any concepts that I should be studying/investigating in further detail?

Mr.doob Mr.doob · Accepted Answer · 2011-10-05T20:43:32

<html>
    <body>
        <script>

            var video = document.createElement( 'video' );
            video.autoplay = true;
            video.addEventListener( 'loadedmetadata', function ( event ) {

                var scale = 0.5;
                var width = video.videoWidth * scale;
                var height = video.videoHeight * scale;
                var items_total = ( window.innerWidth * window.innerHeight ) / ( width * height );

                for ( var i = 0; i < items_total - 1; i ++ ) {

                    var canvas = document.createElement( 'canvas' );
                    canvas.width = width;
                    canvas.height = height;

                    canvas.context = canvas.getContext( '2d' );
                    canvas.context.scale( scale, scale );

                    document.body.appendChild( canvas );

                }

                setInterval( function () {

                    var child = document.body.insertBefore( document.body.lastChild, document.body.children[ 1 ] ); // children[ 0 ] == <script>
                    child.context.drawImage( video, 0, 0 );

                }, 1000 / 30 );

            }, false );
            video.src = 'video.ogv';

        </script>
    </body>
</html>

Displaying multiple frames of a video element simultaneously

1 Answers