Recording video simultaneously with audio in chrome blocks on main thread, causing invalid audio

Question

So, I have what I think is a fairly interesting and, hopefully, not intractable problem. I have an audio/video getUserMedia stream that I am recording in Chrome. Individually, the tracks record perfectly well. However, when attempting to record both, one blocks the main thread, hosing the other. I know that there is a way to resolve this. Muaz Khan has a few demos that seem to work without blocking.

Audio is recorded via the web audio API. I am piping the audio track into a processor node which converts it to a 16b mono channel and streams it to a node.js server.

Video is recorded via the usual canvas hack and Whammy.js. When recording, video frames are drawn to a canvas and then the resulting image data is pushed into a frames array which is later encoded into a webm container by Whammy, subsequently uploaded to the node.js server.

The two are then muxed together via ffmpeg server-side and the result stored.

The ideas I've had so far are:

Delegate one to a worker thread. Unfortunately both the canvas and the stream are members of the DOM as far as I know.
Install headless browser in node.js and establish an rtc connection with the client, thereby exposing the entire stream server-side

The entire situation will eventually be solved by Audio Worker implementation. The working group seems to have stalled public progress updates on that while things are shuffled around a bit though.

Any suggestions for resolving the thread blocking?

Web Audio Connections:

var context = new AudioContext();
var source = context.createMediaStreamSource(stream);
var node = context.createScriptProcessor(2048, 1, 1);
node.onaudioprocess = audioProcess;
source.connect(node);
node.connect(context.destination);

Web Audio Processing:

if (!recording.audio) return;
var leftChannel = e.inputBuffer.getChannelData(0);
Socket.emit('record-audio', convertFloat32ToInt16(leftChannel));

Video Frame Buffering:

if (recording.video) {
players.canvas.context.fillRect(0, 0, players.video.width, players.video.height);
players.canvas.context.drawImage(players.video.element, 0, 0, players.video.width, players.video.height);
    frames.push({
        duration: 100,
        image: players.canvas.element.toDataURL('image/webp')
    });
    lastTime = new Date().getTime();
    requestAnimationFrame(drawFrame);
} else {
    requestAnimationFrame(getBlob);
}

Update: I've since managed to stop the two operations from completely blocking one another, but it's still doing it enough to distort my audio.

This is a problem I ran into on some weaker machines myself. The way I corrected it was not to do ANY processing of the packets(audio and video) until after they hit my server., The only processing you should do before shipping it off to your server is changing your dataURL to a jpeg. I would not do any media stuff like Whammy unless you are utilizing asm.js — Benjamin Trent
I ended up ultimately doing a peer connection to a server(to allow weaker clients, like phones). However, Here is the code I used when grabbing the buffers and frames. It could be improved through moving the URI to View canvas code to an asm.js function. — Benjamin Trent
@BenjaminTrent, I guess the problem might also be, using requestAnimationFrame for next video frame, I use a setInterval(fn, 1000/frameRate), this way you have a better control over frame rate... — mido
may be requestAnimationFrame calling a heavy process repeatedly is the problem... — mido
Also, have you tried increasing the buffer size for the ScriptProcessorNode? It seems sort of obvious, but I didn't see it mentioned above. — aldel

madhouse madhouse · Accepted Answer · 2015-03-25T18:42:10

There are a few key things that allow for successful getUserMedia recording in Chrome at the moment, as taken from a conglomeration of information gleaned from the helpful comments attached to the original question and my own experience.

When harvesting data from the recording canvas, encode as jpeg. I had been attempting webp to satisfy the requirements of Whammy.js. Generating a webp dataURI is apparently a cycle hog.
Delegate as much of the non-DOM operations as possible to worker threads. This is especially true of any streaming / upload operations (e.g., audio sample streaming via websockets)
Avoid requestAnimationFrame as a means of facilitating recording canvas drawing. It is resource intensive, and as Aldel has pointed out, can fail if the user switches tabs. Using setInterval is much more efficient/reliable. It also allows for better framerate control.
For Chrome at least, avoid client-side AV encoding for the time being. Stream audio samples and video frames server-side for processing. While client-side AV encoding libraries are very cool, they simply don't seem efficient enough for production quite yet.

Also, for Node.js ffmpeg automation, I highly recommend fluent-ffmpeg. Special thanks to Benjamin Trent for some practical examples.

Recording video simultaneously with audio in chrome blocks on main thread, causing invalid audio

2 Answers