How to use getUserMedia to record raw PCM audio data for use with WebAudio?

Question

My goal is to be able to record the microphone audio stream as raw float PCM data so I can use it to fill an audio buffer and manipulate the sound programmatically and do visualizations.

I can create a array buffer containing the audio data then use audioCtx.decodeAudioData and feed it to webAudio API and it plays back as it should, but I can't change the contents of the buffer so I'm confused at how that works.

I got it partly working by using a Analyser node in the getUserMedia promise.

It looks like this:

var audioCtx = new (window.AudioContext || window.webkitAudioContext)();
var audioBuffer = audioCtx.createBuffer(1, audioCtx.sampleRate * 3, audioCtx.sampleRate);

navigator.mediaDevices.getUserMedia({ audio: true })
.then(spectrum).catch(console.log);

    function spectrum(stream) {
        const audioCtx = new AudioContext();
        const analyser = audioCtx.createAnalyser();
            
        audioCtx.createMediaStreamSource(stream).connect(analyser);

        setInterval(() => {

            analyser.getFloatTimeDomainData(data);

            if(isRecording){
                for (let i = 0; i < data.length; i++) {
                    if(i < buffer.length)
                        buffer.push(data[i]);
                    else
                        break
                }
            }
        }, 1000 / audioCtx.sampleRate);
};

Where the audio buffer is being filled up with the data from the analyser when isRecording is true.

Then I can manipulate and play the audio with something like:

function manipulateAndPlayAudioBuffer(){
    let buf = buffer.getChannelData(0);

    for (let i = 0; i < buf.length; i++) {
        let t = (i / buf.length) * 3;

        // Add a sine wave to the audio
        let tone = Math.sin(3.1415*2*440*t);
        buf[i] += tone * 0.4;
    }

    var source = audioCtx.createBufferSource();
    source.buffer = buffer;
    source.connect(audioCtx.destination);
    source.start();

}

This almost works except the audio playback is poppy and sounds like data is missing.

There must be a correct solution to what I want to achieve but I have not found none.

it would help us to help you if you updated your post to show us copy and pastable minimal html code so we can run what you have to do battle ... I have done what you're asking but its been a while — Scott Stensland
An AnalyserNode is definitely not the way to go because it's not synchronized with the audio thread. Unless you're very, very luck, you'll get duplicated samples or gaps. And the way you're starting the source pretty much means there will be gaps because start() means start now, but doesn't mean it will actually start "now". A jsfiddle or something as @Cole-Peterson mentions would be very, very helpful. — Raymond Toy

Davide C Davide C · Accepted Answer · 2021-07-05T07:47:44

The answer is the AudioworkletProcessor class. It gives you direct access to the DSP soundcard buffer (running on the audio thread). At that point you can push your own values into the buffer, pull them somewhere else, manipulate them in real time etc..

You can derive your own processor from the main parent class and then connect it to your microphone, in the audio chain:

Microphone --> AudioProcessor --> Speakers (audioctx.destination).

You can use the audioworkletnode.port() method to export the audio data from the AudioworkletProcessor to the main thread.

How to use getUserMedia to record raw PCM audio data for use with WebAudio?

1 Answers