Twilio Programmable video - Integrating with Web Audio API

Question

I'm trying to use the Web Audio API to manipulate the audio streams of remote participants in a Twilio Programmable Video Room.

You can get access to the MediaStreamTrack that corresponds to each Track in the onTrackAdded Handler trivially, by calling track.mediaStreamTrack

However to use it with the Web Audio API - you must have a reference to the MediaStream so you can call audioContext.createMediaStreamSourceNode(stream). My understanding is that a MediaStream object can contain many MediaStreamTracks. (making it more akin to the participant object in the Twilio abstraction).

I have found a way around this by using private properties of the room construct that the twilio client provides, but it is very ugly:

function trackAdded(track, room) {
// Check if it is an audio node
if (track.kind.toLowerCase() === 'audio') {
    // Find the relevant pc to get the stream from
    var remotePC;
    room.room._signaling._peerConnectionManager._peerConnections.forEach(function(pc) {
        var remoteStream = pc.getRemoteStreams()[0]
        if (remoteStream.getAudioTracks()[0] === track.mediaStreamTrack) {
            // This is the pc we are interested in
            console.log('found PC to connect to audio API');
            remotePC = pc;
        }
    })
    var origin = context.createMediaStreamSource(remotePC.getRemoteStreams()[0]);
    origin.connect(<Chain of filter Nodes>)
}

I guess my question is, is there a way to get a reference to a MediaStreamTrack's MediaStream container in JS?

OR

Is there a more Twilio idiomatic way to get a reference to the MediaStream that corresponds to a particular participant?

philnash philnash · Accepted Answer · 2017-05-19T16:14:23

Twilio developer evangelist here.

Rather than listen to the trackAdded event on a room, you can listen to trackAdded for each participant. That way you can associate the track and participant easier. You can get hold of each participant as they join a room by listening for the participantConnected event on a room.

Once you get the trackAdded event, you can use the Track object and get the underlying MediaStreamTrack by using the mediaStreamTrack property.

Twilio Programmable video - Integrating with Web Audio API

1 Answers