Live streaming: node-media-server + Dash.js configured for real-time low latency

9
votes

We're working on an app that enables live monitoring of your back yard. Each client has a camera connected to the internet, streaming to our public node.js server.

I'm trying to use node-media-server to publish an MPEG-DASH (or HLS) stream to be available for our app clients, on different networks, bandwidths and resolutions around the world.

Our goal is to get as close as possible to live "real-time" so you can monitor what happens in your backyard instantly.

The technical flow already accomplished is:

  1. ffmpeg process on our server processes the incoming camera stream (separate child process for each camera) and publishes the stream via RTSP on the local machine for node-media-server to use as an 'input' (we are also saving segmented files, generating thumbnails, etc.). the ffmpeg command responsible for that is:

    -c:v libx264 -preset ultrafast -tune zerolatency -b:v 900k -f flv rtmp://127.0.0.1:1935/live/office

  2. node-media-server is running with what I found as the default configuration for 'live-streaming'

    private NMS_CONFIG = {
server: {
  secret: 'thisisnotmyrealsecret',
},
rtmp_server: {
  rtmp: {
    port: 1935,
    chunk_size: 60000,
    gop_cache: false,
    ping: 60,
    ping_timeout: 30,
  },
  http: {
    port: 8888,
    mediaroot: './server/media',
    allow_origin: '*',
  },
  trans: {
    ffmpeg: '/usr/bin/ffmpeg',
    tasks: [
      {
        app: 'live',
        hls: true,
        hlsFlags: '[hls_time=2:hls_list_size=3:hls_flags=delete_segments]',
        dash: true,
        dashFlags: '[f=dash:window_size=3:extra_window_size=5]',
      },
    ],
  },
},

    };

  3. As I understand it, out of the box NMS (node-media-server) publishes the input stream it gets in multiple output formats: flv, mpeg-dash, hls. with all sorts of online players for these formats I'm able to access and the stream using the url on localhost. with mpeg-dash and hls I'm getting anything between 10-15 seconds of delay, and more.

My goal now is to implement a local client-side mpeg-dash player, using dash.js and configure it to be as close as possible to live.

my code for that is:

<!doctype html>
<html>
    <head>
        <title>Dash.js Rocks</title>
        <style>
            video {
                width: 640px;
                height: 480px;
            }
        </style>
    </head>
    <body>
        <div>
            <video autoplay="" id="videoPlayer" controls=""></video>
        </div>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/dashjs/3.0.2/dash.all.min.js"></script>

        <script>
            (function(){
                // var url = "https://dash.akamaized.net/envivio/EnvivioDash3/manifest.mpd";
                var url = "http://localhost:8888/live/office/index.mpd";
                var player = dashjs.MediaPlayer().create();
                
                

                // config
                targetLatency = 2.0;        // Lowering this value will lower latency but may decrease the player's ability to build a stable buffer.
                minDrift = 0.05;            // Minimum latency deviation allowed before activating catch-up mechanism.
                catchupPlaybackRate = 0.5;  // Maximum catch-up rate, as a percentage, for low latency live streams.
                stableBuffer = 2;           // The time that the internal buffer target will be set to post startup/seeks (NOT top quality).
                bufferAtTopQuality = 2;     // The time that the internal buffer target will be set to once playing the top quality.


                player.updateSettings({
                    'streaming': {
                        'liveDelay': 2,
                        'liveCatchUpMinDrift': 0.05,
                        'liveCatchUpPlaybackRate': 0.5,
                        'stableBufferTime': 2,
                        'bufferTimeAtTopQuality': 2,
                        'bufferTimeAtTopQualityLongForm': 2,
                        'bufferToKeep': 2,
                        'bufferAheadToKeep': 2,
                        'lowLatencyEnabled': true,
                        'fastSwitchEnabled': true,
                        'abr': {
                            'limitBitrateByPortal': true
                        },
                    }
                });

                console.log(player.getSettings());

                setInterval(() => {
                  console.log('Live latency= ', player.getCurrentLiveLatency());
                  console.log('Buffer length= ', player.getBufferLength('video'));
                }, 3000);

                player.initialize(document.querySelector("#videoPlayer"), url, true);

            })();

        </script>
    </body>
</html>

with the online test video (https://dash.akamaized.net/envivio/EnvivioDash3/manifest.mpd) I see that the live latency value is close to 2 secs (but I have no way to actually confirm it. it's a video file streamed. in my office I have a camera so I can actually compare latency between real-life and the stream I get). however when working locally with my NMS, it seems this value does not want to go below 20-25 seconds.

Am I doing something wrong? any configuration on the player (client-side html) I'm forgetting? or is there a missing configuration I should add on the server side (NMS) ?

2
node.jsffmpeglive-streamingmpeg-dashdash.js
You are doing everything right, except your initial expectation of being able to achieve low latency with this technology. WebRTC is what you need to use. – user1390208

2 Answers

7
votes

HLS and MPEG DASH are not particularly low latency as standard and the figures you are getting are not unusual.

Some examples from a publicly available DASH forum document (linked below) include:

enter image description here

enter image description here

Given the resources of some of these organisations, the results you have achieved are not bad!

There is quite a focus in the streaming industry at this time on enabling lower latency, the target being to come as close as possible to traditional broadcast latency.

One key component of the latency in chunked Adaptive Bit Rate (ABR, see this answer for more info: https://stackoverflow.com/a/42365034/334402 ) is the need for the player to receive and decode one or more segments of the video before it can display it. Traditionally the player had to receive the entire segment before it could start to decode and display it. The diagram from the first linked open source reference below illustrates this:

enter image description here

Low latency DASH and HLS leverage CMAF, 'Common Media Application Format' which breaks the segments, which might be 6 seconds long for example, into smaller 'chunks' within each segment. These chunks are designed to allow the player to decode and start playing them before it has received the full segment.

Other sources of latency in a typical live stream will be any transcoding from one format to another and any delay in a streaming server receiving the feed, from the webcam in your case, and encoding and packaging it for streaming.

There is quite a lot of good information available on low latency streaming at this time both from standards bodies and open source discussions which I think will really help you appreciate the issues (all links current at time of writing). From open source and standards discussions:

and from vendors:

Note - a common use case often quoted in the broadcast world is the case where someone watching a live event like a game may hear their neighbours celebrating a goal or touchdown before they see it themselves, because their feed has higher latency than their neighbours. While this is a driver for low latency, this is really a synchronisation issue which would require other solutions if a 'perfectly' synchronised solution was the goal.

As you can see low latency streaming is not a simple challenge and it may be that you want to consider other approaches depending on the details of your use case, including how many subscribers you have, whether some loss of quality if a fair trade off for lower latency etc. As mentioned by @user1390208 in the comments, a more real time focused video communication technology like WebRTC may be a better match for the solution you are targeting.

If you want to provide a service that provides life streaming and also a recording, you may want to consider using a real time protocol for the live streaming view and HLS/DASH streaming for anyone looking back through recordings where latency may not be important but quality may be more key.

1
votes

Have worked on similar tasks - to get as much close to real-time as possible. During research found that DASH is not quite the fastest way to achieve real-time translation. With some major tune-ups on the partner's mediaserver-side and FFmpeg we could get a 6 seconds delay. That's quite enough for us with the user-viewing part of a system. But for translation moderators and admins we want to be more real-time. But there is another solution and we use it. Websockets/WebRTC. We mostly use the Wowza streaming engine as an enterprise solution and with some tune-ups, we have achieved a 2-second WebRTC delay.

But in case of NMS there is a way to get websocket-flv streaming too just out of the box. Just for testing, with the use of their player solution at their github site, I've achieved just 4-4.5 seconds of delay right now. sample image Maybe this information will be useful.