Getting delay on HLS with varnish cache

Question

I'm using varnish as a cache for my live streaming content. I have on the backend a video server (Mistserver) with HLS output on port 8090. I've configured varnish in the following way:

`vcl 4.0;
import std;
backend default {
    .host = "127.0.0.1";
    .port = "8090";
}
acl purge {
    "127.0.0.1";
}
sub vcl_recv {
    if(req.method == "PURGE"){
        if (!client.ip ~ purge) {
            return(synth(405,"Not allowed."));
        }
        return (purge);
    }
    return (hash);
}
sub vcl_hash {
   return (lookup);
}
sub vcl_hit {
    if (obj.ttl >= 0s) {
        return (deliver);
    }
    if (std.healthy(req.backend_hint)) {
        if (obj.ttl + 10s > 0s) {
            return (deliver);
        }
        else {
            return(fetch);
        }
    }
    else {
        if (obj.ttl + obj.grace > 0s) {
            return (deliver);
        }
        else {
            return (fetch);
        }
    }
    return (fetch); # Dead code, keep as a safeguard
}
sub vcl_purge{
}
sub vcl_pipe {
    return (pipe); 
}
sub vcl_init {
    return (ok);
}
sub vcl_fini {
    return (ok);
}
sub vcl_backend_response {
    set beresp.http.Access-Control-Max-Age = 15;
    set beresp.ttl = 15s;
    set beresp.grace = 15s;
}`

And varnish is being executed with a thread pool timeout of 30 seconds:

/usr/sbin/varnishd -a :6081 -T localhost:6082 -f /etc/varnish/user.vcl -S /etc/varnish/secret -s malloc,1G -t 3 -p thread_pool_timeout=30

The Problem: The content is being cached, apparently, correctly. But after some time the stream is running, I'm getting a difference of time with the original stream of 5 minutes. so for example if the live stream has time 22:25, I might see in the output stream the time 22:20. I can't have such a time difference in a live stream.

I've tried purging the cache, but didn't work. Sometime it is correclty synchronized, If I restart both video server and varnish. What I need is to avoid caching for mor than 1 minute or less.

What am I doing wrong?

Thanks Indeed

Guillaume Quintard Guillaume Quintard · Accepted Answer · 2018-10-21T06:26:15

(not your issue here, but: I'd remove the vcl_hit section, it's probably not doing anything good for you)

For live video, the manifest needs to be treated careful, essentially, you want to lag as little behind as possible. So the first thing to do is to kill the grace, and the second one is to shorten the TTL to half a chunk duration.

Try with something like:

vcl 4.0;
import std;
backend default {
    .host = "127.0.0.1";
    .port = "8090";
}
acl purge {
    "127.0.0.1";
}
sub vcl_recv {
    if(req.method == "PURGE"){
        if (!client.ip ~ purge) {
            return(synth(405,"Not allowed."));
        }
        return (purge);
    }
    return (hash);
}

sub vcl_backend_response {
    if (bereq.url ~ "m3u8") {
        # assuming chunks are 2 seconds long
        set beresp.ttl = 1s;
        set beresp.grace = 0s;
    } else {
        set beresp.ttl = 10m;
        set beresp.grace = 5m;
    }
}

Note: You don't want the manifest delivered by varnish to lag behind the real one. The current setup can deliver data that is 30 seconds old, which is usually way to much for live. Interestingly, it seems the player will just wait until the manifest shows new chunk to resume playing, I guess this is how the delay accumulate. The usual case is actually worse as the player just stop altogether if there's no new chunk once it's done reading the newest chunks.

Getting delay on HLS with varnish cache

1 Answers