My varnish config doesn't seem to be working properly

Question

I'm very new on varnish and I've a business on my hands recently. It's a local magazine website http caching (Tech Stack is Javascript + PHP). I'm trying to use varnish 4 for caching the website. What they want me to do is; any new articles should be appeared on FE immediately, any deleted articles should be erased from the FE immediately, any changes on website's current appereance should be applied directly (changing articles' current locations, they can be dragged anywhere on the website based on articles' popularity change.) and finally any changes on existing articles should be applied to website immediately. As you see on the config below, in sub vcl_recv block I tried to use return(purge) for POST requests, because new articles and article changes is applied via POST request. But it doesn't work at all. When I try create a new dummy content or make changes on existing articles, it's not purging the cache and showing the fresh content even if POST request is successful. Also, on the BE side, I tried to use if (beresp.status == 404) for deleted articles, but it doesn't work too. When I delete the dummy article I created, it's not being deleted too, I'm still seein the stale content. How should I change my config to get all these things done? Thank you.

my varnish config is ;


import directors;
import std;

backend server1 {
    .host = "<some ip>";
    .port = "<some port>";
}

sub vcl_init {
    new bar = directors.round_robin();
    bar.add_backend(server1);
}
sub vcl_recv {
    set req.backend_hint = bar.backend();

    if (req.http.Cookie == "") {
        unset req.http.Cookie;
    }

    set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", "");

    if (req.url ~ "\.(css|js|png|gif|jp(e)?g|swf|ico)") {
        unset req.http.cookie;
    }
    if (req.url ~ "\.*") {
        unset req.http.cookie;
    }
    if (req.method == "POST") {
        return(purge);
    }
}
sub vcl_deliver {

    # A bit of debugging info.
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    }
    else {
        set resp.http.X-Cache = "MISS";
    }
}

sub vcl_backend_response {

    set beresp.grace = 1h;

    set beresp.ttl = 120s;

    if (bereq.url ~ "\.*") {
        unset beresp.http.Set-Cookie;
        unset beresp.http.Cache-Control;
    }
    if (bereq.method == "POST") {
       return(abandon);
   }
    if (beresp.status == 404) {
       return(abandon);
    }
    return (deliver);
}

It doesn't work as you think... The "immediate" purge is impossible for something that was already cached, without some effort on the app's side. For your task (and that is fairly a common one), the app should send PURGE requests to Varnish (which should be coupled with some additions to the VCL). It is important to understand that there should be actual code in the website which talks to Varnish by issuing PURGE requests. — Danila Vershinin
@DanilaVershinin thank you for your answer, I'll talk to developers about this situation. — Oguzhan Aygun

Thijs Feryn Thijs Feryn · Accepted Answer · 2020-02-17T12:16:13

No need to use the director if you only have one backend. Varnish will automatically select the backend you declared if there's only 1 backend.

Purging content

The POST purge call you're doing is not ideal. Please have a look at the following page to learn more about content invalidation in Varnish: https://varnish-cache.org/docs/6.0/users-guide/purging.html#http-purging

The snippet on that page contains an ACL to protect your platform from unauthorized purges.

It's important to know that you'll need to create a hook into your CMS or your MVC controller, that does the purge call.Here's a simple example using curl in PHP:

    $curl = curl_init("http://your.varnish.cache/url-to-purge");
    curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "PURGE");
    curl_exec($curl);

As you can see, this is an HTTP request done in cURL that uses the custom PURGE HTTP request method. This call needs to be executed in your good right after the changes are stored in the database. This post-publishing hook will ensure that Varnish clears this specific object from cache.

VCL cleanup

The statement below doesn't look like a reliable way to remove cookies, because the expression will remove cookies from all pages dat contain a dot:

    if (req.url ~ "\.*") {
        unset req.http.cookie;
    }

The same applies to the following statement coming from the vcl_backend_response hook:

    if (bereq.url ~ "\.*") {
        unset beresp.http.Set-Cookie;
        unset beresp.http.Cache-Control;
    }

I assume some pages do actually need cookies to properly function. An admin panel for example, or the CMS, or maybe even a header that indicates whether or not you're logged in.

The best way forward is to define a blacklist or whitelist of URL patterns that can or cannot be cached.

Here's an example:

    if(req.url !~ "^/(admin|user)" {
        unset req.http.Cookie;
    }

The example above will only keep cookies for pages that start with /admin or /user. There are other ways as well.

Conclusion

I hope the purging part is clear. If not, please take a closer look at https://varnish-cache.org/docs/6.0/users-guide/purging.html#http-purging.

In regards to the VCL cleanup: purging can only work if the right things are stored in cache. Dealing with cookies can be tricky in Varnish.

Just try to define under what circumstances cookies should be kept for specific pages. Otherwise, you can just remove the cookies.

Hope that helps. Good luck. Thijs

My varnish config doesn't seem to be working properly

1 Answers

Purging content

VCL cleanup

Conclusion