Varnish Cache Expiring Objects Too Quickly

Question

I've been having a problem with my varnish (v3.0.2) cache where it keeps resetting the cache of an object after less than 60 seconds despite having a TTL of 24h, cookies stripped, content encoding normalized, non-critical headers unset, Cache-Control set to public, s-maxage=86400 etc.

For some reason, if you access the following URL repeatedly over a minute, you can see that the Age creeps up and then hits zero (with X-Cache returning MISS):

http://data.eyewire.org/volume/83329/chunk/0/1/0/1/tile/xz/32:64

There are no n_lru_nuked objects and the cache is over 60GB. I watched the varnishlog and may have seen something with ExpBan going on, but I can't for the life of me figure out why.

Here are some key parts to my vcl file:

sub vcl_recv {
  set req.grace = 120s;

   # normalize Accept-Encoding to reduce vary
  if (req.http.Accept-Encoding) {
    if (req.http.User-Agent ~ "MSIE 6") {
      unset req.http.Accept-Encoding;
    } 
    elsif (req.http.Accept-Encoding ~ "gzip") {
      set req.http.Accept-Encoding = "gzip";
    } 
    elsif (req.http.Accept-Encoding ~ "deflate") {
      set req.http.Accept-Encoding = "deflate";
    } 
    else {
      unset req.http.Accept-Encoding;
    }
  }

  # This uses the ACL action called "purge". Basically if a request to
  # PURGE the cache comes from anywhere other than localhost, ignore it.
  if (req.request == "PURGE") 
    {if (!client.ip ~ purge)
      {error 405 "Not allowed.";}
    return(lookup);}

  if (req.http.Upgrade ~ "(?i)websocket") {
    return (pipe);
  }

  # ....

  if ( req.http.host ~ "data\.eyewire\.org" ) {
    unset req.http.Cookie;
    unset req.http.Accept-Language;
    unset req.http.Expires;
    unset req.http.Cache-Control;
    unset req.http.User-Agent;
    return(lookup);
  }

  # ....
}


sub vcl_fetch {
  # ....

  if ( req.http.host ~ "data.eyewire.org" ) {
    if ( req.url ~ "^/volume" ) {
      unset beresp.http.Set-Cookie;
      set beresp.ttl = 24h;
      set beresp.http.Cache-Control = "public, s-maxage=86400";
      set beresp.http.X-TTL = beresp.ttl;
      return(deliver);
    }
    elsif (req.url ~ "^/cell") {
      set beresp.ttl = 1h;
      return(hit_for_pass);
    }
  }
}

# from http://blog.bigdinosaur.org/adventures-in-varnish/
sub vcl_pass {
  set bereq.http.connection = "close";
  if (req.http.X-Forwarded-For) {
      set bereq.http.X-Forwarded-For = req.http.X-Forwarded-For;
  }
  else {
      set bereq.http.X-Forwarded-For = regsub(client.ip, ":.*", "");
  }
}

# from http://blog.bigdinosaur.org/adventures-in-varnish/
sub vcl_pipe {
  #we need to copy the upgrade header
  if (req.http.upgrade) {
      set bereq.http.upgrade = req.http.upgrade;
      set bereq.http.connection = req.http.connection;
  }

  set bereq.http.connection = "close";
  if (req.http.X-Forwarded-For) {
      set bereq.http.X-Forwarded-For = req.http.X-Forwarded-For;
  }
  else {
      set bereq.http.X-Forwarded-For = regsub(client.ip, ":.*", "");
  }
}

# from http://blog.bigdinosaur.org/adventures-in-varnish/
sub vcl_hit {
  if (req.request == "PURGE") {
      purge;
      error 200 "Purged.";
  }
}

# from http://blog.bigdinosaur.org/adventures-in-varnish/
sub vcl_miss {
  if (req.request == "PURGE") {
      purge;
      error 200 "Purged.";
  }
}

sub vcl_deliver {  
    # Display hit/miss info
    if (obj.hits > 0) {
      set resp.http.X-Cache = "HIT";
      set resp.http.X-Cache-Hits = obj.hits;
    }
    else {
        set resp.http.X-Cache = "MISS";
    }

    # Security Non-Disclosure
    remove resp.http.X-Varnish;
    remove resp.http.X-Powered-By;
    remove resp.http.Server;

    return(deliver);
}

Thanks!

Edit: FYI: I had to revert some changes to my VCL to solve a problem in production, but the problem is still essentially the same.

If thats actually the VCL thats being used I find it very odd that the X-TTL is not present in the response... Might there be a return before you reach the host if clause? (#.... I guess indicates what you assume does not play in) — Clarence
Hey Clarence, I'm sorry I should add that part back in. I reverted some of the debugging info when I rolled the VCL back. However, I can add the VCL deliver section back in no problem. Let me do that now. — SapphireSun
Can you confirm that there is no external process issuing PURGE requests on the content? If content is not being evicted by LRU nuke, then it's most likely being purged/banned from the cache. — Ketola

Brian van Rooijen Brian van Rooijen · Accepted Answer · 2014-10-24T12:44:33

I experienced the same kind of behaviour where the hits where going up and without any reason it seemed the cache was purged. After some research i found the cause was the vary: Accept-Encoding, User-agent header that made a different cache being saved per user agent.

Try setting a vary header only for Accept-Encoding.

Varnish Cache Expiring Objects Too Quickly

2 Answers