Why is Varnish caching ESI fragments (when I tell it not to)?

Question

II'm running into a peculiar problem with Varnish ESI.

Each page on my site has a ESI fragment that spits out the (logged-in) user's account information.

It looks like Varnish is returning the cached result of a user session ESI fragment unless the browser sends a Cache-Control: no-cache / Pragma: no-cache (force refresh) which is when it requests the ESI content from the server.

my vcl_recv contains rules to bypass this:

if(req.url ~ "^/session-header") {
    return(pass);
}

vcl_fetch has a rule as well:

if(req.url ~ "^/session-header") {
    return(hit_for_pass);
} else {
  set beresp.ttl = 1h;
}

Basically, this is the behavior from the browser side:

Cache-Control: max-age=0 => Varnish returns cached ESI fragment

Cache-Control: no-cache / Pragma: no-cache => Varnish fetches ESI fragment from the server

I can't seem to figure it out and I have a feeling I've missed some fairly obvious configuration.

Answer 1

In case someone sees this, figured out the issue.

Turns out the backend was setting Etag which Varnish used to decide whether or not to serve fresh ESI fragments from the server (the browser would send this Etag in the If-None-Match request heeader unless it was a force refresh).

I simply added this one line to vcl_deliver and now it works like a charm!

subl vcl_deliver {
  ...
  unset resp.http.Etag;
  ...
}

Answer 2

I had to whack the If-Modified-Since and If-None-Since request headers for the esi template to prevent the esi-assembled output from being served.

sub vcl_recv {
...
      unset req.http.If-Modified-Since;
      unset req.http.If-None-Match;
...
}