Cannot exclude robot.txt from varnish cache

Question

I tried to exclude robots.txt form varnish cache by using the following lines of code in default.vcl

if(req.url ~ "^/robots\.txt$") {
   return(pass);
}

Now Network tab in dev tools, it shows a Age: 0 and X-Cache:MISS. But,for some reason varnish does not exclude the file from being cached. I even deleted the file from its location. But still its loading the url https://www.example.com/robots.txt

I also purged varnish cache using following commands

curl -X PURGE www.example.com/robots.txt

and

varnishadm "ban req.http.host == www.example.com && req.url ~ ^/robots.txt"

and

varnishadm "ban req.http.host ~ www.example.com && req.url ~ ^/robots.txt"

It shows the 200 Purged message, but still no luck.

Can anyone help me out ?

If the Age is 0 and it is a MISS, it's not clear what you mean by "being excluded from being cached" since it appears that's exactly what's happening. — Joshua DeWald

Hgfjj Hhjgf Hgfjj Hhjgf · Accepted Answer · 2017-10-21T10:18:18

"I even deleted the file from its location. But still its loading the url https://www.example.com/robots.txt" -- may be your browser is caching it.
Sending PURGE request to varnish will only remove the object from the cache, not from the backend, so if you PURGE something from varnish and then send a GET request to it, it will serve the object to you for sure.
If you want it to be lost forever, you need to remove it from your backend.

Cannot exclude robot.txt from varnish cache

1 Answers