I tried to exclude robots.txt form varnish cache by using the following lines of code in default.vcl
if(req.url ~ "^/robots\.txt$") {
return(pass);
}
Now Network tab in dev tools, it shows a Age: 0 and X-Cache:MISS. But,for some reason varnish does not exclude the file from being cached. I even deleted the file from its location. But still its loading the url https://www.example.com/robots.txt
I also purged varnish cache using following commands
curl -X PURGE www.example.com/robots.txt
and
varnishadm "ban req.http.host == www.example.com && req.url ~ ^/robots.txt"
and
varnishadm "ban req.http.host ~ www.example.com && req.url ~ ^/robots.txt"
It shows the 200 Purged message, but still no luck.
Can anyone help me out ?