NSURLCache caching random responses that should not be cached

Question

I am implementing an app which does a lot of networking calls to a rest-api that we also control. Recently we decided to introduce caching headers on the server side to save some valuable networking and server time. As we do not know beforehand for how long the data will be valid, we are not sending Cache-control: max-age or Expires headers, all we do is send a Last-Modified header together with a E-tag, so we always hit the server but responses are pretty fast most of the times with a 304. Everything seemed to work fine at first, with many requests being cached. However, I am experiencing some random data errors on the app due to the caching.

For some reason I can not understand, at some point requests are being locally cached and used as "updated" data without hitting the server, when they actually are not. The problem keeps there until some time passes. Then everything goes to server normally again, exactly as it would behave with a cache-control header, but without it!. So, my question is:

How can NSURLCache together with NSURLConnection decide that a particular request does not need to go online when the original request did not come with Cache-control: max-age or Expires headers? Has anyone experienced similar effects? And how can I solve it without removing the whole cache?

Some more background info:

I am using AFNetworking, but it relies on NSURLConnection so I do not think it changes anything
The cache used is the default [NSURLCache sharedURLCache] instance

It is a GET request, and when I check the headers from the cached response this is what I get:

po [response allHeaderFields]

"Access-Control-Allow-Headers" = "Content-Type";
"Access-Control-Allow-Methods" = "GET, POST, DELETE, PUT";
"Access-Control-Allow-Origin" = "*";
Connection = "keep-alive";
"Content-Encoding" = gzip;
"Content-Length" = 522;
"Content-Type" = "application/json";
Date = "Mon, 02 Sep 2013 08:00:38 GMT";
Etag = "\"044ad6e73ccd45b37adbe1b766e6cd50c\"";
"Last-Modified" = "Sat, 31 Aug 2013 10:36:06 GMT";
Server = "nginx/1.2.1";
"Set-Cookie" = "JSESSIONID=893A59B6FEFA51566023C14E3B50EE1E; Path=/rest-api/; HttpOnly";

I can not predict or reproduce when the error is going to happen so solutions that rely on deleting the cache are not an option.
I am using iOS5+

No chance that there's a clock sync issue between the device and the server? Specs says the behavior of the client is undefined in that case (up to the client to do what appropriate). Also have you tried to instrument afnetworking code just to gain more insight? — Alex
the time issue was my first thought but I could not find a repeatable scenario (it happens very randomly). Anyway, it happend on devices with proper time and server side is also correct, so it does not seem to be so :( thanks for the tip! — Angel G. Olloqui

Caleb Caleb · Accepted Answer · 2013-09-10T10:39:38

How can NSURLCache together with NSURLConnection decide that a particular request does not need to go online when...

Section 13.2 of RFC 2616 says:

Since origin servers do not always provide explicit expiration times, HTTP caches typically assign heuristic expiration times, employing algorithms that use other header values (such as the Last-Modified time) to estimate a plausible expiration time. The HTTP/1.1 specification does not provide specific algorithms, but does impose worst-case constraints on their results. Since heuristic expiration times might compromise semantic transparency, they ought to used cautiously, and we encourage origin servers to provide explicit expiration times as much as possible.

So, it's possible for the URL loading system to decide that the cached data is "fresh enough" even though you haven't provided a specific lifetime for the data.

For best results, you should try to provide a specific lifetime in your response headers. If adding such a header is impossible, perhaps you could change the request instead. if-modified-since or cache-control could each help you avoid cached data.

NSURLCache caching random responses that should not be cached

3 Answers