Any alternatives to Memcache on Google App engine?

Question

I am currently using the Memcache service provided by GAE to cache content on the server. The current size of the cache is close to 20~30MB.

Initially the cache had a lifetime of 6-7 hours .. with increasing traffic, the lifetime of the cache has fallen to 20 minutes.

We are planning to increase the cache size to 1-2GBs. Are there any alternative implementations of Distributed Caching on GAE we can use?

The List of methods that I have already tried are mentioned below. But, these steps do not fix our need to have better caching service on GAE

Using Memcache (cons - limited cache size)
Store object in the Instance Memory (cons - data consistency across instances cannot be maintained)
Compressing JAVA Objects being stored ( slight improvement - only 20% improvement in lifetime of cache)

What are you storing in the Memcache? Raw data, or files like HTML and images? If you are storing files there, you can use @dlebech's option. — Brent Washburne
@BrentWashburne I am storing raw java objects. The average size of each object is around 1KB. I use GzipCompressorInputStream to compress the object before storing them in cache. The content in the cache is highly dynamic (changes every 2-3 minutes). The cache is updated using 'task queues' based on different events triggered by the user. Using Edge Cache will not enable me to regularly update the content shown to the client browser — user1226511
I think your only option (on App Engine) is to put all the data into the data store and cache the data upon request. That way only the data being requested is in the cache. — Brent Washburne
@BrentWashburne I am already doing this ... We have only 2500 JAVA objects being stored in cache. The number was fine for low traffic sites. But as the site scales, it requires larger cache size to store more number of objects. 20-30MB of cache is too low for. Due to the short lifetime/size of the cache, the response times are getting longer for most requests. — user1226511
You could create a distributed cache by creating more Applications and sharding your data across them. The new applications would simply respond to HTTP requests from your primary application and perform memcache calls. — Brent Washburne

dlebech dlebech · Accepted Answer · 2013-06-11T14:34:12

Since you were originally relying on a cache of 6-7 hours, this sounds like an excellent use case for taking advantage of Google's Edge Cache. This is, in theory, a free cache based on Google's distributed caching of websites.

Basically, you want to set caching headers such as:

Cache-Control: public, max-age=600

See this SO answer and this Google Groups post.

Any alternatives to Memcache on Google App engine?

2 Answers