Propagate change in distributed in-memory cache

Question

I've an application deployed on a cluster of 1000 commodity boxes. While starting, each instance of the application loads a non-trivial amount of data from database and uses this as cache. During a day, around 20%of this cached data needs to be updated.

What are the efficient ways of near simultaneous update of in-memory data of entire cluster? I thought of JMX, Zookeeper, but not sure if that would be really efficient/fast enough.

You can use a distributed caching solution like redis or hazelcast. — Shubham Chaurasia

XGoVoid XGoVoid · Accepted Answer · 2017-02-13T19:25:34

Well assuming you're using Memcached's consistent hashing, go a step further and have each cache replicate to their closest successor. This can lessen the problem but not entirely alleviate it but it's a simple solution, Gossip + CRDTs are another solution, Dynamo and Riak use a combination of Gossip, Consistent Hashing, and CRDTs.

Propagate change in distributed in-memory cache

1 Answers