Memcached request loop or cache stampede in WordPress

Question

I have a clustered WordPress webapp that is using a Memcached cluster consisting of six nodes to store WordPress's object cache, but I'm seem to be experiencing some sort of sporadic Memcached set/replace loop or cache stampede that occasionally causes a single node in the Memcached cluster to completely saturate the Memcached and database network link.

The set/replace loop or stampede seems to be caused by lots of attempts trying to “add” the WordPress “wp_:options:alloptions” key which Memcached replies with NOT_STORED to all those particular queries. While at the same time majority of MySQL queries look like this:

SELECT option_name, option_value 
FROM wp_options 
WHERE autoload = 'yes'

The only change that I've made to the Memcached Object Cache plugin (http://wordpress.org/extend/plugins/memcached/), is the default expiration setting.

#var $default_expiration = 0; //original
var $default_expiration = 1800;

So far I have been able to mitigate the problem by increasing the Memcached daemon maximum connection limit. I also changed the database storage engines from MyISAM to InnoDB, and also increased the APC shm_size from 128M to 1536M.

Is the object cache set/replace loop or stampede inevitable or is it a consequence of my current setup?

Environment:

WordPress Version 3.7.1
Memcached 2.0.2 Plugin http://wordpress.org/extend/plugins/memcached/

Memcached 1.4.14-1

PORT="11211"
USER="nobody"
MAXCONN="10240"
CACHESIZE="4096"
OPTIONS=""

PHP 5.3.28
memcache Version 3.0.8 memcache support => enabled memcache.allow_failover => 0 => 0 memcache.chunk_size => 32768 => 32768 memcache.compress_threshold => 20000 => 20000 memcache.default_port => 11211 => 11211 memcache.hash_function => fnv => fnv memcache.hash_strategy => consistent => consistent memcache.lock_timeout => 15 => 15 memcache.max_failover_attempts => 20 => 20 memcache.protocol => ascii => ascii memcache.redundancy => 1 => 1 memcache.session_redundancy => 2 => 2 Registered save handlers => files user memcache
APC Version => 3.1.9 APC Debugging => Disabled MMAP Support => Enabled MMAP File Mask => Locking type => pthread mutex Locks Serialization Support => broken Directive => Local Value => Master Value apc.cache_by_default => On => On apc.canonicalize => On => On apc.coredump_unmap => Off => Off apc.enable_cli => Off => Off apc.enabled => On => On apc.file_md5 => Off => Off apc.file_update_protection => 2 => 2 apc.filters => no value => no value apc.gc_ttl => 3600 => 3600 apc.include_once_override => Off => Off apc.lazy_classes => Off => Off apc.lazy_functions => Off => Off apc.max_file_size => 1M => 1M apc.mmap_file_mask => no value => no value apc.num_files_hint => 1000 => 1000 apc.preload_path => no value => no value apc.report_autofilter => Off => Off apc.rfc1867 => Off => Off apc.rfc1867_freq => 0 => 0 apc.rfc1867_name => APC_UPLOAD_PROGRESS => APC_UPLOAD_PROGRESS apc.rfc1867_prefix => upload_ => upload_ apc.rfc1867_ttl => 3600 => 3600 apc.serializer => default => default apc.shm_segments => 1 => 1 apc.shm_size => 1536M => 1536M apc.slam_defense => On => On apc.stat => On => On apc.stat_ctime => Off => Off apc.ttl => 0 => 0 apc.use_request_time => On => On apc.user_entries_hint => 4096 => 4096 apc.user_ttl => 0 => 0 apc.write_lock => On => On

Have you tried adding a random, additional delay to the expiration time, so that all the keys doesn't expire at the same-ish time (+rand(15,45))? — MatsLindh

AlexL AlexL · Accepted Answer · 2014-10-27T23:50:37

MatsLindh is right. Add random expiration to the options to minimize the number of entries that expire at the same time. You can also minimize stampedes with locks while the 1st racer writes the cache.

Memcached request loop or cache stampede in WordPress

1 Answers