PHP memcache::addServer, weight

Question

My question is related to the "weight" parameter in the addServer function of the PHP Memcache extension.
For last couple of months, I have been using "weight = 1" for all servers. I am now trying to apply the configurations below to eventually remove "10.10.10.3" from the rotation and minimize data loss.
With the new weight values in place, my PHP client fails to retrieve value for keys that it used to be able to fetch. If I revert to "weight = 1", all keys can be fetched without any problem.

Is there a configuration or anything I am missing in order to use the "weight" option correctly with "Memcache::addServer"?

Thank you for your help.

 $hosts = array(
      array('ip' => '10.10.10.1', 'port' => 11211, 'weight' => 100),
      array('ip' => '10.10.10.2', 'port' => 11211, 'weight' => 100),
      array('ip' => '10.10.10.3', 'port' => 11211, 'weight' => 1),
      array('ip' => '10.10.10.4', 'port' => 11211, 'weight' => 100),
 );

 $memcache = new Memcache();
 foreach($hosts as $host) {
    $host['port'] = isset($host['port']) ? (int) $host['port'] : 11211;
    $host['weight'] = isset($host['weight']) ? (int) $host['weight'] : 1;
    $memcache->addserver($host['ip'], $host['port'], false, $host['weight'], 1, 15);
 }

---

PHP version = 5.3.10

Memcache PHP variables
- memcache support => enabled
- memcache.allow_failover => 1 => 1
- memcache.chunk_size => 32768 => 32768
- memcache.compress_threshold => 20000 => 20000
- memcache.default_port => 11211 => 11211
- memcache.hash_function => crc32 => crc32
- memcache.hash_strategy => consistent => consistent
- memcache.lock_timeout => 15 => 15
- memcache.max_failover_attempts => 20 => 20
- memcache.protocol => ascii => ascii
- memcache.redundancy => 1 => 1
- memcache.session_redundancy => 2 => 2

Memcached versions
10.10.10.1 / 10.10.10.2 / 10.10.10.3 are running 1.4.5
10.10.10.4 is running 1.4.4

Answer 1

The weight parameter effects the consistent hashing used to determine which server to read/write keys from. Changing the weight of any 1 server in the pool will result in some cache misses. The number of servers in the pool and how much you change the weight will factor in on how many misses you may experience.

What you need to understand is that memcached is distributed which means that since you have 4 servers, keys are distributed amongst each of those servers (as close to evenly as possible [weight will effect even distribution]). If one server goes down, the data stored on that server is no longer accessible and will have to be fetched from the database as it is not available on any other servers. *Note that the PHP extensions memcache and memcached are just clients that access a memcached cluster (memcached was the newer of the two and supported more features, but both talk to servers in a memcached cluster).

When you want to store or retrieve a value from the cache, a hash is computed which determines where in the cluster the data should be put or read from. A common way to illustrate this is with a 360 degree circle as seen below. You calculate a hash, and use the node that it lands "closest" to in the circle. Adding or removing servers, or changing the weight of any one server will effect the outcome of the hashing and result in a miss.

Source: http://alpha.mixi.co.jp/blog/?p=158

If you want to phase a server out of the cluster slowly, I would recommend gradually decreasing its weight until it is 0 and you can then remove the server from the list completely. Keep in mind, any small change in weight can/will result in cache misses, but how much the weight changes (and how many servers you have) influences how many misses you will experience.

Here is a snippet from the Memcached tutorial story that may help explain some of this as well.

The preface is that two sysadmins set up a memcached cluster with 3 servers, and told memcached to use 1GB on each server...

So again, he takes keys that the Programmer uses and looks for them on his memcached servers. 'get this_key' 'get that_key' But each time he does this, he only finds each key on one memcached! Now WHY would you do this, he thinks? And he puzzles all night. That's silly! Don't you want the keys to be on all memcacheds?

"But wait", he thinks "I gave each memcached 1 gigabyte of memory, and that means, in total, I can cache three gigabytes of my database, instead of just ONE! Oh man, this is great," he thinks. "This'll save me a ton of cash. Brad Fitzpatrick, I love your ass!"

"But hmm, the next problem, and this one's a puzzler, this webserver right here, this one runing memcached it's old, it's sick and needs to be upgraded. But in order to do that I have to take it offline! What will happen to my poor memcache cluster? Eh, let's find out," he says, and he shuts down the box. Now he looks at his graphs. "Oh noes, the DB load, it's gone up in stride! The load isn't one, it's now two. Hmm, but still tolerable. All of the other memcacheds are still getting traffic. This ain't so bad. Just a few cache misses, and I'm almost done with my work. So he turns the machine back on, and puts memcached back to work. After a few minutes, the DB load drops again back down to 1, where it should always be.

"The cache restored itself! I get it now. If it's not available it just means a few of my requests get missed. But it's not enough to kill me. That's pretty sweet."

The moral is that values stored in memcached are only stored on one server (that is the keys are distributed amongst all servers) and if any one of those servers becomes unavailable, the data is not available from cache and has to be fetched from the source.

In order to properly store and retrieve data from the correct nodes in the cluster, keys must be consistently hashed so the client knows which server is the correct one in the pool for that particular key.

As you may guess, different clients use different hashing techniques so using a Perl client and a PHP client to store/retrieve values will not work as intended since they hash differently. The exception to this is any client that uses libmemcached since then they will all be using the same hashing algorithm. PHP's memcached extension uses libmemcached, where the memcache does not and uses its own hashing algorithm (as far as I know).

Further Reading:
- memcached wiki and the adventure in learning memcached
- Consistent Hashing
- Distributed Hash Tables

Answer 2

The problem is that you are using memcache vs memcached. Memcached is distributed, meaning that all servers added to the pool act as one cache server. So if server1 is down, you can still retrieve keys/values because they are distributed across the other three servers. Whereas with memcache (no d), each server is basically a silo, meaning that key/values stored on server1 are not guaranteed to be present on any of the other cache servers.

If you want to deprecate the cache server in question, I'd suggest adding some extra logic to your api call to pull the value from the persistent store, then store it across at least two of the other valid cache servers. I've seen projects where a cache value will be saved across two cache servers to ensure that one server is not longer responding, the other server can pick up the slack.

-- Update --

Regardless which PHP memcache API is used, if a server is removed from the pool, it will result in cache misses for keys residing on now missing server. When the $cache->get() fails and goes to the persistent store, and $cache->set() is called, the cache will now be stored on a different, active server. Thanks to Drew010 for clarification on this.

You can still write your app to write the same cache value to multiple servers if you absolutely cannot afford any cache misses.