PHP memcache returns false with increase in number of keys

Question

We are working with PHP and are using memcached v1.4.6 for caching. We are using memcache extension memcache-2.2.6 from PECL. We are using persistent connection to connect to memcached.

Recently we made some changes that has doubled the number of keys stored in memcached. These keys are at max 75 to 80 characters long. The values stored are integers.

Whenever we try working with the new code, the system works fine for the first few seconds (usually less than 10 seconds). After the first few seconds, memcache starts returning "false" for every request (get, set, increment)

If at this stage we revert back to the old code, things start working fine again.

The request rate on our memcached server is about 270 requests per second (with the old code). This is expected to grow to over 1000 requests per second with the new code.

When memcache starts returning "false", around 15% of the allocated memory is free.

What could be causing this behaviour?

What is the command used to start the memcached server? Maybe it's a max memory problem, you say when it starts returing false, there is only 15% memory free, is that overall OS memory, or just the memory allocated for memcached? — capi
We start memcached server using the command: /usr/bin/memcached -d -m 2048 -u root. When it starts returning "false", 15% of the memory allocated to memcached is free. Please correct me if I am wrong, but even in case there was no free memory, shouldn't memcached evict an older item when we issue the set command. The response to set should not be "false" in such a case. Thanks. — Nikhil
Try looking at the eviction rates using munin or phpmemcachedadmin (really easy to install, link in my post below). Might offer more information on what's going on inside memcached. You can also drill down to slabs and see what items are stored and their values. As a last resort, try using the Memcached library, it has more runtime options, and the code is similar, might not be such a big change if you have a wrapper already in place. — capi
I have installed phpmemcachedadmin. Thanks for introducing me to this tool. It turns out that what was shown as free memory in livebookmark.net/journal/2008/05/21/… actually is wasted memory. Also, too many pages allocated to slabs with higher chunk size, resulting in too many evictions in the first few slab classes. Though I still cannot understand why set command returns false in my case. I will try using the Memcached library from PECL. — Nikhil
Increasing memcached server threads to 8 (from default 4) fixed the issue. The set command no longer returns false. — Nikhil

capi capi · Accepted Answer · 2012-03-29T08:29:03

Looks like it could be a compression or serialization problem. I would suggest doing more debugging to see the exact server response. First, try doing a telnet and setting and getting the keys manually :

telnet a.b.c.d 11211
SET key_name 0 300 3
123
STORED
GET a
VALUE a 0 3
123
END

See :

http://code.google.com/p/memcached/wiki/NewCommands http://code.sixapart.com/svn/memcached/trunk/server/doc/protocol.txt

for protocol commands.

If all looks well you could try a monitor on the server to see the commands being executed :

telnet a.b.c.d 11211
STATS detail on
//wait a while
STATS detail off
STATS detail dump
//list of commands will be dumped

You can tweak the library a bit by its runtime configuration in php.ini :

http://www.php.net/manual/en/memcache.ini.php

And maybe you could check out PHP's Memcached (notice the D) library as an alternative.

Also, a good monitoring tool :

http://code.google.com/p/phpmemcacheadmin/

PHP memcache returns false with increase in number of keys

1 Answers