Why are my code changes not reflected immediately?

Question

I am maintaining a large wordpress site and I am attempting to troubleshoot an unrelated problem by adding trace statements in the code which are nothing more than calls to error_log(). In brief my server setup is a file server which shares out the website's document root via nfs. There are 2 web servers which are running nginx+apc+php/fpm. I am editing the php files directly on the file server and when I view the files from the web servers i can see my changes however when I tail the error log i can observe that my changes are not immediately reflected. I continue to see old trace statements as if I hadn't made the change at all.

My running theory is that the code is getting cached either in apc (duh) or in a client-side nfs cache (unlikely since i can see the edits with vi). I try to mitigate this by flushing the apc cache using a script which runs apc_clear_cache(). Additionally I have restarted nginx as well as php-fpm hoping that something will flush the old cached php code. None of these methods have worked and I must resort to waiting up to an hour or more before i see my code changes reflected in the logs.

The website I am troubleshooting is fairly high traffic so remounting the nfs share is not really an option for me. My assumption is that the apc opcode cache is not actually clearing but I have been watching the stats with apc.php and I do see what looks like the cache being rebuilt after i run my flush script. I've been at this for a couple days and troubleshooting a simple problem has turned into a huge headache. Can anyone provide any ideas of other things to look at or try to make my code changes more immediate?

Why are you editing files directly on a high traffic production server? — GolezTrol
Because the issue i'm troubleshooting does not occur in the sandbox environment, only on the live server. — sphoid
So you're clearing the apc cache and restarting nginx on both web servers... You might be able to narrow your search by inserting some invisible HTML in your PHP. e.g.  If that shows up when you view source on the front-end then you know your new code is running and you can rule out APC and nginx. Could it be that writes to the error log are just being buffered so you see old entries appear when you tail the log? — John Watson

Arend Arend · Accepted Answer · 2011-11-15T23:59:06

Some thoughts:

Temporarily setting the apc.stat to true might be useful if you're tweaking; this means a small penalty in performance but will force APC to recheck the opcache before each request.
This is my shotgun method to refresh the APC cache (forcing the invalidation of each individual file).
Is FPM always a shared cache (the manual seems to state so); or are there more caches? (Not sure! I'm not very familiar with FPM).

    $info = apc_cache_info();
    $files = $info['cache_list'];
    $prefix = "/";
    if (isset($_GET['PREFIX']))
    {
        $prefix = $_GET['PREFIX'];
    }
    $user = apc_cache_info("user");
    $cachestore = array();
    // save the user cache to an array
    foreach ($user['cache_list'] as $info)
    {
        $cachestore[$info['info']] = apc_fetch($info['info']);
    }
    apc_clear_cache();
    apc_clear_cache('user');
    // Recache all the files that were in the opcache before..
    foreach ($files as $file)
    {
        if (strpos($file['filename'], $prefix) === 0)
        {
            print $file['filename'] . " : ";
            print apc_compile_file($file['filename']) ? "SUCCESS\n" : "FAILE             D\n";
        }
    }

    foreach ($cachestore as $key => $value)
    {
        apc_store($key,$value);
    }

Why are my code changes not reflected immediately?

2 Answers