Laravel Daemon Queue Memory Leak

6
votes

I am using laravel 5.1 and using supervisor to monitor the queue job. Queue Driver is Database.

[program:queue]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/artisan queue:work database --sleep=3 --tries=1 --daemon
autostart=true
autorestart=true
user=root
numprocs=1
redirect_stderr=true
stdout_logfile=/var/www/html/storage/logs/supervisord.log

RAM used by Queue Listener increases after processing each job and it goes up to 150-200 MB. All global variables are assigned null.

namespace App\Jobs;
use App\Jobs\Job;
use App\Compatibility;
use App\Like;
use App\Message;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Bus\SelfHandling;
use Illuminate\Contracts\Queue\ShouldQueue;

class CalculateInteractionLike extends Job implements SelfHandling, ShouldQueue
{
    use InteractsWithQueue, SerializesModels;
    protected $userAId;
    protected $userBId;
    protected $gender;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct($userAId, $userBId, $gender)
    {
        $this->userAId = $userAId;
        $this->userBId = $userBId;
        $this->gender = $gender;
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        echo 'start CalculateInteractionLike '. memory_get_usage() . "\n";
        $userArray = array();
        Compatibility::where('userAId', $this->userBId)->where('userBId', $this->userAId)->update(['interactionAB'=>0.0001]);
        $profiles = Compatibility::where('userAId', $this->userBId)->where('interactionAB', '>',0)->orderBy('compatibilityAB', 'desc')->take(4)->get(['compatibilityAB']);
        $compatible = array();
        for($i=0; $i<sizeof($profiles);$i++){
            $compatible[] = $profiles[$i]->compatibilityAB;
        }
        $std = sizeof($compatible)>1 ? $this->_calculateStd($compatible) : 0;
        $messagedProfile = Message::where('userBId', $this->userBId)->where('type', '1')->get(['userAId']);
        for($i=0;$i<sizeof($messagedProfile);$i++){
            Compatibility::where('userAId',$this->userBId)->where('userBId', $messagedProfile[$i]->userAId)->update(array('interactionAB' => $std));
        }
        $this->userAId = null;
        $this->userBId = null;
        $this->gender = null;
        $userArray = null;
        $profiles = null;
        $compatible = null;
        $std = null;
        $messagedProfile = null;

    }

    private function _calculateStd($compatible){
        return sqrt(array_sum(array_map([$this,"_stdSquare"], $compatible, array_fill(0,count($compatible), (array_sum($compatible) / count($compatible))))) / (count($compatible)-1));
    }

    private static function _stdSquare($x, $mean){
        return pow($x - $mean, 2);
    }

    public function __destruct(){
        $this->cleanup();
        echo 'end CalculateInteractionLike '. memory_get_usage() . "\n";
    }

    public function cleanup() {
        //cleanup everything from attributes
        foreach (get_class_vars(__CLASS__) as $clsVar => $_) {
            unset($this->$clsVar);
        }
    }
}

If above job is processed, each time there is some increase in RAM. Any Idea ?

2
laravelqueuedaemon
How much memory does it consume when the process has started? – Andrej Ludinovskov
It continuously increases and goes to 190-200 MB. After that I restarted it. – Sachin Singh
memory=128 means that the worker will be restarted automatically after at least one job will be processed but not before. – Andrej Ludinovskov
yeah, right. I added memory=128 after I saw that the memory goes to 190-200 MB. – Sachin Singh
In theory, the class actually gets destroyed after handling the job, therefore there should be no change by cleaning all variables. I would look more into the size of the arrays and collections you are handling there. Could it be that is not actually a memory leak, but instead bigger size on your lists of elements? Why don't you also Log the size of the collections/arrays? Also, can you tell us your OS version, PHP Version, supervisor version? if there is a memory leak is more likely that is a problem of the specific implementation of one of those or one library. – Nestor Mata Cuthbert

2 Answers

1
votes

As a work around - if you cannot find the source of the memory leak - you can just switch to queue:listen without any daemon flag.

This will "reboot" the framework after each job, releasing all memory by PHP.

This is compatible with Supervisor, which will ensure the queue:listen is always restarted.

0
votes

The only place you are not cleaning the memory is inside for()

        for($i=0;$i<sizeof($messagedProfile);$i++){
            Compatibility::where('userAId',$this->userBId)->where('userBId', $messagedProfile[$i]->userAId)->update(array('interactionAB' => $std));
        }

the first where is static (::used) but internally objects created and you start to use -> to call the functions. Did you try to check if there are desctuctors and manually call garbage collector?

 for($i=0;$i<sizeof($messagedProfile);$i++){
            Compatibility::where('userAId',$this->userBId)->where('userBId', $messagedProfile[$i]->userAId)->update(array('interactionAB' => $std));
            gc_collect_cycles();
        }

The second thing the where(), update() etc can return reference on object so if yes you would like to try:

            for($i=0;$i<sizeof($messagedProfile);$i++){
                $tmp = Compatibility::where('userAId',$this->userBId)->where('userBId', $messagedProfile[$i]->userAId)->update(array('interactionAB' => $std));
$tmp = null;
            }

Please note that php < 7.0 is not automatically destruct the objects of the classes that were created inside a upper class. Check this: https://codereview.stackexchange.com/questions/57847/case-of-the-hidden-memory-leak