Laravel Artisan Queues - high cpu usage

23
votes

I have set up queues in Laravel for my processing scripts. I am using beanstalkd and supervisord. There are 6 different tubes for different types of processing.

The issue is that for each tube, artisan is constantly spawning workers every second. The worker code seems to sleep for 1 second and then the worker thread uses 7-15% cpu, multiply this by 6 tubes... and I would like to have multiple workers per tube.. my cpu is being eaten up.

I tried changing the 1 second sleep to 10 seconds. This helps but there is still a huge cpu spike every 10 seconds when the workers wake back up. I am not even processing anything at this time because the queues are completely empty, it is simply the workers looking for something to do.

I also tested to see the cpu usage of laravel when I refreshed the page in a brower and that was hovering around 10%.. I am on a low end rackspace instance right now so that could explain it but still... it seems like the workers spin up a laravel instance every time they wake up.

Is there no way to solve this? Do I just have to put a lot of money into a more expensive server just to be able to listen to see if a job is ready?

EDIT:

Found a solution... it was to NOT use the artisan queue:listener or queue:work I looked into the queue code and there doesn't seem to be a way around this issue, it requires laravel to load every time a worker checks for more work to do.

Instead I wrote my own listener using pheanstalk. I am still using laravel to push things into the queue, then my custom listener is parsing the queue data and then triggering an artisan command to run.

Now my cpu usage for my listeners is under %0, the only time my cpu shoots up now is when it actually finds work to do and then triggers the command, I am fine with that.

3
laravellaravel-4supervisordbeanstalkd
Would you mind sharing your code to the custom listener? I'm having the same issue. – greatwitenorth
I'm also interested to see your custom listener! – Bouke Versteegh
I'd like to see your solution as well – Matt Foxx Duncan
Could you please share your code ? or maybe point us to the right direction ? – Leeeeeeelo

3 Answers

14
votes

The problem of high CPU is caused because the worker loads the complete framework everytime it checks for a job in the queue. In laravel 4.2, you can use php artisan queue:work --daemon. This will load the framework once and the checking/processing of jobs happen inside a while loop, which lets CPU breathe easy. You can find more about daemon worker in the official documentation: http://laravel.com/docs/queues#daemon-queue-worker.

However, this benefit comes with a drawback - you need special care when deploying the code and you have to take care of the database connections. Usually, long running database connections are disconnected.

5
votes

I had the same issue.

But I found another solution. I used the artisan worker as is, but I modified the 'watch' time. By default(from laravel) this time is hardcoded to zero, I've changed this value to 600 (seconds). See the file: 'vendor/laravel/framework/src/Illuminate/Queue/BeanstalkdQueue.php' and in function 'public function pop($queue = null)'

So now the work is also listening to the queue for 10 minutes. When it does not have a job, it exits, and supervisor is restarting it. When it receives a job, it executes it after that it exists, and supervisor is restarting it.

==> No polling anymore!

notes:

  • it does not work for iron.io queue's or others.
  • it might not work when you want that 1 worker accept jobs from more than 1 queue.
2
votes

According to this commit, you can now set a new option to queue:listen "--sleep={int}" which will let you fine tune how much time to wait before polling for new jobs.
Also, default has been set to 3 instead of 1.