when using laravel queue job processing, getting stuck

Question

i am on 5.3.31

so it cannot be related to

https://github.com/laravel/framework/issues/15179

after 300ish jobs I get:

[2017-04-11 13:51:53] local.ERROR: Symfony\Component\Debug\Exception\FatalThrowableError: Call to a member function beginTransaction() on null in /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Database/Connection.php:612
Stack trace:
#0 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php(175): Illuminate\Database\Connection->beginTransaction()
#1 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(175): Illuminate\Queue\DatabaseQueue->pop('default')
#2 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(145): Illuminate\Queue\Worker->getNextJob(Object(Illuminate\Queue\DatabaseQueue), 'default')
#3 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(75): Illuminate\Queue\Worker->runNextJob('database', 'default', Object(Illuminate\Queue\WorkerOptions))
#4 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(100): Illuminate\Queue\Worker->daemon('database', 'default', Object(Illuminate\Queue\WorkerOptions))
#5 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(83): Illuminate\Queue\Console\WorkCommand->runWorker('database', 'default')
#6 [internal function]: Illuminate\Queue\Console\WorkCommand->fire()
#7 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Container/Container.php(508): call_user_func_array(Array, Array)
#8 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Console/Command.php(169): Illuminate\Container\Container->call(Array)
#9 /var/www/html/www.myapp.com/vendor/symfony/console/Command/Command.php(256): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#10 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Console/Command.php(155): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /var/www/html/www.myapp.com/vendor/symfony/console/Application.php(820): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /var/www/html/www.myapp.com/vendor/symfony/console/Application.php(187): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /var/www/html/www.myapp.com/vendor/symfony/console/Application.php(118): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /var/www/html/www.myapp.com/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(121): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /var/www/html/www.myapp.com/artisan(35): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 {main}

this error keeps repeating

the jobs are stuck, nothing advances, nothing gets moved to failed-queue neither

the only solution is that I have found is stopping the supervisor workers via cronjob every couple minutes, and starting them again. Which is far from optimal

seems as well as if the whole server disappears once in a while quickly. Maybe a memory problem? i have 1 gb of memory for my live server.

I had a similar issue with beanstalkd queues. After a few hundred in it would just stop firing. I had supervisor running and nothing. The only thing that would get them working again was a server reboot. I moved over to Amazon SQS. The messages aren't in any order, but they do get fired consistently. It is pretty straight forward to get it running as well. — whoacowboy
i saw my server running out of memory actually when running the jobs... — Toskan
Post your schedule function from app\Console\kernel.php and how you run the cron job. — Dimitri Mostrey

Laurence Laurence · Accepted Answer · 2017-05-06T22:51:05

It is probably one of two things:

1. Memory Leak

If you are running your queues as a daemon process, then it is possible that over time they will run out of memory. If this occurs, you have three choices.

Find a stop the memory leak
Switch to using queue:listen to ensure PHP restarts each time
Run a scheduler that includes $schedule->command('queue:restart')->hourly(). This gives you the best of both worlds, because you get the benefit of a daemon process, and just restart it once an hour to allow PHP to reset.

2. Failed database connection

There is currently a known issue that if a DB connection goes away during a daemon process, it may not reconnect. A PR is in process here - so check if this gets accepted and the bug should be fixed: https://github.com/laravel/framework/pull/19080

when using laravel queue job processing, getting stuck

1 Answers