Remove delay in Laravel Queue

0
votes

I'm fan of Laravel-Lumen frameworks. These are so good and nice designed and we can start to implement the apps. But there is a small problem. May be this isn't a problem but in my opinion this is a small problem. Let me explain it.

For example I have a model and I'm using elastic search. When a model created (insert to db) then I'm dispatching a job for indexing this model.

public static function boot()
{
    parent::boot();

    static::created(function ($model) {
        $modelClass = get_class($model);
        lgi('>>> model created:' . $modelClass, $model);

        dispatch(new ElasticIndexerJob('creted', $model));
    });

    static::updated(function ($model) {
        $modelClass = get_class($model);
        lgi('>>> model updated:' . $modelClass, $model);

        dispatch(new ElasticIndexerJob('updated', $model));
    });

    static::deleted(function ($model) {
        $modelClass = get_class($model);
        lgi('>>> model deleted:' . $modelClass, $model);

        dispatch(new ElasticIndexerJob('deleted', $model));
    });
}

This outputs that log:

[2021-04-18 11:28:36] local.INFO: /app/Models/Traits/Indexable.php:25 [
    ">>> model updated:App\\Models\\City",
    {
        "id": 3,
        "country_id": 85,
        "state_id": 1,
        "zip_code": "87506",
        "name": "Nambe",
        "lat": 35.8890389,
        "lng": -106.0657318,
        "status": "passive",
        "created_at": "2021-04-13 09:38:09",
        "updated_at": "2021-04-18 11:28:36"
    }
]

The time is 11:28:36. After that I'm looking to queue log output.

[2021-04-18 11:28:38][27] Processing: App\Jobs\ElasticIndexerJob
[2021-04-18 11:28:38] local.INFO: /app/Jobs/ElasticIndexerJob.php:32 [
    "App\\Models\\City",
    "updated",
    {
        "id": 3,
        "country_id": 85,
        "state_id": 1,
        "zip_code": "87506",
        "name": "Nambe",
        "lat": 35.8890389,
        "lng": -106.0657318,
        "status": "passive",
        "created_at": "2021-04-13 09:38:09",
        "updated_at": "2021-04-18 11:28:36"
    }
]  
[2021-04-18 11:28:38][27] Processed:  App\Jobs\ElasticIndexerJob

Queue output time is 11:28:38. As you can see there is a 2 seconds delay (or difference). This problem is occuring all queues (sending mails, executing other jobs etc...). When sending mails this isn't a problem but sometimes I need realtime execution. I want to execute job immediatelly when I dispatch.

Normally if I use beanstalkd myself then I can receive messages (jobs) immediatelly and I can make something with that message. But in Laravel (or Lumen) there is a delay and this is annoying me.

Why Laravel has delay and how can I remove this delay?

I tested this in database queue and beanstalkd queue. Same behaviour occurs:

QUEUE_CONNECTION=beanstalkd
; QUEUE_CONNECTION=database

Note: lgi() function is an envelope of Log::info()

2
laravelqueuebeanstalkd
If you need realtime execution, why do you queue it in the first place? – Aless55
So do you mean that: queue's purpose is not executing something in realtime? – kodmanyagha
laravel.com/docs/8.x/notifications#queueing-notifications In my understanding this is used for tasks that take a longer time, for example if you have to send 10000 notifications at once – Aless55
Yes you're correct. We can use queue for sending push notifications or converting a large video to different resolutions or anythink like that. But i think it must start immediatelly. – kodmanyagha

2 Answers

0
votes

Have you tried dispatchSync, I don't know if that's available on Lumen. Laravel dispatchSync

0
votes

I found the solution. I look to Laravel's source code and I see that code in this file: Illuminate\Queue\Listener

/**
 * The amount of seconds to wait before polling the queue.
 *
 * @var int
 */
protected $sleep = 3;

I investigated a little more deeply and I found that:

/**
 * Create the command with the listener options.
 *
 * @param  string  $connection
 * @param  string  $queue
 * @param  \Illuminate\Queue\ListenerOptions  $options
 * @return array
 */
protected function createCommand($connection, $queue, ListenerOptions $options)
{
    return array_filter([
        $this->phpBinary(),
        $this->artisanBinary(),
        'queue:work',
        $connection,
        '--once',
        "--name={$options->name}",
        "--queue={$queue}",
        "--backoff={$options->backoff}",
        "--memory={$options->memory}",
        "--sleep={$options->sleep}",
        "--tries={$options->maxTries}",
    ], function ($value) {
        return ! is_null($value);
    });
}

As you can see there is a CLI param which is --sleep=[second]. We can set this value to zero (0). After that queue worker not waiting for process the next job.

php artisan queue:work --sleep=0.1

I think these parameters must be in documentation but Laravel developers didn't add this. Or may be I missed this I don't know.

Edit: If you're using database as queue backend then this usage can load up your db. Becouse db driver is sending SELECT query to jobs table. I suggest that use beanstalkd driver with this usage.

Edit 2: If you set --sleep to 0 (zero) then it may eat the cpu.

Edit 3: I see this now. Probably I must go to a ophthalmologist :S https://laravel.com/docs/8.x/queues#worker-sleep-duration