How to queue a non-existing worker

0
votes

I'm trying to queue a Sidekiq worker that does not exist in my project but exists in some other project sharing the same redis server.

In our current situation, we have two servers:

  • API server runs on Heroku. Web instance for the REST API and worker with Sidekiq listening to the :api queue.

  • Processing server runs on AWS ElasticBeanstalk with a puma instance running a REST Full API with Sinatra. This instance runs Sidekiq and listens to the queue :processing

Side note: In the beginning, our API was doing all the work but as some workers started taking too much memory, we switch to the processing server.

When we need to execute a job on the processing server we have these actions happening:

  1. API calls POST processing.com/some_job

  2. Processing queues SomeJobWorker on queue processing

  3. Processing executes SomeJobWorker from queue processing

  4. Processing calls POST api.com/webhooks/some_job_result

  5. API queues SomeJobResultWorker on queue api

  6. API executes SomeJobResultWorker from queue processing

This got me thinking if I could somehow, from the API server directly queue SomeJobWorker then I could get rid of the REST API on our processing server.

... then I could get rid of the webhooks endpoint on the API server and do the same from the processing server

I would end up with something like this:

  1. API calls queues SomeJobWorker

  2. Processing executes SomeJobWorker

  3. Processing queues SomeJobResultWorker

  4. API executes SomeJobResultWorker

Is there a way for me to queue a worker that does not exist on my local code base?

1
ruby-on-railsrubysidekiq

1 Answers

1
votes

You have two shards. Each Redis is a shard. Read this page:

https://github.com/mperham/sidekiq/wiki/Sharding

You can push a job using the class name:

client = Sidekiq::Client.new(PROCESSING_REDIS)
client.push('queue' => 'api', 'class' => 'FooWorker', 'args' => [1,2,"foo"])