Node Worker Threads vs Heroku Workers

Question

I'm trying to understand difference between Node Worker Threads vs Heroku Workers.

We have a single Dyno for our main API running Express.

Would it make sense to have a separate worker Dyno for our intensive tasks such as processing a large file.

worker: npm run worker

Some files we process are up to 20mb and some processes take longer than the 30s limit to run so kills the connection before it comes back.

Then could I add Node Worker Threads in the worker app to create child processes to handle the requests or is the Heroku worker enough on its own?

Answer 1

It may make more sense for you to keep a single dyno and scale it up, which means multiple instances will be running in parallel.

See https://devcenter.heroku.com/articles/scaling

Answer 2

After digging much deeper into this and successfully implementing workers to solve the original issue, here is a summary for anyone who comes across the same scenario.

Node worker threads and Heroku workers are similar in that they intend to run code on separate threads in Node that do not block the main thread. How you use and implement them differs and depends on use case.

Node worker threads

These are the new way to create clustered environments on NODE. You can follow the NODE docs to create workers or use something like microjob to make it much easier to setup and run separate NODE threads for specific tasks.

https://github.com/wilk/microjob

This works great and will be much more efficient as they will run on separate worker threads preventing I/O blocking.

Using worker threads on Heroku on a Web process did not solve my problem as the Web process still times out after a query hits 30s.

Important difference: Heroku Workers Do not!

Heroku Workers

These are separate virtual Dyno containers on Heroku within a single App. They are separate processes that run without all the overhead the Web process runs, such as http.

Workers do not listen to HTTP requests. If you are using Express with NODE you need a web process to handle incoming http requests and then a Worker to handle the jobs.

The challenge was working out how to communicate between the web and worker processes. This is done using Redis and Bull Query together to store data and send messages between the processes.

Finally, Throng makes it easier to create a clustered environment using a Procfile, so it is ideal for use with Heroku!

Here is a perfect example that implements all of the above in a starter project that Heroku has made available.

https://devcenter.heroku.com/articles/node-redis-workers