We have been moving our apis from python to node. We have used task queues with our Python app engine apis. With node.js now supported on app engine, do you suggest we use task queues or cloud pub/sub for tasks? What would the pros / cons be for each including reliability, portability, etc.
1 Answers
14
votes
Task queues and Pub/Sub have different messaging models, and thus are useful for different things. While Pub/Sub uses the publisher-subscriber model, Task Queue collects and distributes tasks to workers.
Using Pub/Sub to distribute tasks would not be as natural as using task queue. All subscribers to a Pub/Sub topic see all messages, whereas in task queue, a message is only consumed by one receiver. On the flip side, task queues aren't designed with the purpose of broadcasting messages.
For the moment, task queues work best on App Engine. Though there is an experimental Task Queue REST API, task queues are native to and operate best on App Engine. Pub/Sub can be used from anywhere. While task queue tasks are distributed to App Engine instances automatically, you'd need to do additional setup for using Pub/Sub for intra-application communication.
Task queues also provide special features, such as delaying messages.
Edit:
Unfortunately it looks like push task queues aren't ready yet on the Node.js App Engine runtime.
The Pub/Sub solution Kamal points out below is a nice trick for simulating pull-based load balancing. Using Pub/Sub is probably the right way to go if you are OK with pull mode, though I would like to make it clear that an AE taskqueue API is available (https://github.com/google/google-api-nodejs-client/tree/master/apis) for node.js when using pull.
One thing to consider whether you use pull or push is that the retry policy is slightly different. Task queue items can be configured to retry indefinitely, while I think Pub/Sub will stop trying to deliver a message if it was not acknowledged after 7 days.
Other things to consider if you decide to use PubSub push mode are:
- Some feature of AE task queue such as delaying messages or configuromg a message with specific retry or time to live settings are not available in PubSub
- You'll need to know the endpoints for the instances you want consuming tasks from the queue.
On a separate note (in response to the first response below), it does look like memcache integration is supported. It has a docs page located at https://cloud.google.com/appengine/docs/flexible/nodejs/caching-application-data.
