We want to build a master slave pattern on Google Cloud. We planned to use Pub/Sub for that (similar to JMS pattern) letting each worker to grab a task from the queue and ack when done.
But, it seems like a subscriber can't get messages sent before it started. And we're not sure how to make sure each message will be processed by a single 'slave'.
Is there a way to do it? Or another mechanism on google cloud for that?
As far as I understand the master slave pattern, the slaves do the tasks in parallel and the master harvest the result. I'd create a topic for queuing the tasks, and a single subscription attached to this topic, so that all the slaves use this subscription to fetch the task.
Also I'd create another topic/sub pair for publishing results from slaves and the master harvest the result. Alternatively the result can be stored into shared datastore like Cloud Datastore.
You can do this by creating 'single' subscription which is than used by all the slaves. pubsub service delivers new message only once to given subscription so you can be sure that given message will be processed only by 1 slave.
You can also adjust acknowledgement deadline appropriately so that delivery retry doesn't happen. If retry happens than it will result in multiple slaves getting same message.
