Storm: What happens with multiple workers?

Question

Say I deploy a topology with 2 workers, the topo has 1 spout and 1 bolt with 2 tasks. Then my understanding is, 1 worker will run spout executor and 1 bolt executor, the other worker will run 1 bolt executor.

Is my understanding correct?

If my understanding is correct, then my question comes. Say the bolt is implemented by Python. Since storm transfers data between multi-lang bolts via stdout/stdin, if the 2 workers run on different hosts, how spout can send data to bolt that locates on the other host?

Answer 1

Little more clarification to your question. Storm uses various types of queue for data/tuple transfer between various components of topology

Example :

1) Intra-worker communication in Storm (inter-thread on the same Storm node): LMAX Disruptor

2) Inter-worker communication (node-to-node across the network): ZeroMQ or Netty

3) Inter-topology communication: nothing built into Storm, you must take care of this yourself with e.g. a messaging system such as Kafka/RabbitMQ, a database, etc.

For further reference :

http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/

Answer 2

To give a more detailed answer:

Storm will sent the data to both bolt executors. For the spout-local bolt, this happens in-memory; for the other bolt via network. Afterwards, each bolt-instance will deliver the input to an local-running python process. Thus, your describe stdout/stdin delivery happens locally on each machine. The data is transfer to each bolt before the data delivery from Java to Python happens.

Thus, stdout/stdin bridge is used within each bolt, and not from spout to bolt.

Answer 3

I have done a test by myself. Storm can properly deliver spout emitted data to bolts on different hosts.