Can twitter storm worker get only parts of the topology?

2
votes

Supposing that you have the following topology

spout

spout (sends tuple to )bolt1
bolt1 (sends tuple to) bolt2

and the following settings:

3 process workers

spout,parallel hint=1
bolt1, parallel hint=2
bolt2,parallel hint=3

my questions are:

  • Am I right with the following (possible) distribution among the workers:

    Worker1                  
   spout1
   bolt 2 
Worker2
   bolt 1
   bolt 2
Worker3
  bolt1
  bolt 2
  • In the above conditions, will the bolt1 (from worker2, worker3 ) get any tuple from spout?
  • will the bolt 2 from the worker 1 get any data from bolt 1?
  • Do we have to keep a proportional rate between number of workers and spouts/bolts in order that each of the worker to get a a least a spout/or bolt?
2
twitterapache-storm

2 Answers

2
votes

You need to differentiate between how the different tasks are distributed among your workers and the grouping between your spouts and bolts. The grouping decides how tuples are routed in your topology. For instance if you use an all grouping between spout and bolt1 every task in bolt1 will receive every tuple sent by the spout. Storm has many different groupings, I suggest that you read storms concept page carefully this should answer your questions: https://github.com/nathanmarz/storm/wiki/Concepts.

2
votes

In short, you don't need to worry about which worker is handling a particular task. Storm takes care of routing the messages around so that you don't need to pay attention to it. In this case, everything will work as expected: the spout will send tuples to bolt1 workers and botl1 workers will send tuples on to bolt2 workers, no matter where they are physically located in the cluster.

That said, you are somewhat incorrect about the distribution of tasks among workers. A worker can only contain tasks for one bolt or spout at a time. A supervisor may be running multiple workers, though, in which case you will have multiple tasks of different bolts or spouts running on the same hardware, but not in the same process.