How does Storm manage sharing data on cluster mode?

3
votes

recently I'm developing some timing tools for storm topology, but I still have some questions about sharing data in storm cluster:

  1. If a component(spout/bolt) is configured with more than one executors per worker, say the worker number is one, the parallelism_hint of the component is 3 and the task number uses default setting(i.e. 1), does that mean there are 3 instances of the component in the worker? If not, should the field of the component be used in a synchronized block?

  2. If an additional thread named "athread" is created in a component(within prepare() or open() method), how many "athread" instances are there in the storm cluster?

  3. As Understanding the Parallelism of a Storm Topology says, a worker is a separate process, and a worker process executes a subset of a topology. Does that mean global variables (such as public static fields or other static variables) of the topology can only be shared in one worker?

  4. If a spout's parallelism_hint is configured greater than 1, and there is a Utils.sleep(1000) sentence in nextTuple() method, does that mean the number of emitted tuples of the spout is equal to the executors'(threads) number of the spout every second?

Thanks very much.

1
distributed-computingapache-storm
If the paralelism_hint is set to 3 then you will have 3 instances of your bolt spread across workers. For instance, if you have one worker, then you will have 3 instances into the same JVM. However, if you have 3 workers you will have one instance per JVM. So if you create a new thread into the open()/prepare() method you will have as many threads as bolt instances. Yes, static fields are shared among components running on the same worker. - fhussonnois
Yes, you are right but by default Storm instantiates for a specific component as many instances as executors if NumTask is not set. - fhussonnois
@fhuz parallelism hint specify the number of executor aka threads to be allocated not the instance. Your statement holds true only in case you do not configure no of tasks as in that case storm will run 1 tasks per executor (setting parallelism hint = 2 will create 2 tasks by default unless configured explicitly), otherwise it is possible to run a component having one tasks with multiple executors. - user2720864
yes true . sorry I missed the comment before updating mine - user2720864
Thanks @fhuz . But I still wonder about what would happened to the instances of the component and the related threads it created when a topology is rebalanced? - Yohn

1 Answers

2
votes

1) Setting parallelism hint = 3 will ask storm to allocate 3 executors ( Threads ) and will create 3 tasks by default( Please note if you do not explicitly configure the number of tasks storm will run one tasks per executor by default) for that component. If you set the no task = 1 , then three threads will be operating on the same component instance.

2) Assuming you are running with default settings ( 1 executor & 1 task/component) it will create one single instance of that thread object as the prepare/open method will only be called once.

3) Static variables will be shared by all the instances of a component within a given worker.

4) Not sure what exactly you mean, if you are running with multiple executors and one tasks then when Thread-A is sleeping another Thread-B might be processing the tuple