Storm supervisors,workers allocation of Memory

4
votes

I'm having storm cluster with 3 Nodes: supervisors-2, Nimbus-1. 3 Nodes has 30 Gb of Ram and 4 cpu-cores. In configs I had configured: 

supervisor.slots.ports : 8 (6700-6708)
supervisor.childopts  :-Xmx4096m _JAAS_PLACEHOLDER 
ui.childopts :-Xmx1024m _JAAS_PLACEHOLDER
nimbus.childopts :-Xmx6144m _JAAS_PLACEHOLDER

I want to know:

  1. How this slots acts on supervisor nodes? In my opinion for each supervisor node we are assigning 8 ports . That means how many workers I will get for the cluster?
  2. For supervisor childopts it is configured that 4 Gb of Ram. what is this means? similarly nimbus childopts?
  3. suppose if I am running a topology with 3 workers by this configuration what is the Ram used and CPU cores used? How can I calculate Ram utilisation and CPU for particular Topology
2
apache-storm

2 Answers

2
votes
  1. 8 ports -> 8 workers. Each worker is assigned one of the ports.
  2. supervisor.childopts are passed to the supervisor process. Same goes for nimbus.childopts.
  3. workers may be running on different machines. Each worker is a separate process. You can note the memory/cpu of each process.
0
votes

To be able to set requested resources(RAM & CPU) by each component(bolt/spout) and available resources on each node, you can use this scheduler. this way you can specify available RAM & CPU on each node and determine how much of each resource should be allocated to each worker process.

Requested Resources by Components

setMemoryLoad(Number onHeap):

builder.setBolt("exclaim1", new ExclamationBolt(), 3)
        .shuffleGrouping("word").setMemoryLoad(512.0);

setCPULoad(Double amount):

builder.setBolt("exclaim2", new HeavyBolt(), 1)
            .shuffleGrouping("exclaim1").setCPULoad(450.0);

Available Resources in worker nodes

add this parameters to storm.yaml

supervisor.memory.capacity.mb: [amount<Double>]

supervisor.cpu.capacity: [amount<Double>]

for more details on how Scheduling process is done, you can read this article