Does a port in supervisor.slots.ports translate to one physical JVM?

Question

supervisor.slots.ports: "For each worker machine, you configure how many workers run on that machine with this config. Each worker uses a single port for receiving messages, and this setting defines which ports are open for use. If you define five ports here, then Storm will allocate up to five workers to run on this machine."

And from the storm concepts:

Workers: Topologies execute across one or more worker processes. Each worker process is a physical JVM and executes a subset of all the tasks for the topology.

My storm.yaml defines:

supervisor.slots.ports:
  - 6700
  - 6701
  - 6702
  - 6703

And then I run a topology with topology.workers set to 3 (kafka-spout-parallelism set to 1 and solr-bolt-parallelism set to 2, no other bolts).

Storm-ui also shows that my topology is running fine with 1 spout and 2 bolts.

But when I login to the storm machines and run ps -aef | grep storm or jps -l, I do not see the JVM processes for workers anywhere. Only processes I see are:

Machine 1:

jps -l
30675 backtype.storm.daemon.supervisor
30583 backtype.storm.daemon.logviewer

Machine 2:

jps -l
6818 backtype.storm.ui.core
6995 backtype.storm.daemon.supervisor
6739 backtype.storm.daemon.nimbus
6904 backtype.storm.daemon.logviewer

Does storm not create one physical JVM per worker? And does that not translate to one JVM per port mentioned in supervisor.slots.ports?

P. Taylor Goetz P. Taylor Goetz · Accepted Answer · 2016-10-13T22:47:33

One worker slot equates to one JVM slot. Slots are only taken up when a topology is deployed.

It's possible you checked the processes before the workers actually started. Check the Storm UI to make sure the topology is up, running, and processing data. If it's not, then use the log viewer to look for errors. It is possible the workers are crashing due to uncaught exceptions.

Does a port in supervisor.slots.ports translate to one physical JVM?

1 Answers