Storm cluster not working in Production mode

Question

I have a storm topology which in two nodes. One is the nimbus and the other is the supervisor.

A proxy which is not part of storm accepts an HTTP request from a client and passes it to the storm topology.

The topology is like this: 1. The proxy passes data to a storm spout. 2. The spout passes data to multiple bolts. 3. The result is passed back to the proxy by the last bolt.

I am running the proxy and passing data to storm. I am able to connect a socket to the listener at the topology side. The data emitted by the spout is shown to be 0 in the UI. The same topology works fine in a local mode.

Thought it was a problem with supervisor, but the supervisor seems to be running fine because I am able to see the supervisor description and the individual spouts and bolts. But none of them emit anything.

Now, I am confused if the problem is the data being passed to the wrong machine or something. In order to communicate to the spout, Im creating the socket from the proxy as follows:

            InetAddress stormInetAddr=InetAddress.getByName("198.18.17.16");
            int stormPort=4321;                 
            Socket stormSocket=new Socket(stormInetAddr,stormPort);

Here 198.18.17.16 is the nimbus IP. And 4321 is the port where data is being expected.

I tried giving the supervisor IP here, and it didnt connect. However, this does. Now the proxy waits for the output on a specific port.

On the other side, after processing, data is read from the bolt. And there seems to be no activity from the cluster. But, I am getting a response which is basically the same request I had sent with some jumbled up data. And this response is supposed to be sent by the last bolt to a specific port which I had defined. And I GET data back, but the cluster shows NO ACTIVITY. I know this is very vague, but, does anyone have any idea as to whats happening?

jake256 jake256 · Accepted Answer · 2014-08-01T05:41:37

It sounds like Storm is working fine, but your proxy/network settings are not. If it were a storm error, you should see exceptions in Nimbus UI and/or in the Storm supervisor logs.

Consider temporarily shutting down storm and use nc -l 4321 on the supervisor machines to assert your proxy is working as expected.

However...

You may have a fundamental flaw in your model. Storm's spouts are pull-based, so it seems odd to have incoming requests pushed to them. This is possible, of course, if you have your spouts start listening when they spin up and simply queue the requests. However, this presents another challenge for your model: you will likely have multiple spouts running on a single machine and they cannot share the same port (4321).

If you want to meld these two world of push & pull; then consider using a Kafka Spout.

Storm cluster not working in Production mode

1 Answers