Vertx Eventbus failed to send inter-nodes

Question

I had a small demo which demonstrate how Vertx send messages between nodes through Eventbus with Hazelcast Cluster Manager.

So far, I had made nodes recognize each other successfully.

Members [2] {
    Member [192.168.150.193]:5701 - 0c8120d8-73dc-4fc2-af6d-16288dbd707a
    Member [192.168.130.41]:5701 - 19239bb1-e7e8-4990-a389-817d6a7d128a this
}

But then, I encountered a problem with EventBus communication between Cluster members. According to Vertx Manual Page.

Cluster managers do not handle the event bus inter-node transport, this is done directly by Vert.x with TCP connections.

However, even when I'd get a clustered event bus from the Cluster Manager beforehand by:

public static EventBus clusterEB = null;
...
Vertx.clusteredVertx(options, res -> {
        if (res.succeeded()) {
            Vertx vertx = res.result();
            clusterEB = vertx.eventBus();
            vertx.deployVerticle(new DemoVerticle());
        }
    });

When the main verticle of the local node use that static eventbus object to send message to the remote node.

eb.send("worker.count", request_count, res -> {
                if (res.succeeded()) {
                    System.out.println("Received reply No." + res.result().body());
                    worker_busy--;
                    System.out.println("WORKER_BUSY = " + worker_busy);
                }
            });

An exception was thrown out:

io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:50509

With netstat, I found out that 50509 is the port which the remote node eventbus is listening. Hence, I guessed my configuration missed something that led to Vertx mistook the address, please correct me if I'm wrong.

Those pieces of information are the ones I researched so far and I really look forward to hearing any suggestions from you guys on this problem. Feel free to ask me if you need to have more info.

p/s: Event Bus communication between local nodes is still fine. Code running on Ubuntu (with UFW disabled), remote machine is a Windows with firewall turned off (I think firewall is not a problem here because there is still tcp connections between nodes to list all of cluster members in the network)

Answer 1

It looks that vert.x is picking the wrong IP address for your hosts (127.0.0.1) while they are in fact:

• 192.168.150.193
• 192.168.130.41

You might want to force each node to select the right IP address by adding the following extra command line argument:

-cluster-host 192.168.150.193

And the same for the other node with:

-cluster-host 192.168.130.41

Answer 2

For people who encountered the same problem, hope that my answer will help you.

Thanks to the suggestion from @Paulo_Lopes, my problem was solved with a little tuning.

I have to set Cluster host programmatically with something like:

VertxOptions options = new VertxOptions().setClusterManager(mgr).setClusterHost("192.168.130.21");

Vertx Hazelcast's document also had a brief info about this:

When running Vert.x is in clustered mode, you should also make sure that Vert.x knows about the correct interface. When running at the command line this is done by specifying the cluster-host option...

But to be honest, I think they should put it easier to understand way. In short, you have to specify your machine IP to be cluster host because clustered event bus will take that IP and registered a socket for inter-node communication. If no cluster host config was set, Vertx will take your 'localhost' (127.0.0.1) as the IP and every message send through Vertx will only be able to reach local nodes.

That's all. Hope it will help people in need and many thanks to you, @Paulo.