Akka (.net) cluster with remote nodes: Disassociated exception

Question

Using akka (.net) I am trying to implement simple cluster use case.

Cluster - for nodes up/down events.
Remote - for sending message to specific node.

There are two actors: Master Node which listening cluster events and Slave Node which connecting to the cluster.

Address address = new Address("akka.tcp", "ClusterSystem", "master", 8080);
cluster.Join(address);

When ClusterEvent.MemberUp message is reseived Master Node creating actor link:

ClusterEvent.MemberUp up = message as ClusterEvent.MemberUp;
ActorSelection nodeActor = system.ActorSelection(up.Member.Address + "/user/slave_0");

Sending message to this actor causes an error:

Association with remote system akka.tcp://ClusterSystem@slave:8090 has failed; address is now gated for 5000 ms. Reason is: [Disassociated]

master config:

    akka {
        actor {
            provider = ""Akka.Cluster.ClusterActorRefProvider, Akka.Cluster""
        }

        remote {
            helios.tcp {
                port = 8080
                hostname = master
                bind-hostname = master
                bind-port = 8080
                send-buffer-size = 512000b
                receive-buffer-size = 512000b
                maximum-frame-size = 1024000b
                tcp-keepalive = on
            }
        }
        cluster{
            failure-detector {
                heartbeat - interval = 10 s
            }
            auto-down-unreachable-after = 10s
            gossip-interval = 5s
        }
        stdout-loglevel = DEBUG
        loglevel = DEBUG

        debug {{  
            receive = on 
            autoreceive = on
            lifecycle = on
            event-stream = on
            unhandled = on
        }}
    }

slave config:

akka {
        actor {
            provider = ""Akka.Cluster.ClusterActorRefProvider, Akka.Cluster""
        }

    remote {
        helios.tcp {
            port = 8090
            hostname = slave
            bind-hostname = slave
            bind-port = 8090
            send-buffer-size = 512000b
            receive-buffer-size = 512000b
            maximum-frame-size = 1024000b
            tcp-keepalive = on
        }
    }
    cluster{
        failure-detector {
            heartbeat - interval = 10 s
        }
        auto-down-unreachable-after = 10s
        gossip-interval = 5s
    }
    stdout-loglevel = DEBUG
    loglevel = DEBUG

    debug {{  
        receive = on 
        autoreceive = on
        lifecycle = on
        event-stream = on
        unhandled = on
    }}

}

So are you using Akka.Cluster and Akka.Remote interchangeably? I.e. using Remote to connect to nodes that aren't part of the cluster? — Aaronontheweb
No, all nodes are part of the cluster. I need a RPC way for each node in cluster. Is there another way to send message to the specific node of the cluster instead of actorRef.Tell()? — qrux

Aaronontheweb Aaronontheweb · Accepted Answer · 2015-09-03T21:40:01

Here's your problem:

cluster{
            failure-detector {
                heartbeat - interval = 10 s
            }
            auto-down-unreachable-after = 10s
            gossip-interval = 5s
        }

heartbeat-interval and auto-down-unreachable-after are the same duration - therefore your nodes will almost always disassociate automatically after 10s, because you're betting on a race condition that the failure detector might lose.

auto-down-unreachable-after is a dangerous setting - do not use it. You'll end up with a split brain or worse.

And make sure your failure detector interval is always lower than your auto-down interval.

Akka (.net) cluster with remote nodes: Disassociated exception

1 Answers