Akka cluster: node identity crisis

Question

I have a cluster of 15 nodes/boxes. I start the nodes roughly at the same time. One of the nodes is behaving oddly and continually logging "Ignoring received gossip intended for someone else". However, the node does seem to work for a while before being being dropped from the cluster. Basically this one node seems to think it is someone else, whilst also behaviouring as itself. The code and config is exactly the same on all 15 nodes so I don't understand why I'm getting this issue on only one node. Maybe this is a hardware issue?

Some logging:

11:27:45.412 INFO  [main] Remoting - Starting remoting
11:27:45.638 INFO  [main] Remoting - Remoting started; listening on addresses :[akka.tcp://[email protected]:7000]
11:27:45.660 INFO  [main] Cluster(akka://streaming) - Cluster Node [akka.tcp://[email protected]:7000] - Starting up...
11:27:45.714 INFO  [main] Cluster(akka://streaming) - Cluster Node [akka.tcp://[email protected]:7000] - Registered cluster JMX MBean [akka:type=Cluster]
11:27:45.715 INFO  [main] Cluster(akka://streaming) - Cluster Node [akka.tcp://[email protected]:7000] - Started up successfully
11:27:45.830 INFO  [streaming-akka.actor.default-dispatcher-3] a.a.LocalActorRef - Message [akka.cluster.InternalClusterAction$InitJoinAck] from Actor[akka.tcp://[email protected]:7000/system/cluster/core/daemon#1997515880] to Actor[akka://streaming/system/cluster/core/daemon/joinSeedNodeProcess-1#1132911] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
11:27:45.872 INFO  [streaming-akka.actor.default-dispatcher-5] Cluster(akka://streaming) - Cluster Node [akka.tcp://[email protected]:7000] - Welcome from [akka.tcp://[email protected]:7000]
11:27:45.911 INFO  [streaming-akka.actor.default-dispatcher-2] Cluster(akka://streaming) - Cluster Node [akka.tcp://[email protected]:7000] - Ignoring received gossip intended for someone else, from [akka.tcp://[email protected]:7000] to [UniqueAddress(akka.tcp://[email protected]:7000,-1482656725)]
11:27:45.943 INFO  [streaming-akka.actor.default-dispatcher-16] Cluster(akka://streaming) - Cluster Node [akka.tcp://[email protected]:7000] - Ignoring received gossip intended for someone else, from [akka.tcp://[email protected]:7000] to [UniqueAddress(akka.tcp://[email protected]:7000,-1482656725)]
11:27:46.122 INFO  [streaming-akka.actor.default-dispatcher-16] Cluster(akka://streaming) - Cluster Node [akka.tcp://[email protected]:7000] - Ignoring received gossip intended for someone else, from [akka.tcp://[email protected]:7000] to [UniqueAddress(akka.tcp://[email protected]:7000,-1482656725)]

Config:

akka {
  cluster {
    seed-nodes = [
      "akka.tcp://[email protected]:7000"
      "akka.tcp://[email protected]:7000"
    ]
  }
  remote.netty.tcp.hostname = 172.17.102.128
}

Benjamin Benjamin · Accepted Answer · 2014-07-02T16:25:06

Problem resolved. I was running two clusters (dev and prod) and somehow (the mystery remains) the dev cluster was interacting with this one box in the prod cluster. My advice to anyone else who sees this issue is to check the IP addresses of the "from" nodes.

Akka cluster: node identity crisis

1 Answers