Akka cluster singleton does not answer when using akka-discovery by config

Question

I am building an akka service and for local testing purposes using akka-discovery in config mode. I discovered that when switching from defined seed nodes for the cluster creation to the discovery-mechanism (akka-discovery) my cluster singleton stops to work. On an ask I simply get a Timeout from the respective singleton (I am using the proposed mechanism of cummunicating to that singleton via a proxy). I could neither find a solution to that problem nor understand it currently since I create the SingletonManager instance on each node after having started the cluster bootstrap. Even when I include a wait of around a minute after the ClusterBootstrap start() call before creating the singleton manager I get the same result, where I would think such cluster functionality is already available since the cluster is up.

Rough code samples:

Sbt:

"com.lightbend.akka.management" %% "akka-management-cluster-bootstrap" % "0.19.0", "com.lightbend.akka.management" %% "akka-management" % "0.19.0", "com.lightbend.akka.management" %% "akka-management-cluster-http" % "0.19.0", "com.lightbend.akka.discovery" %% "akka-discovery-config" % "0.19.0", "com.typesafe.akka" %% "akka-cluster" % "2.5.18", "com.typesafe.akka" %% "akka-cluster-metrics" % "2.5.18", "com.typesafe.akka" %% "akka-cluster-tools" % "2.5.18", "com.typesafe.akka" %% "akka-contrib" % "2.5.18", "io.kamon" % "sigar-loader" % "1.6.6-rev002", "org.scalactic" %% "scalactic" % "3.0.5", "com.typesafe.akka" %% "akka-stream-testkit" % "2.5.18" % Test, "org.scalatestplus.play" %% "scalatestplus-play" % "3.0.0" % "test"

Cluster Node Start (code executed on all nodes):

     val system = ActorSystem(applicationName, baseConfig)

  AkkaManagement(system).start()

   ClusterBootstrap(system).start()

val clusterSingletonProperties = ClusterSingletonManager.props(
      singletonProps = SingletonActor.props,
      terminationMessage = PoisonPill,
      settings = ClusterSingletonManagerSettings.apply(singletonConfig)
    )
    actorSystem.actorOf(clusterSingletonProperties, "singleton")

controller call (I just sent the ask, in the singleton actor I , on any received string return another string without any processing):

val workerService: ActorRef = actorSystem.actorOf(
    ClusterSingletonProxy.props(
      singletonManagerPath = "/user/singleton",
      settings = ClusterSingletonProxySettings.create(singletonProxyConfig)
    )
  )

(workerService ? "holla").mapTo[String].map { message =>
      val result = Json.toJson({"result" -> message})
      Ok(result)
    })

The above works when seed-nodes are defined, but not in discovery. The order by which I start the system, the bootstrap and the singleton manager to me suggests that there should be no cause of cluster service not yet being available, since cluster is up when creating the singleton manager.

Any suggestions out there?

Thanks, Best regards, Andi

Make sure that cluster formation has finished. Add registerOnMemberUp callback to print that cluster is formed. If this doesn't happen, you have a problem with cluster formation and it's not related to singleton. Increase logging level for Akka for further debugging — Ivan Stanislavciuc
Thanks for your comment, Ivan. I forgot to clarify above that I also tried wrapping the ClusterSingletonManager creation in such a Cluster(actorSystem).registerOnMemberUp(..) call. Even set a scheduled wait within the registerOnMemberUp. As opposed to setting seed nodes in the config, this does not yield expected results. — awagen
Ah, and the cluster forms correctly as stated by logs. Each of the three nodes joins and starts the ClusterSingletonManager. Just communicating from play controller to this singleton via the respective proxy yields no response from the singleton (it is created on the oldest node though). — awagen
I narrowed down the problem. It has to do with creating the proxy within the controller. I customized the ProdServerStart a bit to include above snipppets for cluster singleton manager creation and bootstrap process. When I create proxy there and send message, singleton actor is found. When I do it in the respective controller of the play app, singleton actor is NOT found. Since frontend works over different port, maybe this is split up from the cluster. Only question then remains why this works when seed nodes are defined .... — awagen

awagen awagen · Accepted Answer · 2018-12-20T22:53:12

For those coming along the same problem: Reason was copying the ProdServerStart.class by which Play is natively started and adding creation of actorSystem and the abovementioned steps (node discovery, start of ClusterSingletonManager) into it. Since Play starts an actor system already (make sure it is named the same as the other nodes in the cluster, which must be the name given in your discover config), it helped to instead stick to the original ProdServerStart.class class and use an eager singleton which on startup gets the actor system started by Play app wired into and executes above steps. It seems a bit subtle that starting an actor system with the same name within the server start causes above behavior.

Akka cluster singleton does not answer when using akka-discovery by config

1 Answers