How to guarantee process uniqueness in an Erlang cluster?

Question

I'm trying to figure out what's the best way to approach the following load-distribution/process-uniqueness-guarantee problem for an Elixir application.

The application

My Elixir application is started on n different nodes (randomly chosen from a large pool, no fixed IP or host name known upfront) forming a cluster (I am not sure what's the best way of doing node discovery yet but let's ignore that for now).

In short, the application's main purpose is to keep two systems in-sync over time, basically an integration. There is an integration per user and a new integration can be added or an existing one can be removed at any time.

The problem

I'd like to have one Erlang process per integration as it is very elegant conceptually and brings many benefits (such as having a natural synchronization point for each integration). It seems it's the way to go to scale the system as well.

The issue is that obviously this process needs to be unique across the whole cluster (difficult to predict what could happen to the data if two processes attempts to synchronize the same integration) and I'd like to redistribute the work automatically as nodes fail or a new integration comes in.

Also, when deploying a new version of the application, the new cluster is started before the old one is shut down (we do not rely on hot-code reloading). This phase of transition needs to be handled somehow.

Possible solution

One solution could be to rely on a global process. When starting, nodes register themselves, connect to other registered nodes then attempt to start their copy of a global Scheduler process whose only role is to start integration processes across nodes.

Although this provides fault-tolerance, it does not guarantee one process per integration as the cluster can be split into two by a network partition. It also does not handle the brief period where both the old and new cluster are online and the old cluster is still doing work.

Some kind of global locking mechanism (via a shared Redis instance?) could be used to deal with both network partitions and the application restarting, but that seems fairly hacky.

Any suggestions?

Thanks!

Answer 1

For the purpose of this explanation let's assume that:

• Each user is identified by an ID that is unique across the whole system - UID
• Each process is identified by an ID that is unique across one node - PID
• Each node is identified by an ID that is unique across nodes - NID
• The integrator process per user is identified by tuple {NID, PID}

The problem is to ensure that there is 1-to-1 mapping between UID and {NID, PID}.

There are basically two ways of solving this:

I. Introduce a shared state, something like a register, which keeps track on the mapping between UID and {NID, PID}. Let's call it The Register. If this is a mnesia database with shared schema, a Redis instance, a separate node or anything else is an implementation detail. In any case each new process would need to register at The Register before starting to integrate a specific UID. In case of network partitioning, nodes going down or other disasters you deal with it in the standard way, e.g. you design The Register with the appropriate CAP theorem in mind depending on your requirements.

II. Assign PID to NID for the given UID algorithmically. This would work similarly to a hashtable. As an example let's say that UID is an integer (if it's not then any data structure can be reduced to an integer with a hash function). You select the node like this:

NID = UID % NX

Where NX is the amount of nodes (% is of course the modulo operation). On each node you can register the process as UID. Once that's done you can address each integrator process uniquely based on the UID - you use the % operation to get NID and the UID itself to get PID on the node.

The second approach requires that the amount of nodes doesn't change, e.g. nodes are monitored and if one node goes down another one is brought up to replace it. It could also work with each node being a master-slave pair with some replication happening between them.

The difference between those two approaches is that in the first case you have a single point of failure - if The Register becomes unavailable no new integrators can be started. Whereas in the second case the assignment between UID and its integrator process is completely distributed and asynchronous - if one node goes down the other nodes work uninterrupted, which makes it easier to scale.

But if the amount of nodes changes the first approach still works as before, whereas in the second approach this causes the hash function to change as well. This requires that the processes are rebalanced (moved between nodes) so that they still can be properly addressed.