Erlang: inter-application communication

Question

To continue on with my journey to Erlands I'm developing simple IM system using OTP.

There are two OTP applications: a server (one instance) and a client (multiple instances). A set-up is shown below:

╭── node1@host ──╮
│  Server        │
│   └gen_server  │
╰────────────────╯

╭── node2@host ──╮
│  Client        │
│   └gen_server  │
╰────────────────╯
╭── node3@host ──╮
│  Client        │
│   └gen_server  │
╰────────────────╯
      ...

Client functions

Using Erlang shell, we can issue next commands to the client application:

1. Connect to the server and receive a random name from it (I'm fond of names like turbo-octopus, miniature-octocat etc. :)
2. Get a list of other connected clients.
3. Send a message to the client with specified name.
4. Send a message to all clients (broadcast).

Also client should be able to print message in stdout upon receiving.

Implementation details

All messages go through server.

Both server and client applications contain gen_servers (chat_server.erl and chat_client.erl respectively) responsible for handling messages. Server's chat_server process registered as global and visible on all nodes:

%% chat_server.erl
start_link() ->
    gen_server:start_link({global, ?SERVER}, ?MODULE, [], []).

When a client connects, it sends pid of its gen_server process. Doing this, we can store clients pids in server's state to distinguish them and sending/broadcasting messages.

%% chat_client.erl
connect() ->
    Res = gen_server:call({global, ?REMOTE_SERVER}, {connect, client_pid() ...}),
...

%% pid of the client's gen_server
client_pid() -> whereis(?CLIENT_SERVER). 

Server connect handle:

%% chat_server.erl
handle_call({connect, Pid}, _From, State) ->
    %% doing stuff like generating unique name, 
    %% adding client to list, etc.
    {reply, {connected, Name}, UpdatedState}.

Messaging (pun intended)

Well, it's pretty straightforward. The server handles cast from a client, seeks recipient's pid by given name and cast message to it/broadcasts to everyone. And this is it.

A Question will be asked

While developing this system, I wondered if a chosen approach is appropriate. I mean,

1. Passing around client's gen_server pid seems more or less acceptable at least because it allows uniquely identify clients and use all gen_server firepower on both ends. Is this the way you do it?
2. I have read here and there that explicit interface (calling exported functions) is preferable to direct messaging (thing I do in my client with gen_server:calls). Is there any way to fix this (rpc for example), or it's okay?
3. Given the same set-up (a node with a server application and N nodes with clients), will you use the same approach with gen_servers, or there is a better approach I'm unaware of?

Answer 1

Personally I think your architecture is slightly off.

If you want your client to accept incoming messages (e.g. when another client is sending a message to you or a doing a broadcast) then there currently doesn't seem to be a process where the server can send the messages to. gen_server is typically not the vehicle for that; it's mainly for server processes.

I think the idea should be to start a new process for every client. The process will become that specific client's main loop. If you (the user) wants to do something, you send a message to that specific process. That can be hidden behind function calls. Then the client's main loop will interact with the server.

The client's main loop - which is a separate process, is always ready to receive messages, so the server can send messages to your client if someone is sending to you.

BTW: I hope that your defines ?SERVER and ?REMOTE_SERVER are identical because, if I understand correctly, they both refer to the globally registered chat server. Better stick with one unique name.

Another issue is that you do not typically expose the gen_server:call() methods. The clients only call methods in the chat_server module without knowing what the name of the server is or whereever it lives (that's the beauty of Erlang!).

In chat_server.erl you put code like this; basically the client API. You will notice that in chat_client.erl there will only be calls to methods in the chat_server module. Very clean and transparent!

%% let a new client connect, all we need is it's Pid
new_client(Pid) ->
   gen_server:call({global, ?SERVER}, {connect, Pid}).

send_msg(From, To) ->
   gen_server:call({global, ?SERVER}, {sendmsg, From, To}).

logout_client(Pid) ->
    gen_srver:call({global, ?SERVER), {exit_client, Pid}).

The client code below (deliberately) does not automatically register the client's Pid, unless you restrict your system to allow exactly only one client per node. You cannot register more than one Pid under the same name. The code below does not register the new Pid as a name but it can be made to do so trivially, if that is what you desire or need.

Typically the client's code looks like:

%% start a new client, we spawn a new process for this
%% particular client and return their Pid, to be used
%% when you want your client to do something
connect(Server) ->
   spawn( ?MODULE, start_client, [] ).

%% client startup code
start_client() ->
   %% Initialize client state, if you wish
   State = 42,
   %% Now connect to chat server
   chat_server:new_client( self() ),
   %% And fall into our own main loop
   client_loop( State ).

%% This is the client's main loop
client_loop( State ) ->
   %% Wait for stuff to happen ...
   receive
       %% chat server sends message to us
       {message , Msg, From} ->
             io:format("~p sais ~p~n", [From, Msg]),
             client_loop( State );
       %% message sending is delegated to the server - see your own protocol
       {send, Msg, To} ->
             chat_server:send_msg(Msg, To),
             client_loop( State );
       %% terminate?
       done ->
             %% de-register with server
             chat_server:logout_client(self())
    end.

Now all that's needed is some utility functions to interact with your client process like below. Note that if you go to "each Erlang node is a single client" by registering the client's Pid locally, you can get rid of passing the Pid explicitly. But the mechanics remain the same.

send_message(Pid, Msg, To) ->
    Pid ! {send, Msg, To}.

logout(Pid) ->
    Pid ! done.

%% If you force your client's Pid to be registered to e.g. 'registered_name'
%% it would look like
send_message(Msg, To) ->
    registered_name ! {send, Msg, To}.

Answer 2

I agree with haavee that your architecture is not what I'd expect, but that's because I'm imagining something more low-level-TCP.

Regarding your questions:

Passing around client's gen_server pid seems more or less acceptable at least because it allows uniquely identify clients and use all gen_server firepower on both ends. Is this the way you do it?

Sure, I see nothing wrong with that part of the code. Your server keeps a mapping between PID and client name and it's like calling register/2 but only the server gets the mapping and you control how it works.

---

I have read here and there that explicit interface (calling exported functions) is preferable to direct messaging (thing I do in my client with gen_server:calls). Is there any way to fix this, or it's okay?

If you compiled your client and server applications together (one code-base, two entry-points) then you could do this. Instead of, on the client-side doing

connect() ->
    Res = gen_server:call({global, ?REMOTE_SERVER}, {connect, client_pid() ...}),

you'd have

-module(client).
connect() ->
    server:client_connect(client_pid()).

and

-module(server).
client_connect(ClientPID) ->
    Res = gen_server:call({global, ?REMOTE_SERVER}, {connect, ClientPID ...}).

But if you want to use net_kernel to connect nodes and you want to compile the source code independently, then your way is how you do it.

---

Given the same set-up (a node with a server application and N nodes with clients), will you use the same approach with gen_servers, or there is a better approach I'm unaware of?

What you're doing with net_kernel is building a distributed system. If you expect a few clients, that's fine. If you expect a ton of clients, then you have to remember that distributed Erlang defaults to a totally-connected mesh. So all your clients are actually connected to each other, as well as the server.

When I look at your description, I imagine a chat-server, and for this I would use gen_tcp for networking instead of net_kernel.

Advantages of net_kernel:

• It's very high-level. You don't need to think much about connection drops, and messages are very pure-Erlang.
• It's easier to debug. You can use the rpc module from a shell to run anything on any connected node, which is cool.

Advantages of gen_tcp:

• Server and clients are less connected. You could swap out a client or the server for a different version with the same network API (including swapping for something non-Erlang) and nobody else would know or care.
• Clients aren't interconnected (you can also do this with hidden nodes)
• You can use popular port numbers to get past dumb firewalls

I'd put your "client" and "server" modules both on the server machine. You listen for TCP connections and spawn off a "client" for each connection. The "client" module's job is to translate between the remote client talking over the network and the "server" module, talking over Erlang messages.