How to structure MQ consumer loop in Erlang/OTP?

Question

I need to create simple application that consumes message queue and asynchronously handles messages using Erlang/OTP. Consider this pseudo-example in Golang:

var queue chan
func main() {
    for req := range queue {
        go handleRequest(req)  //handle asynchronously 
    }
}

How to correclty structure this following OTP principles?
I've been looking for gen_server, but in this case where do I define my loop resursive?
Also, how can I start asynchronous handle? Should I create another supervisor and use supervisor:start_child on every new message?

legoscia legoscia · Accepted Answer · 2019-04-26T12:58:52

The gen_server module in the standard library defines the recursive loop for you. The only thing you need to do is to implement callback functions to handle messages. If the message queue is sending Erlang messages to your gen_server process, you'd do something like:

handle_info({incoming_request, Request}, _From, State) ->
    async_handle_request(Request),
    {noreply, State}.

To handle the requests asynchronously, async_handle_request would start a process for each incoming request. There are two ways of doing that: either just spawn a process, or start each process under a simple_one_for_one supervisor. The differences come down to error handling and shutdown behaviour. What do you do if handling a request fails? Do you ignore the error, or let it propagate to the gen_server process, or do you have the supervisor restart the process and retry the request?

This question explains when you might use a simple_one_for_one supervisor. If you just want to spawn a process, that would look like this:

async_handle_request(Request) ->
    spawn_link(fun() -> handle_request(Request) end).

And then the actual request handling logic is implemented in handle_request.

How to structure MQ consumer loop in Erlang/OTP?

1 Answers