How do I tell supervisor to start 1000 instances of a specific gen_server?

Question

As part of a solving a computationally intensive task, I wish to have 1000 gen_servers doing small task and update the global database. How can I achieve this in erlang OTP? In most of the examples the supervisor supervises only a single gen_server. Can a supervisor supervise more than a thousand instances of the same gen_server?

e.g. Say I want to find maximum of a extremely long array and each gen_server instance should create work on a part of the array and update the global minimum.

Answer 1

Is it possible: yes. For example you can create a pool of 1000 processes with the following supervisor:

-module (big_supervisor).

-export([start_link/0]).

-behaviour(supervisor).
-export([init/1]).

start_link() ->
    supervisor:start_link({local, ?MODULE}, ?MODULE, {}).

%% @private
init({}) ->
    Children = create_child_specs(1000),
    RestartStrategy = {one_for_one, 5, 10},
    {ok, {RestartStrategy, Children}}.

create_child_specs(Number) ->
    [{{child_process,X},{child_process, start_link, []},permanent, 5000, worker,[child_process]} || X <- lists:seq(1,Number)].

Is it a good architecture, I don't know. Until now I have found 2 kinds of architectures:

• One with a limited and well identified (by role) chidren
• One with a kind of process factory, creating dynamically as many children as needed on demand, using the simple_one_for_one strategy and the start_child/2 terminate_child/2 functions.

Notes also that the supervisors are not mandatory if you want to spawn processes. In your explanation it seems that the processes could be created for a very limited time, in order to compute in parallel something. Two remarks in this case:

• it is not worth to spawn more processes that the number of threads that will effectively run in parallel on your VM.
• one exception is if the work to achieve in each process will have to wait for an external information, for example the return of an external database. In this case it may be interesting to spawn more processes, the optimal number depending on the external access limits.

Answer 2

Like Pascal said, it is possible to start a set number or children but the use case you described would probably work better with a simple_one_for_one strategy as all children are the same. This lets you add as many of the same type of children as needed at a smaller cost. gen_servers have overhead, and even though it's not too big, when you're talking about 1000 processes crunching numbers it makes a difference.

If your processes will be doing something very simple and you want it to be fast I would consider not using gen_servers, but instead just spawning processes. For real power, you would have to spawn processes on different nodes using spawn/4 to make use of more cores. If you are using machines in different locations you can also use a message buss as a load balancer to distribute the work between nodes. All depends on how much work you need done.

Also keep in mind that Erlang is not the best for crunching numbers. You could use C code to do the crunching and have each Erlang process you spawn/each child call a nif.