How to spawn processes in Erlang more efficiently?

Question

I have these two functions

check_requests(Timestamp,Plan,AddInfo)->
    {Day,_Month,_Year,Hour,Minute,Second,Micro} = Timestamp,
    ExpiredRequests = lists:filter(fun(Elem) -> 
                               {_ID,Time,_Key,_Action} = Elem,
                               Time < {Day,Hour,Minute,Second,Micro}
                               end, Plan),
    NewPlan = Plan -- ExpiredRequests,
    make_requests(ExpiredRequests,AddInfo),
    NewPlan.

make_requests([],_)->ok;
make_requests([ExpiredRequest|Rest],AddInf)->
    {_ID,_Time,Key,Action} = ExpiredRequest,
    spawn(?MODULE,request,[AddInf,Action,Key]),
    make_requests(Rest,AddInf).

The main idea is that I have plan where each request is given a time when it expires and should be executed. Upon expiry, I want to execute the request concurrently with other requests that may expire in time so I spawn the request function with new process. Now, If I have millions of requests, I will have millions of processes and I assume that is not desired. The request execution may last maximally one minute. I would like to know what happens to the process when the function I spawned is finished. Is it killed? or does it still exists and should be garbage collected? I really need requests to be concurrent and have no idea how else could I implement it. Also is there some OTP behaviour that could be helpful in this case? I have not studied OTP yet, I'm somewhere in the middle of the whole Erlang framework.

Answer 1

The spawned process exits when the function finishes running. Nothing needs to be garbage collected; the entire process heap is deallocated. (Except for binaries over 64 bytes, which are kept in a shared heap.)

A note about your code: if you need to separate a long list into the elements that satisfy a certain predicate and those that don't, it's more efficient to use lists:partition/2 than using lists:filter and then --. The former runs in linear time to the length of the list, while -- can run in quadratic time if you're unlucky.

Answer 2

Your task is completely a "max heap/priority queue" template. It should be solved not as filtering a list, but as popping out elements with highest priorities(the closer expiration date to given moment, the higher priority) In priority queue extracting an element would take O(1) time. You would need to traverse all the queue ONLY if all processes expired simultaniously.

Also, I would split this queue into large amount of priority queues(independent if it's possible), and handle them concurrently.

And in Erlang, you really can monitor other processes(to watch their "exit" codes). Also, this may be done using OTP supervisors.

Answer 3

Considering the worst case scenario here (from what I understood from your question/requirement), of a million requests expiring at the same time, you have got not much option other than to process them in batch by:
1. Increase the process limit from default 32k to some more reasonable value. (Depends upon you resource availability and testing results on it) and process the expired results (32k - some offset) (default case) at a time.
2. If you have an option to distribute the process across multiple nodes (on the same machine/ different machine), you can further work on above approach. But again multiple nodes on same machine will depend upon your resource availability.

In erlang, once the function being executed by the spawned process exits, the process itself exits.