Erlang: blocking C NIF call behavior

Question

I have observed a blocking behavior of C NIFs when they were being called concurrently by many Erlang processes. Can it be made non-blocking? Is there a mutex at work here which I'm not able to comprehend?

P.S. A basic "Hello world" NIF can be tested by making it sleep for a hundred microseconds in case of a particular PID calling it. It can be observed that the other PIDs calling the NIF wait for that sleep to execute before their execution.

Non blocking behavior would be beneficial in cases where concurrency might not pose an issue(e.g. array push, counter increment).

I am sharing the links to 4 gists which comprise of a spawner, conc_nif_caller and niftest module respectively. I have tried to tinker with the value of Val and I have indeed observed a non-blocking behavior. This is confirmed by assigning a large integer parameter to the spawn_multiple_nif_callers function.

Links spawner.erl,conc_nif_caller.erl,niftest.erl and finally niftest.c.

The line below is printed by the Erlang REPL on my Mac.

Erlang/OTP 17 [erts-6.0] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Answer 1

NIF's themselves don't have any mutex. You could implement one in C, and there is one when you load NIF's object, but this should be done only once with loading module.

One thing that's might be happening (and I would bet that's what is going on), is you C code messes up Erlang scheduler(s).

A native function that do lengthy work before returning will degrade responsiveness of the VM, and may cause miscellaneous strange behaviors. Such strange behaviors include, but are not limited to, extreme memory usage, and bad load balancing between schedulers. Strange behaviors that might occur due to lengthy work may also vary between OTP releases.

and description of what lengty work means and how you could solve it.

In very few words (with quite few simplifications):

For core one scheduler is created. Each has a list of processes which he can run. If ones scheduler list is empty, he will try to still work from another one. This can fail, if there is nothing (or not enough) to still.

Erlang schedulers spends some amount of work in one process, than moves to another, spend there some amount of work, and move to another. And so on, and so one. This is very similar to scheduling in system processes.

One thing that very important here is calculating amount of work. As default each function call has assigned some number of reductions. Addition could have two, calling function in your module will have one, sending a message also a one, some build-in could have more (like list_to_binary). If we collect 2 000 reductions we move to another process.

So what is the cost of your C function? It's only one reduction.

Code like

loop() ->
   call_nif_function(),
   loop().

could be taking all whole hour, but scheduler will be stuck in this one process, because he still haven't count to 2 000 reductions. Or to put it in other words, he could be stuck inside NIF without possibility to move forward (at least any time soon).

There are few ways around this but general rule is stat NIF's should not take long time. So if you have long running C code, maybe you should use drivers instead. They should be much easier to implement and manage, that tinkering with NIF's.

Answer 2

I think the responses about long-running NIFs are off the mark, since your question says you're running some simple "hello world" code and are sleeping for just 100 us. It's true that ideally a NIF call shouldn't take more than a millisecond, but your NIFs likely won't cause scheduler issues unless they run consistently for tens of milliseconds at a time or more.

I have a simple NIF called rev/1 that takes a string argument, reverses it, and returns the reversed string. I stuck a usleep call in the middle of it, then spawned 100 concurrent Erlang processes to invoke it. The two thread stacktraces shown below, based on Erlang/OTP 17.3.2, show two Erlang scheduler threads both inside the rev/1 NIF simultaneously, one at a breakpoint I set on the NIF C function itself, the other blocked on the usleep inside the NIF:

Thread 18 (process 26016):
#0  rev (env=0x1050d0a50, argc=1, argv=0x102ecc340) at nt2.c:9
#1  0x000000010020f13d in process_main () at beam/beam_emu.c:3525
#2  0x00000001000d5b2f in sched_thread_func (vesdp=0x102829040) at beam/erl_process.c:7719
#3  a0x0000000100301e94 in thr_wrapper (vtwd=0x7fff5fbff068) at pthread/ethread.c:106
#4  0x00007fff8a106899 in _pthread_body ()
#5  0x00007fff8a10672a in _pthread_start ()
#6  0x00007fff8a10afc9 in thread_start ()

Thread 17 (process 26016):
#0  0x00007fff8a0fda3a in __semwait_signal ()
#1  0x00007fff8d205dc0 in nanosleep ()
#2  0x00007fff8d205cb2 in usleep ()
#3  0x000000010062ee65 in rev (env=0x104fcba50, argc=1, argv=0x102ec8280) at nt2.c:21
#4  0x000000010020f13d in process_main () at beam/beam_emu.c:3525
#5  0x00000001000d5b2f in sched_thread_func (vesdp=0x10281ed80) at beam/erl_process.c:7719
#6  0x0000000100301e94 in thr_wrapper (vtwd=0x7fff5fbff068) at pthread/ethread.c:106
#7  0x00007fff8a106899 in _pthread_body ()
#8  0x00007fff8a10672a in _pthread_start ()
#9  0x00007fff8a10afc9 in thread_start ()

If there were any mutexes within the Erlang emulator preventing concurrent NIF access, the stacktraces would not show both threads inside the C NIF.

It would be nice if you were to post your code so those willing to help resolve this issue could see what you're doing and perhaps help you find any bottlenecks. It would also be helpful if you were to tell us what version(s) of Erlang/OTP you're using.

Answer 3

NIF calls block the scheduler to which the process that called them is bound. So, for your example, if those other processes are on the same scheduler, they cannot call into the NIF until the first process finishes.

You cannot make an NIF call non-blocking in this regard. You can, however, spawn your own threads and offload the brunt of your work to them.

Such threads can send messages to local Erlang processes (processes on the same machine), and as such you can still get the response you desire by waiting for your spawned thread to send back a message.

A bad example:

static ERL_NIF_TERM my_function(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {
    MyStruct* args = new MyStruct(); // I like C++; so sue me
    args->caller = enif_self();
    ErlNifTid thread_id;
    // Please remember, you must at some point rejoin the thread, 
    // so keep track of the thread_id
    enif_thread_create("my_function_thread", &thread_id, my_worker_function, (void*)args, NULL);
    return enif_make_atom(env, "ok");
}
void* my_worker_function(void* args) {
    sleep(100);
    ErlNifEnv* msg_env = enif_alloc_env();
    ERL_NIF_TERM msg = enif_make_atom(msg_env, "ok");
    enif_send(NULL, args->caller, msg_env, msg);
    delete args;
    return NULL;
}

And in your erlang source:

test_nif() -> 
    my_nif:my_function(),
    receive
        ok -> ok
    end.

Something to that effect, anyway.