I was using reducers library in some of the places in my code on a production server with 32 cores to leverage some parallelism. But the Fork/Join frameworks seems to utilize cores so heavily that other processes choke out and become unresponsive.
Is there some way to limit the no. of cores being used or thread being spawned by reducers library on a jvm instance?
It seems it isn't possible to adjust the standard reducers forkjoin threadpool size through function or configuration parameters. You need to change core.reducers itself.
From core.reducers source
(def pool (delay (java.util.concurrent.ForkJoinPool.)))
This corresponds with the default java constructor without arguments
ForkJoinPool() Creates a ForkJoinPool with parallelism equal to Runtime.availableProcessors(), using the default thread factory, no UncaughtExceptionHandler, and non-async LIFO processing mode.
instead of
ForkJoinPool(int parallelism) Creates a ForkJoinPool with the indicated parallelism level, the default thread factory, no UncaughtExceptionHandler, and non-async LIFO processing mode.
It would be a nice addition to have at least the option to control the number of cores (there's also an even more configurable version of ForkJoinPool), but for now the only option would be to fork core.reducers and change that line to the number of max cores you want used:
(def pool (delay (java.util.concurrent.ForkJoinPool. 28)))
