Why with Python parallelism CPU usage is less?

Question

Today is my first day I apply parallelism in python

I feel that I had been wasting my time with standard processing before, as clearly ML is much faster with parallelism.

This is first example I build

    xtrain, xval, ytrain, yval = train_test_split(X, y, test_size=0.2, random_state=1013)
    mdl = LogisticRegression()
    clf = OneVsRestClassifier(mdl)
    y_pred = cross_val_predict(clf, X, y, cv=3)

I only added n_jobs=-1 to make it run parallel

so it became like this

    y_pred = cross_val_predict(clf, X, y, cv=3, n_jobs=-1)

and I did a comparison between with parallelism vs without parallelism

Without parallelism

The process took 0:02:14 (2 minutes 14 sec)

When I looked at the CPU usage it looked like this

With parallelism

The process took 0:00:43 (43 sec)

When I looked at the CPU usage it looked like this

Which shows clearly that without parallelism has more CPU usage in all CPUs

My question is how come without parallelism consumes more CPU in all CPUs more than with parallelism??

If the job is spread on all CPUs then more CPUs should be busy

while without parallelism only one CPU should be busy and the rest are not as busy!!

What is the reason for this behaviour ?

Another question

I saw many example on people using joblib, and dask.. and it was too complex

does using dask and joblib give faster results than just adding n_jobs?

Thanks

The analysis required to figure out where bottlenecks are is context-dependent -- it's liable to be different based on the specific dataset, hardware, etc. The right tool to use to answer questions about it is typically a sampling profiler -- which can tell you exactly what each thread of your process was spending each percent of its time on, even when that's something like "parked, waiting for I/O" (and thus not using CPU resources at all). There's no generic answer as to why you see a given pattern of CPU usage reported; answering these questions requires measuring your specific instance. — Charles Duffy

William Mou William Mou · Accepted Answer · 2021-04-18T22:23:01

From my experience: By setting export OPENBLAS_NUM_THREADS=1, Maybe could reduce cpus usage in the first case (without parallelism).

The problem may be that libraries such as OpenBLAS take up a lot of your CPU usage, but they do not cause a significant impact on the overall performance.

Why with Python parallelism CPU usage is less?

1 Answers