OpenMP thread affinity with GOMP_CPU_AFFINITY and sched_setaffinity

Question

I'm trying to bind OpenMP threads to CPUs. Two methods have been tried.

The first one is sched_setaffinity. Every time threads are forked, I invoke sched_setaffinity to bind OpenMP threads to CPUs. It seems that binding for every time is far too costly. Is it really needed to bind for every time or just bind for once?

The second one is GOMP_CPU_AFFINITY. I set GOMP_CPU_AFFINITY=0-7 and OMP_NUM_THREADS=4. Two nodes are used for computing. Each node has 2 chips and each chip has 4 cores. I place a process in each node and each process forks 4 threads. If GOMP_CPU_AFFINITY is not set, OpenMP does speed up the program. If GOMP_CPU_AFFINITY is set, the following are returned and OpenMP does not work:

OMP: Warning #123: Ignoring invalid OS proc ID 1.
OMP: Warning #123: Ignoring invalid OS proc ID 2.
OMP: Warning #123: Ignoring invalid OS proc ID 3.
OMP: Warning #123: Ignoring invalid OS proc ID 4.
OMP: Warning #123: Ignoring invalid OS proc ID 5.
OMP: Warning #123: Ignoring invalid OS proc ID 6.
OMP: Warning #123: Ignoring invalid OS proc ID 7.

The output looks like as if the OpenMP runtime sees your system as having a single CPU only. What compiler and operating system is that? — Hristo Iliev
@HristoIliev I actually got this today too, with Intel Fortran 14 and GOMP_CPU_AFFINITY="0-7:2". Strange is I am using this set-up regularly, now just ssh-ing from home there. — Vladimir F
@VladimirF, it could happen if your shell session is somehow limited inside a cgroups container with a very narrow CPU mask. I simply cannot come up with other plausible explanations. — Hristo Iliev
@Hristolliev OS is linux and kernel release is 2.6.18-164.el5.perfctr. The compiler is gcc 4.6.0. — matricer
@Hristolliev the cluster I use is a little bit old. I have no idea if this can be a problem. — matricer

hahaha hahaha · Accepted Answer · 2014-11-10T08:03:42

with openmp 4.0, one can set affinity per parallel region

#pragma omp parallel proc_bind(close)

with nested parallelism, one can also useproc_bind(spread) in the outer region to do coarse-grain pining first.

OpenMP thread affinity with GOMP_CPU_AFFINITY and sched_setaffinity

1 Answers