openMp and the number of cores vs cpus

Question

I'm wondering about how openmp figures out how many threads it can run via the omp_get_max_threads() library call. I'm running on a centOS linux machine using gcc -fopenmp. My machine has 16 AMD Opteron(tm) Processor 6136 CPUs, each with 8 cores, all according to /proc/cpuinfo. If I run omp_get_num_procs() it returns 16. But omp_get_max_threads() also returns 16. Why isn't the max threads number 16*8?

When I run a program that uses 16 threads I see the program in top running at ~1600% of CPU and if I toggle 'Last used cpu (SMP)' that number moves around a bit. So the 1600% makes sense but is there any way to know which cores of which CPUs the threads are running on?

I'm pretty new to openmp so sorry if these questions seem naive.

Answer 1

You can use the hwloc tool set to know the binding of the threads of any application to the hardware threads/cores. You need only the name or the PID of the target running process. Here is an example:

$ hwloc-ps --pid 2038168 --threads --get-last-cpu-location
2038168 Machine:0       ./a.out
 2038168    Core:5      a.out
 2038169    Core:3      a.out
 2038170    Core:1      a.out
 2038171    Core:4      a.out
 2038172    Core:0      a.out
 2038173    Core:2      a.out

Here we can see that the process a.out (with the PID 2038168) uses 6 threads each map on different cores. However, the mapping of threads on cores over time can change if you do not configure OpenMP properly (a starting point is to set the environment variables OMP_PROC_BIND and OMP_PLACES).

Additionally, you can use  hwloc-ps to understand the topology of your machine (how many cores there are, how many threads, how they are connected, etc.). I am very surprise you can have 16 "AMD Opteron(tm) Processor 6136 CPUs". Indeed, this processor use the G34 socket which is available in up to 4-socket arrangements (and 8 dies). So, please check this with hwloc-ps!

An alternative way is to use a profiling tool (such as Intel VTune).