Assigning different thread numbers in OpenMP do-loops

Question

I have two do-loops inside OpenMP parallel region as follows:

!$OMP PARALLEL
...
!$OMP DO
...
!$OMP END DO
...
!$OMP DO
...
!$OMP END DO
...
!$OMP END PARALLEL

Let's say OMP_NUM_THREADS=6. I wanted to run first do-loop with 4 threads, and the second do-loop with 3 threads. Can you show how to do it? I want them to be inside one parallel region though. Also is it possible to specify which thread numbers should do either of the do-loops, for example in case of first do-loop I could ask it to use thread numbers 1,2,4, and 5. Thanks.

Answer 1

Well, you can add the num_threads clause to an OpenMP parallel directive but that applies to any directive inside the region. In your case you could split your program into two regions, like

!$OMP PARALLEL DO num_threads(4)
...
!$OMP END PARALLEL DO
...
!$OMP PARALLEL DO num_threads(3)
...
!$OMP END PARALLEL DO

This, of course, is precisely what you say you don't want to do, have only one parallel region. But there is no mechanism for throttling the number of threads in use inside a parallel region. Personally I can't see why anyone would want to do that.

As for assigning parts of the computation to particular threads, again, no, OpenMP does not provide a mechanism for doing that and why would you want to ?

I suppose that I am dreadfully conventional, but when I see signs of parallel programs where the programmer has tried to take precise control over individual threads, I usually see a program with one or more of the following characteristics:

• OpenMP directives are used to ensure that the code runs in serial with the result that run time exceeds that of the original serial code;
• the program is incorrect because the programmer has failed to deal correctly with the subtleties of data races;
• it has been carefully arranged to run only on a specific number of threads.

None of these is desirable in a parallel program and if you want the level of control over numbers of threads and the allocation of work to individual threads you will have to use a lower-level approach than OpenMP provides. Such approaches abound so giving up OpenMP should not limit you.

Answer 2

What you want cannot be achieved with the existing OpenMP constructs but only manually. Imagine that the original parallel loop was:

!$OMP DO
DO i = 1, 100
...
END DO
!$OMP END DO

The modified version with custom selection of the participating threads would be:

USE OMP_LIB

INTEGER, DIMENSION(:), ALLOCATABLE :: threads
INTEGER :: tid, i, imin, imax, tidx

! IDs of threads that should execute the loop
! Make sure no repeated items inside
threads = (/ 0, 1, 3, 4 /)

IF (MAXVAL(threads, 1) >= omp_get_max_threads()) THEN
   STOP 'Error: insufficient number of OpenMP threads'
END IF

!$OMP PARALLEL PRIVATE(tid,i,imin,imax,tidx)
! Get current thread's ID
tid = omp_get_thread_num()
...
! Check if current thread should execute part of the loop
IF (ANY(threads == tid)) THEN
   ! Find out what thread's index is
   tidx = MAXLOC(threads, 1, threads == tid)
   ! Compute iteration range based on the thread index
   imin = 1 + ((100-1 + 1)*(tidx-1))/SIZE(threads)
   imax = 1 + ((100-1 + 1)*tidx)/SIZE(threads) - 1
   PRINT *, 'Thread', tid, imin, imax
   DO i = imin, imax
      ...
   END DO
ELSE
   PRINT *, 'Thread', tid, 'not taking part'
END IF
! This simulates the barrier at the end of the worksharing construct
! Remove in order to implement the "nowait" clause
!$OMP BARRIER
...
!$OMP END PARALLEL

Here are three example executions:

$ OMP_NUM_THREADS=2 ./custom_loop.x | sort
STOP Error: insufficient number of OpenMP threads
$ OMP_NUM_THREADS=5 ./custom_loop.x | sort
 Thread           0           1          33
 Thread           1          34          66
 Thread           2 not taking part
 Thread           3 not taking part
 Thread           4          67         100
$ OMP_NUM_THREADS=7 ./custom_loop.x | sort
 Thread           0           1          33
 Thread           1          34          66
 Thread           2 not taking part
 Thread           3 not taking part
 Thread           4          67         100
 Thread           5 not taking part
 Thread           6 not taking part

Note that this is an awful hack and goes against the basic premises of the OpenMP model. I would strongly advise against doing it and relying on certain threads to execute certain portions of the code as it creates highly non-portable programs and hinders runtime optimisations.

---

If you decide to abandon the idea of explicitly assigning the threads that should execute the loop and only want to dynamically change the number of threads, then the chunk size parameter in the SCHEDULE clause is your friend:

!$OMP PARALLEL
...
! 2 threads = 10 iterations / 5 iterations/chunk
!$OMP DO SCHEDULE(static,5)
DO i = 1, 10
   PRINT *, i, omp_get_thread_num()
END DO
!$OMP END DO
...
! 10 threads = 10 iterations / 1 iteration/chunk
!$OMP DO SCHEDULE(static,1)
DO i = 1, 10
   PRINT *, i, omp_get_thread_num()
END DO
!$OMP END DO
...
!$OMP END PARALLEL

And the output with 10 threads:

$ OMP_NUM_THREADS=10 ./loop_chunks.x | sort_manually :)
 First loop
 Iteration     Thread ID
       1           0
       2           0
       3           0
       4           0
       5           0
       6           1
       7           1
       8           1
       9           1
      10           1
 Second loop
 Iteration     Thread ID
       1           0
       2           1
       3           2
       4           3
       5           4
       6           5
       7           6
       8           7
       9           8
      10           9