OpenMP won't utilize all cores?

Question

I'm trying to use OpenMP to make some code parallel.

    omp_set_num_threads( 8 );
    #pragma omp parallel 
    for (int i = 0; i < verSize; ++i)
    {
        #pragma omp single nowait
        { 
            neighVec[i].index = i;
            mesh.getBoxIntersecTets(mesh.vertexList->at(i), &neighVec[i]);
        }
    }

verSize is about 90k, and getBoxIntersecTets is quite expensive. So I expect the code to fully utilize a quad core cpu. However the CPU usage is only about 25%. Any ideas?

I also tried using omp parallel for construct, but same story.

getBoxIntersecTets uses STL unordered_set, vector and deque, but I guess OpenMP should be agnostic about them, right?

Thanks.

Ben Voigt Ben Voigt · Accepted Answer · 2011-09-26T17:52:03

First up, #pragma omp single is disabling parallel execution, you definitely don't want that.

Try this instead:

#pragma omp parallel for private(tempVec)
for (int i = 0; i < verSize; ++i)
{
    auto tempVec = neighVec[i];
    tempVec.index = i;
    mesh.getBoxIntersecTets(mesh.vertexList->at(i), &tempVec);
    neighVec[i] = tempVec;
}

The problem with your original code is that different threads are using adjacent elements of an array. Adjacent elements are placed next to each other in memory, which means they probably share a cache line. Since only one core can own a cache line at once, only one core can get work done at once. Or worse, your program may spend more time transferring ownership of the cache line than doing actual work.

By introducing a temporary variable, each worker can operate on an independent cache line, and then you only need access to the shared cache line at the end to store results. You should do the same thing for the first parameter if it's being passed by non-const reference.

OpenMP won't utilize all cores?

1 Answers