Linux kernel unresponsive when multiple high priority thread run on multiple cores

Question

Wrote a sample C++ multithreaded program that runs with 10 threads, each thread set to high priority and affinity. Compiled and ran this code on dell machine that has 16 cores, running centos 7 (Linux kernel - 3.10.0-229), disabled hyperthreading. After I ran this code, in few seconds, our Linux machine becomes unresponsive, in the sense that, if I open up Eclipse editor, and save a file or save a file in vi editor those applications hang. Interestingly, once I stop this program / process, then all other applications resume from where they left off. Also I don't see this issue if I remove priority from these 10 threads.

Questions:

1) Out of 16 cores, 6 cores still left on the machine (above cpu usage shows, cpu executed 62.9% user space, and was idle for 37.1%. Interestingly 0% cpu usage in kernel space), so ideally kernel should have used those 6 cores to handle other application, what could be the reason that other application does not able to run? How to resolve this issue without introducing sleep / changing priority?

2) Like to know better approach other than introducing a sleep / waiting on a event (which introduce minimal latency due to kernel context switch) in the thread to achieve parallelism?

Ran top command (top -H):

%Cpu(s):  62.9 us,  0.0 sy,  0.0 ni, 37.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st

1107 arun      rt   0   96748   1112    932 R 99.9  0.0   0:25.78 PthreadTest
1115 arun      rt   0   96748   1112    932 R 99.9  0.0   0:24.79 PthreadTest
1118 arun      rt   0   96748   1112    932 R 99.9  0.0   0:22.79 PthreadTest
1120 arun      rt   0   96748   1112    932 R 99.9  0.0   0:20.79 PthreadTest
1123 arun      rt   0   96748   1112    932 R 99.9  0.0   0:18.79 PthreadTest
1117 arun      rt   0   96748   1112    932 R 94.1  0.0   0:23.78 PthreadTest
1119 arun      rt   0   96748   1112    932 R 94.1  0.0   0:21.78 PthreadTest
1122 arun      rt   0   96748   1112    932 R 94.1  0.0   0:19.78 PthreadTest
1124 arun      rt   0   96748   1112    932 R 94.1  0.0   0:17.78 PthreadTest
1125 arun      rt   0   96748   1112    932 R 94.1  0.0   0:16.76 PthreadTest

Code below:

#include <unistd.h>
#include <iostream>
#include <cstdlib>
#include <pthread.h>

using namespace std;

#define NUM_THREADS 10

void *PrintHello(void *threadid)
{
   long tid;
   tid = (long)threadid;

   cout << "Hello World! Thread ID, " << tid << endl;
   while(true)
   {
        continue;
   }
   pthread_exit(NULL);
}

int main ()
{
   pthread_t threads[NUM_THREADS];
   pthread_attr_t  threads_attr[NUM_THREADS];
   struct sched_param     params;
   params.sched_priority = sched_get_priority_max(SCHED_FIFO);
   int rc;
   int i;
   int cpu_num = 0;

   for( i=0; i < NUM_THREADS; i++ ){

      cpu_set_t cpu;
      CPU_ZERO(&cpu);
      CPU_SET(cpu_num, &cpu);
      cout << "main() : creating thread, " << i << "cpu_num : "<<cpu_num<<endl;
      pthread_attr_init(&threads_attr[i]);
      pthread_attr_setscope(&threads_attr[i], PTHREAD_SCOPE_SYSTEM);
      rc = pthread_create(&threads[i], NULL,
                          PrintHello, (void *)i);
      if (rc){
         cout << "Error:unable to create thread," << rc << endl;
         exit(-1);
      }

      sleep(1);


      int ret = pthread_setaffinity_np(threads[i], sizeof(cpu_set_t), &cpu);
      if(ret == 0 && CPU_ISSET(cpu_num, &cpu))
      {
        cout << "Thread " << i << " affinity set " <<endl;
      }


      ret = pthread_setschedparam(threads[i], SCHED_FIFO, &params);
      if(ret == 0)
      {
        cout << "Thread " << i << " priority set " <<endl;
      }
      cpu_num++;
   }


// free attribute and wait for the other threads
   void *status;
   for( i=0; i < NUM_THREADS; i++ )
   {
        rc = pthread_join(threads[i], &status);
        if (rc){
            cout << "Error:unable to join," << rc << endl;
            exit(-1);
        }
        cout << "Main: completed thread id :" << i ;
        cout << "  exiting with status :" << status << endl;
   }

   pthread_exit(NULL);
}

Compile:

g++ -std=c++14 -g -o PthreadTest busywait.cpp -lpthread

The problem is interesting, but out of curiosity, why are you running this stuff at high priority? Having higher priority CPU-eating threads is exactly the recipe to hang lower priority programs... — Matteo Italia
Threads that always want the CPU should have lowered priority if anything. — David Schwartz
What CPUs does this machine have? And why are you specifying -lpthread with GCC/linux? (The correct flag is -pthread.) — David Schwartz
@MatteoItalia, i'm doing an optimization to our current production application (multithreaded) has similar kind of architecture. Was getting better results if multiple consumer thread spins on data queue rather than wait/wakeup on the data when notified. But if multiple consumer thread spins on a core we're seeing the issue as mentioned above. — arun

David Schwartz David Schwartz · Accepted Answer · 2015-11-05T00:46:25

The effects of suddenly depriving the kernel of any use of a live core for an unlimited amount of time are unspecified and unknown. Anything attached to that core before exclusive ownership of it was taken, which could include threads that are waiting to be scheduled on it, is forever lost to the system.

DON'T DO THIS!

Linux kernel unresponsive when multiple high priority thread run on multiple cores

2 Answers