cuda infinite kernel

3
votes

I am working on an application for which it is necessary to run a CUDA kernel indefinitely. I have one CPU thread that writes stg on a list and gpu reads that list and resets (at least for start). When I write inside the kernel 

while(true)
{
//kernel code
}

the system hangs up. I know that the GPU is still processing but nothing happens of course. And I am not sure that the reset at the list happens.

I have to mention that the GPU used for calculations is not used for display, so no watchdog problem.

The OS is Ubuntu 11.10 and cuda toolkit 4.1. I could use any help/examples/links on writing infinite kernel successfully.

2
debuggingcudainfinite
CUDA scheduler is really bad at handling infinite loops, spin-locks, etc, since such "objects" are totally alien for GPU architecture. Much more common and predictable way is to just run your kernel once in a while to check whether new elements have appeared. - aland
Also, new elements can't just appear. You have to put them there. So you know when it's necessary to rerun the kernel. - Roger Dahl
Power usage on a high end GPU can jump up by 250W when a kernel is running, so there's money to save by being selective about when to run the kernel. More environmentally friendly too. - Roger Dahl
"the infinite kernel is mandatory for the current project. the goal is a gpu controller so, the gpu has to work autonomously without cpu interference (except of course for the kernel call)." Your entire idea sounds completely flawed IMO. You should go back and carefully rethink it. Take to heart what I said earlier: New elements can't just appear. You have to put them there. So you know when it's necessary to rerun the kernel. - Roger Dahl
For what seems to be your problem you want to run a complete process in the background or at least a thread, not just a CUDA kernel. - leftaroundabout

2 Answers

2
votes

The CUDA programming language and the CUDA architecture do not currently support infinite kernels. I suggest you consider Roger's suggestion.

If you want to pursue this I suggest you add the following debug code to your kernel:

  1. Increment a variable in pinned memory every N clocks (may want a different location for each SM) and,
  2. Periodically read a memory location that can be updated by CPU to tell the kernel to exit.

This is a software watchdog.

You can use clock() or clock64() to control how often you do (1) and (2).

You can use cuda-gdb to debug your problem.

Infinite loops are not supported in the language. The compiler may be stripping code. You may want to review the PTX and SASS. If the compiler is generating bad code you can fake it out by making the compiler think there is a valid exit condition.

0
votes

As already pointed out by @Greg Smith, CUDA compiler does not generate proper assembly for infinite loops. And of course there are certain situations when it's a perfect solution, e.g. running a background service kernel, which receives updates from host, pushed over host-mapped memory.

One workaround, which works as of CUDA 9.2:

volatile int infinity = 1;
while (infinity)
{
  ...
}

Doing infinite loop inside a divergent branch is obviously not a good idea. Other than that, improper handling of while (1) construct IMO is definitely a bug.