Concurrent Kernels with CUSPARSE Library

Question

I would like to ask you a question about the concurrent kernel execution in Nvidia GPUs. I explain us my situation. I have an code which launchs 1 sparse matrix multiplication for 2 different matrix (one for each one). These matrix multiplications are performed with the cuSPARSE Library. I want both operations can be concurrently performed, so I use 2 streams to launch them. With Nvidia Visual profiler, I´ve observed that both operations (cuSPARSE kernels) are completely overlaped. The time stamps for both kernels are:

• Kernel 1) Start Time: 206,205 ms - End Time: 284,177 ms.
• Kernel 2) Start Time: 263,519 ms - End Time: 278,916 ms.

I´m using a Tesla K20c with 13 SMs which can execute up 16 blocks per SM. Both kernels have 100% occupancy and launch an enough amount of blocks:

• Kernel 1) 2277 blocks, 32 Register/Thread, 1,156 KB shared memory.
• Kernel 2) 46555 blocks, 32 Register/Thread, 1,266 KB shared memory.

With this configuration, both kernels shouldn´t show this behaviour, since both kernels launch an enough number of blocks to fill all SMs of the GPU. However, Nvidia Visual Profiler shows that these kernels are being overlaped. Why?. Anyone could explain me why this behaviour can occur?

Many thanks in advance.

Answer 1

With this configuration, both kernels shouldn´t show this behaviour, since both kernels launch an enough number of blocks to fill all SMs of the GPU.

I think this is an incorrect statement. As far as I know, the low-level scheduling behavior of blocks is not specified in CUDA.

Newer devices (cc3.5+ with hyper-Q) can more easily schedule blocks from concurrent kernels at the same time.

So, if you launch 2 kernels (A and B), each with large numbers of blocks, concurrently, then you may observe either

• blocks from kernel A execute concurrently with kernel B
• all (or nearly all) of the blocks of kernel A execute before kernel B
• all (or nearly all) of the blocks of kernel B execute before kernel A

Since there is no specification at this level, there is no direct answer. Any of the above are possible. The low level block scheduler is free to choose blocks in any order, and the order is not specified.

If a given kernel launch "completely saturates" the machine (i.e. uses enough resources to fully occupy the machine while it is executing) then there is no reason to think that the machine has extra capacity for a second concurrent kernel. Therefore there would be no reason to expect much, if any, speed up from running the two kernels concurrently as opposed to sequentially. In such a scenario, whether they execute concurrently or not, we would expect the total execution time for the 2 kernels running concurrently to be approximately the same as the total execution time if the two kernels are launched or scheduled sequentially (ignoring tail effects and launch overheads, and the like).