Does code of CUDA kernel-function upload to the GPU-RAM at once or for each call to it?

0
votes

Does the executable code of CUDA kernel function upload to the GPU at once when starting program, or does the code upload each time a kernel function is called? Or in which cases may be one way or the other?

This can greatly influence the choice of programming methods:

  • A lot of calls to kernel-function from CPU-host
  • The use of dynamic parallelism and a lot of calls to kernel-functions from GPU-device
1
cudagpgpunvidia
The main overhead in launching a kernel is due to copying the kernel code itself to the GPU as well as copying the arguments. I think this is done each time the kernel is launched. - Vitality
@JackOLantern: can you cite a reference to support the first part of your comment? - talonmies
@talonmies PGI User Forum, kernel launch overhead post. - Vitality
Copying the kernel code to the GPU only has to be done once. Originally, it was done at module load time (for CUDA runtime apps, during the deferred initialization for a given GPU), but NVIDIA may have revisited that heuristic since. For example, they may load the microcode on first invocation of a given kernel. - ArchaeaSoftware

1 Answers

2
votes

When using the CUDA runtime API, kernel code is downloaded to the device once. This happens as an implicit action right after CUDA runtime context creation. When using the CUDA driver API, the app has control over when kernels get downloaded, and how often. It seems this is not currently covered by the CUDA documentation, I will file an enhancement request for that.