Quite often, I get the CUDA library to completely fail and return with an error 46 ("all CUDA-capable devices are busy or unavailable") even for simple calls like cudaMalloc. The code runs successfully if I restart the computer, but this is far from ideal. This problem is apparently quite common.
My setup is the following:
- OSX 10.6.8
- NVIDIA CUDA drivers : CUDA Driver Version: 4.0.31 (latest)
- GPU Driver Version: 1.6.36.10 (256.00.35f11)
I tried many solutions from the Nvidia forum, but it didn't work. I don't want to reboot every time it happens. I also tried to unload and reload the driver with a procedure I assume to be correct (may not be)
kextunload -b com.nvidia.CUDA
kextload -b com.nvidia.CUDA
But still it does not work. How can I kick the GPU (or CUDA) back into sanity ?
This is the device querying result
CUDA Device Query (Runtime API) version (CUDART static linking)
Found 1 CUDA Capable device(s)
Device 0: "GeForce 9400M"
CUDA Driver Version / Runtime Version 4.0 / 4.0
CUDA Capability Major/Minor version number: 1.1
Total amount of global memory: 254 MBytes (265945088 bytes)
( 2) Multiprocessors x ( 8) CUDA Cores/MP: 16 CUDA Cores
GPU Clock Speed: 1.10 GHz
Memory Clock rate: 1075.00 Mhz
Memory Bus Width: 128-bit
Max Texture Dimension Size (x,y,z) 1D=(8192), 2D=(65536,32768), 3D=(2048,2048,2048)
Max Layered Texture Size (dim) x layers 1D=(8192) x 512, 2D=(8192,8192) x 512
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 16384 bytes
Total number of registers available per block: 8192
Warp size: 32
Maximum number of threads per block: 512
Maximum sizes of each dimension of a block: 512 x 512 x 64
Maximum sizes of each dimension of a grid: 65535 x 65535 x 1
Maximum memory pitch: 2147483647 bytes
Texture alignment: 256 bytes
Concurrent copy and execution: No with 0 copy engine(s)
Run time limit on kernels: Yes
Integrated GPU sharing Host Memory: Yes
Support host page-locked memory mapping: Yes
Concurrent kernel execution: No
Alignment requirement for Surfaces: Yes
Device has ECC support enabled: No
Device is using TCC driver mode: No
Device supports Unified Addressing (UVA): No
Device PCI Bus ID / PCI location ID: 2 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 4.0, CUDA Runtime Version = 4.0, NumDevs = 1, Device = GeForce 9400M
[deviceQuery] test results...
PASSED
This is an example of code that may fail (although in normal conditions it does not)
#include <stdio.h>
__global__ void add(int a, int b, int *c) {
*c = a + b;
}
int main(void) {
int c;
int *dev_c;
cudaMalloc( (void **) &dev_c, sizeof(int)); // fails here, returning 46
add<<<1,1>>>(2,7,dev_c);
cudaMemcpy(&c, dev_c, sizeof(int), cudaMemcpyDeviceToHost);
printf("hello world, %d\n",c);
cudaFree( dev_c);
return 0;
}
I also found out that occasionally I get to revert back to a sane behavior without a reboot. I still don't know what triggers it.
sm_10andsm_20- never seen that error, but I'm only on CUDA 3.2. – koan