CUDA driver version is insufficient for CUDA runtime version , CUDA version 10.0

Question

I get this error when I run keras on GPU.

  File "modeltra.py", line 165, in <module>
    x_train = a_model.predict(y_training)
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/keras/engine/training.py", line 1169, in predict
    steps=steps)
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/keras/engine/training_arrays.py", line 294, in predict_loop
    batch_outs = f(ins_batch)
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 2697, in __call__
    if hasattr(get_session(), '_make_callable_from_options'):
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 186, in get_session
    _SESSION = tf.Session(config=config)
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1570, in __init__
    super(Session, self).__init__(target, graph, config=config)
  File "/home/hgam/anaconda3/envs/gp_ker_pip/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 693, in __init__
    self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts)
tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version

My GPU information is as follows

(gp_ker_pip) [hgam@xx]$ nvidia-smi
Tue Aug 27 17:29:34 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  On   | 00000000:03:00.0 Off |                    0 |
| N/A   34C    P0    33W / 250W |    855MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P100-PCIE...  On   | 00000000:82:00.0 Off |                    0 |
| N/A   32C    P0    31W / 250W |    855MiB / 16280MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

(gp_ker_pip) [hgam@xx]$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

First installed tensorflow using pip install tensorflow-gpu

and installed keras with conda install keras-gpu

What should I follow to make this work? Also how can I check the CUDA runtime version and the driver version?

PS: I also downgraded the cuda version using

>    export PATH=$PATH:/usr/local/cuda-9.0/bin
>    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-9.0/lib64

Still I get the error.

What should I follow to make this work? install the latest driver for your GPU — Robert Crovella

TeenyTinySparkles TeenyTinySparkles · Accepted Answer · 2019-08-28T16:22:51

This issue was solved after installing CUDA 10.1 as @Robert suggested.

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:03:00.0 Off |                    0 |
| N/A   39C    P0    64W /  70W |  14503MiB / 15079MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            Off  | 00000000:82:00.0 Off |                    0 |
| N/A   35C    P0    25W /  70W |    111MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

CUDA driver version is insufficient for CUDA runtime version , CUDA version 10.0

1 Answers