keras(-gpu) + tensorflow-gpu + anaconda on Kubuntu

Question

I have Kubuntu 18.04 and Anaconda 5.2 64. I installed the CUDA drivers and keras-gpu and tensorflow-gpu (automatically also installed tensorflow).

The following code

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras import backend as K

import tensorflow as tf
print('Tensorflow: ', tf.__version__)

gives the output

2018-07-29 12:14:06.821996: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-07-29 12:14:06.880569: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-07-29 12:14:06.880910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties:
name: GeForce GTX 980 major: 5 minor: 2 memoryClockRate(GHz): 1.2155
pciBusID: 0000:01:00.0
totalMemory: 3.95GiB freeMemory: 2.72GiB
2018-07-29 12:14:06.880924: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2018-07-29 12:14:07.058984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-29 12:14:07.059012: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958]      0
2018-07-29 12:14:07.059017: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   N
2018-07-29 12:14:07.059114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/device:GPU:0 with 2430 MB memory) -> physical GPU (device: 0, name: GeForce GTX 980, pci bus id: 0000:01:00.0, compute capability: 5.2)
[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 18195666940796676435
, name: "/device:GPU:0"
device_type: "GPU"
memory_limit: 2548367360
locality {
  bus_id: 1
  links {
  }
}
incarnation: 7016427886680347829
physical_device_desc: "device: 0, name: GeForce GTX 980, pci bus id: 0000:01:00.0, compute capability: 5.2"
]
Using TensorFlow backend.
Tensorflow:  1.9.0

So it seems keras is using tensorflow CPU instead of GPU (using DeepBach, one of my CPU cores is at 100%)? What am I doing wrong?

How can I find out which device is being used by keras/DeepBach? nvidia-smi shows no GPU utilisation during training with keras/DeepBach. How can I tell keras/DeepBach to use the GPU instead of the CPU?

CUDA seems to be installed:

$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176

$ nvidia-smi
Sun Jul 29 12:10:28 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.48                 Driver Version: 390.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 980     Off  | 00000000:01:00.0  On |                  N/A |
|  4%   62C    P0    47W / 180W |   1160MiB /  4040MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1113      G   /usr/lib/xorg/Xorg                           562MiB |
|    0      1385      G   kwin_x11                                     152MiB |
|    0      1395      G   /usr/bin/krunner                               2MiB |
|    0      1399      G   /usr/bin/plasmashell                         167MiB |
|    0     26801      G   ...-token=2DD4BBFEA86302FEC3C179E07D55C897   267MiB |
+-----------------------------------------------------------------------------+

Your log doesn't say anything about which device is being used, you will need to be more specific on the problem. The GPU is also correctly detected by TF, so I don't see any problem. — Dr. Snoopy
Comments are not to ask new questions, they are to clarify your question. — Dr. Snoopy

Afshin Amiri Afshin Amiri · Accepted Answer · 2018-11-08T22:03:02

While you have your code running check system-monitor to see if GPU is involved or not. Check specifically for Gpu's memory usage

keras(-gpu) + tensorflow-gpu + anaconda on Kubuntu

2 Answers