Using GPU error when use TensorFlow to train image

0
votes

When I am runing a tensorflow image train job in the container tensorflow/tensorflow:latest-gpu, it doesn't work.

Error message:

Cannot assign a device for operation InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D: Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:0 ]. Make sure the device specification refers to a valid device.
[[node InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/layers.py:1057)  = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true, _device="/device:GPU:0"](fifo_queue_Dequeue, InceptionV3/Conv2d_1a_3x3/weights/read)]]

GPU info: nvidia-smi Mon Nov 26 07:48:59 2018
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 410.72 Driver Version: 410.72 CUDA Version: 10.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 GeForce GT 630 Off | 00000000:01:00.0 N/A | N/A | | 25% 47C P0 N/A / N/A | 0MiB / 1998MiB | N/A Default | +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 Not Supported | +-----------------------------------------------------------------------------+

1
tensorflowgpu
Are you loading a previously trained model? – bluesummers
here is my step mkdir download_data_flower mkdir train_output python download_and_convert_data.py --dataset_name=flowers --dataset_dir=download_data_flower python train_image_classifier.py --batch_size=64 --model_name=inception_v3 --dataset_name=flowers --dataset_split_name=train --dataset_dir=download_data_flower --train_dir=train_output – tanxiuguang

1 Answers

0
votes

It seems that you Tensorflow is not detecting any gpu as available but maps the operations to GPU:0. First try this: 

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

And you'll get the available devices. Is there /device:GPU:0 ?