I I am studying word2vec of tensorflow. We bought two 1080i for parallel processing of gpu. Mounting was successful and p2p was successful. However, I tried to assign it to gpu using the command with tf.device ('/ gpu: 0') The following error occurs :
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 1 with properties:
name: GeForce GTX 1080 Ti
major: 6 minor: 1 memoryClockRate (GHz) 1.645
pciBusID 0000:66:00.0
Total memory: 10.91GiB
Free memory: 10.21GiB
tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 1
tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y Y
tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 1: Y Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:65:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:66:00.0)
I word2vec_kernels.cc:246] Data file: data/spouse_freebase/input2.nt contains 34966827 bytes, 2620786 words, 11769 unique words, 11769 unique frequent words.
E tensorflow/stream_executor/cuda/cuda_driver.cc:1276] failed to enqueue async memcpy from device to host: CUDA_ERROR_INVALID_VALUE; host dst: 0x104d5000000; GPU src: 0x7f12c800cbc0; size: 8=0x8
I tensorflow/stream_executor/stream.cc:1338] stream 0x39c2160 did not wait for stream: 0x39bf9a0
I tensorflow/stream_executor/stream.cc:3775] stream 0x39c2160 did not memcpy device-to-host; source: 0x3bd0d00
F tensorflow/core/common_runtime/gpu/gpu_util.cc:296] GPU->CPU Memcpy failed
I think this error is the out of memory of the gpu. I wait for your help. thank you.
