Unable to run tensorflow examples. CUBLAS error

0
votes

I had this all working about a month ago and am trying to run everything again. I upgraded all the moving parts and am at the point where I am trying to run the tensorflow examples found here: tensorflow git models.

Setup:

  • Ubuntu 16.04
  • CUDA version: release 8.0, V8.0.61 (able to run deviceQuery etc.)
  • CUDNN_MAJOR 5 (It doesnt matter if this is 5.1 or 6)
  • Tensorflow backend version: 1.4.0 (not able to run examples )
  • keras version: 2.1.1

This is the error I get:

2017-11-20 15:18:00.209182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1) 2017-11-20 15:18:07.457924: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2017-11-20 15:18:07.474515: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2017-11-20 15:18:07.558178: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2017-11-20 15:18:07.575864: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2017-11-20 15:18:07.593372: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2017-11-20 15:18:07.610621: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2017-11-20 15:18:08.431784: E tensorflow/stream_executor/cuda/cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2017-11-20 15:18:08.431816: E tensorflow/stream_executor/cuda/cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM 2017-11-20 15:18:08.431838: F tensorflow/core/kernels/conv_ops.cc:667] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo(), &algorithms) Aborted (core dumped)

Any advice would be much appreciated.

EDIT: This question: tensorflow running error with cublas is similar - but the answer did not work for me.

1
python-2.7tensorflowubuntu-16.04
Please help me understand why the vote to close? Just let me know if there is anything that I can do to improve the question rather than just closing it. Please. – ashley
On which model did it fail? Please provide more information, so that others can either reproduce the problem, isolate the problem, or reduce the problem. I'm having similar error: E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED – Yu Shen
I've put in an issue: github.com/tensorflow/models/issues/2857. At the moment its just getting any model running from github.com/tensorflow/models – ashley
models/tutorials/image/imagenet/ classify_image.py – ashley
I tried: 'python classify_image.py' it worked as expected, no error. It might be related to some environment variations. After you finished installing Tensorflow, there is some simple test program to verify the installation. Have you tried that? Thanks for sharing your case. – Yu Shen

1 Answers

1
votes

This is not much of a solution - but its the only one that worked:

CUDNN 6 doesn’t play nicely with TF. So if you have the library anywhere it crashes. Basically it’s version control not memory or paths. Had to make sure CUDNN v6 was no where to be found and TF was version 1.2 not 1.4.

Final set up: Ubuntu 16.04. CUDA 8, CUDNN 5.1, TF 1.2.

Installing tf version can be done with:

$ sudo apt install tensorflow-gpu==1.2

This helped me through the dark times: TensorFlow Framework and listed a couple tests along the way for sanity checking.