Google Colaboratory ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory when running

Question

I am following this article on AI Agriculture with Google Colaboratory

https://medium.com/deepquestai/ai-in-agriculture-detecting-defects-in-apples-b246799b329c

And when running the python file it gives an error

!python apple_detection_training.py

ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory when running

!pip install tensorflow-gpu==1.13.1
!pip install keras
!pip install opencv-python
!pip install imageai --upgrade
!unzip apple_detection_dataset.zip
!python apple_detection_training.py

Using TensorFlow backend. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in from tensorflow.python.pywrap_tensorflow_internal import * File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "/usr/lib/python3.6/imp.py", line 243, in load_module return load_dynamic(name, filename, file) File "/usr/lib/python3.6/imp.py", line 343, in load_dynamic return _load(spec) ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory During handling of the above exception, another exception occurred: Traceback (most recent call last): File "apple_detection_training.py", line 1, in from imageai.Detection.Custom import DetectionModelTrainer File "/usr/local/lib/python3.6/dist-packages/imageai/Detection/init.py", line 2, in from imageai.Detection.keras_retinanet.models.resnet import resnet50_retinanet File "/usr/local/lib/python3.6/dist-packages/imageai/Detection/keras_retinanet/models/resnet.py", line 19, in import keras File "/usr/local/lib/python3.6/dist-packages/keras/init.py", line 3, in from . import utils File "/usr/local/lib/python3.6/dist-packages/keras/utils/init.py", line 6, in from . import conv_utils File "/usr/local/lib/python3.6/dist-packages/keras/utils/conv_utils.py", line 9, in from .. import backend as K File "/usr/local/lib/python3.6/dist-packages/keras/backend/init.py", line 1, in from .load_backend import epsilon File "/usr/local/lib/python3.6/dist-packages/keras/backend/load_backend.py", line 89, in from .tensorflow_backend import * File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 5, in import tensorflow as tf File "/usr/local/lib/python3.6/dist-packages/tensorflow/init.py", line 24, in from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/init.py", line 49, in from tensorflow.python import pywrap_tensorflow File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in raise ImportError(msg) ImportError: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in from tensorflow.python.pywrap_tensorflow_internal import * File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in _pywrap_tensorflow_internal = swig_import_helper() File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) File "/usr/lib/python3.6/imp.py", line 243, in load_module return load_dynamic(name, filename, file) File "/usr/lib/python3.6/imp.py", line 343, in load_dynamic return _load(spec) ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory

Failed to load the native TensorFlow runtime.

Answer 1

cublas libraries are not available in Colab's CPU runtimes, but they are available on GPU runtimes. You should change to a GPU runtime (Runtime->Change Runtime Type and choose Hardware Accelerator->GPU) and then try your script again.

Answer 2

Update: I missed the part about this being on google colabs. See @Jakevdp answer.

I'm going to leave this answer here in case someone else happens to get this problem on their own machine.

Let's start with your error. libcublas.so.10.0 is being required by tensorflow, but it either isn't installed, or is in an unexpected location (one your python script doesn't have access to). cublas is an NVIDIA cuda implementation of the blas linear algebra routines, so if you don't have an NVIDIA graphics card, it doesn't make sense to use it, and you should just switch to a non-gpu tensorflow.

Otherwise, make sure that you have all of the dependencies listed here (NVIDIA drivers, CUDNN, CUDA toolkit), and that you have your LD_LIBRARY_PATH set to include the CUPTI location (listed in the reference as /usr/local/cuda/extras/CUPTI/lib64. With all of those pieces in place, you should be able to !pip install tensorflow-gpu, and get things running.

Getting GPU acceleration up and running can be a pain, especially if you're not used to dealing with library files. If you wanted to use Enthought's conda program for installation instead of following the directions above, I believe they ship the relevant cuda libraries as part of the package.

To just get things should working, you could try

!pip uninstall tensorflow-gpu
!pip install tensorflow
!pip install keras
!pip install opencv-python
!pip install imageai --upgrade
!unzip apple_detection_dataset.zip
!python apple_detection_training.py

to run the learning model on the CPU only.