tensorflow GPU on AWS elastic beanstalk - tf import error with calling “python” + “sudo” ( libcublas.so.9.0 error)

0
votes

I am getting the

"ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory"

error when I call my application with "sudo python3.6 ..." but not when I call just "python3.6 ..."

elastic beanstalk uses "sudo", so removing the command is not an option.

Only CUDA 9.0 is installed (which is the correct version for our environment), and /usr/local/cuda/lib64/libcublas.so.9.0 exists .

we created a separate ec2 instance with the same config and same AMI ID: CUDA9ClassicAmazonLinuxDLAMIwithMXNetTensorflowandCaffe2, and had no issues. It seems to be an issue present only with the elastic beanstalk.

$LD_LIBRARY_PATH = '/usr/local/cuda/lib64:/usr/local/lib:/usr/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/mpi/lib:/lib/:/home/ubuntu/src/caffe2/build:/home/ec2-user/src/caffe2/build:/usr/local/cuda/lib64:/usr/local/lib:/usr/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/mpi/lib:/usr/local/cuda/lib64:/usr/local/lib:/usr/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/mpi/lib:/lib/:/home/ubuntu/src/caffe2/build:/home/ec2-user/src/caffe2/build:/usr/local/cuda/lib64:/usr/local/lib:/usr/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/mpi/lib:'

$PATH = '/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/home/ubuntu/src/caffe2/build:/home/ec2-user/src/caffe2/build:/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/home/ubuntu/src/caffe2/build:/home/ec2-user/src/caffe2/build:/usr/local/cuda/bin:/usr/local/bin:/opt/aws/bin:/usr/local/mpi/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/aws/bin:/home/ec2-user/.local/bin:/home/ec2-user/bin'

any thoughts?

1
pythonlinuxamazon-web-servicestensorflow
it's probably an environment issue. the sudo environment may not pick up your LD_LIBRARY_PATH variable. As a diagnostic, you could try: sudo -E python3.6 ..., see here - Robert Crovella
sudo -E python3.6 ... also gives the libcublas.so.9.0 error, we set os.environ['LD_LIBRARY_PATH'] within the python code as well, how would we go about setting the LD_LIBRARY_PATH so sudo can see it? - Megan Hardy
more diagnostic: 1. just to confirm, what is the output of echo $LD_LIBRARY_PATH, 2. what is the output of sudo -E bash -c 'echo $LD_LIBRARY_PATH' (you can edit your question with responses, if you wish) - Robert Crovella
I don't know if you were implying this, but os.environ['LD_LIBRARY_PATH'] by itself would not be sufficient. You would need to do something like os.environ['LD_LIBRARY_PATH']='/path/to/cudatoolkit/lib64', but probably that is what you meant. (If you did something like that before any import, I would have thought that would have fixed the issue as well.) - Robert Crovella
yes we are setting "os.environ['LD_LIBRARY_PATH'] = $path/to/libcublas.so.9.0" as shown above - Megan Hardy

1 Answers

4
votes

Add a file for setting the path to the new library(CUDA in your case) in 

/etc/ld.so.conf.d/foo.conf

The contents of the file is the path to the CUDA library (/usr/local/cuda/lib64)

And call 

 sudo ldconfig

to make sure the library is included in the environment.