Keras with Tensorflow backend on GPU. MKL ERROR: Parameter 4 was incorrect on entry to DLASCL

Question

I installed Tensorflow with GPU support and Keras to an environment in Anaconda (v1.6.5) by using following commands:

conda install -n EnvName tensorflow-gpu
conda install -n EnvName -c conda-forge keras-gpu

I have NVIDIA Quadro 2200K on my machine with driver v384.66, cuda-8.0, cudnn 7.0

When I am trying to run a python code with Keras at the stage of training I get the following

Intel MKL ERROR: Parameter 4 was incorrect on entry to DLASCL.

and later

File "/home/User/anaconda3/envs/keras_gpu/lib/python3.6/site-packages/numpy/linalg/linalg.py", line 99, in _raise_linalgerror_svd_nonconvergence raise LinAlgError("SVD did not converge") numpy.linalg.linalg.LinAlgError: SVD did not converge

Other relevant sources suggest to check data for NaNs and Infs, but my data is clean for sure. By the way, CPU version of the installation is working fine, the issue occurs only when trying to run on GPU

I tried to reinstall Anaconda, to reinstall CUDA and numpy, but it didn't work out.

Answer 1

The problem was in package mkl (2018.0.0) - it seems like it has recently been released and conflicts with the version of some packages supplied with Tensorflow(1.3.0) and Keras(2.0.5) via conda*.

So I manually downgraded mkl using Anaconda Navigator to v11.3.3 which led automatically to downgrade of other packages and everything is working well now.

Answer 2

If you're using Anaconda, this will help:

conda update mkl