TensorFlow error tensorflow/stream_executor/cuda/cuda_dnn.cc:338] Failed to create cuDNN handle: CUDNN_STATUS_INTERNAL_ERROR

I must admit this error was a major problem to solve and, to some extent, still is. I found and tried many methods discussed online in various forums, but none were ultimately helpful in permanently fixing the issue, at least in my case.

2019-05-05 09:34:42.298865: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-05-05 09:34:42.469839: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-05-05 09:34:43.086143: E tensorflow/stream_executor/cuda/cuda_dnn.cc:338] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-05-05 09:34:43.100978: E tensorflow/stream_executor/cuda/cuda_dnn.cc:338] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

Of course I carefully followed the library installation instructions as indicated on the official tensorflow site and tried various available CUDA versions from 10.1 down to 9.0 and the corresponding cuDNN versions, but the error could not be eliminated.

In the end I realized the problem lies in the compatibility of the libraries installed in the environment and it is not easy to determine which one in particular causes the issue. After several attempts I discovered that by using Anaconda to install tensorflow-gpu in a virtual environment, CUDA and cuDNN compatible with the version of TensorFlow to be used are automatically installed in the environment. I also verified that the working version is 1.12. Therefore, to solve the problem it is sufficient to create a virtual environment by running, for example, the command:

conda create --name tf12

and install tensorflow-gpu by running the following command:

conda install tensorflow-gpu==1.12

At the moment I haven’t found an alternative way to use the GPU with TensorFlow for more recent versions.

A small note: if there are multiple GPUs it is possible to set which GPU to use by running the following command:

export CUDA_VISIBLE_DEVICES=1

If you have only one GPU and you use the previous setting, no GPU will actually be used and the CPU will be used instead. This workaround is often suggested in posts as a fix for the aforementioned bug, but obviously it is not.

In the case of a single GPU the correct setting should be:

export CUDA_VISIBLE_DEVICES=0