When compiling Tensorflow with GPU support, CUDA_TOOLKIT_PATH unbound variable is encountered

Question

I am following this link to compile Tensorflow with GPU support. After ./configure, I then type in bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package. During the compilation, I see the following error.

ERROR: /home/super/.cache/bazel/_bazel_super/6fad3aac5da8510114d7933cea773172/external/nccl_archive/BUILD.bazel:139:1: Executing genrule @nccl_archive//:device_code_fatbin_h failed (Exit 1)
/bin/bash: CUDA_TOOLKIT_PATH: unbound variable
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.973s, Critical Path: 0.12s
INFO: 4 processes: 4 local.
FAILED: Build did NOT complete successfully

I checked the .tf_configure.bazelrc file and the CUDA_TOOLKIT_PATH is defined.

build --action_env PYTHON_BIN_PATH="/home/super/anaconda3/envs/tf/bin/python"
build --action_env PYTHON_LIB_PATH="/home/super/anaconda3/envs/tf/lib/python3.6/site-packages"
build --python_path="/home/super/anaconda3/envs/tf/bin/python"
build:xla --define with_xla_support=true
build --config=xla
build --action_env TF_NEED_OPENCL_SYCL="0"
build --action_env TF_NEED_ROCM="0"
build --action_env TF_NEED_CUDA="1"
build --action_env CUDA_TOOLKIT_PATH="/usr/local/cuda"
build --action_env TF_CUDA_VERSION="10.0"
build --action_env CUDNN_INSTALL_PATH="/usr/local/cuda-10.0"
build --action_env TF_CUDNN_VERSION="7"
build --action_env TF_NCCL_VERSION=""
build --action_env TF_CUDA_COMPUTE_CAPABILITIES="5.2"
build --action_env LD_LIBRARY_PATH="/usr/local/cuda-10.0/lib64"
build --action_env TF_CUDA_CLANG="0"
build --action_env GCC_HOST_COMPILER_PATH="/usr/bin/gcc"
build --config=cuda
test --config=cuda
build:opt --copt=-march=native
build:opt --copt=-Wno-sign-compare
build:opt --host_copt=-march=native
build:opt --define with_default_optimizations=true
build:v2 --define=tf_api_version=2

My .bashrc has the following.

export PATH=/usr/local/cuda-10.0/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
export CUDA_HOME=/usr/loca/cuda
export CUDA_TOOLKIT_PATH=/usr/local/cuda

My environment settings are as follows.

bazel 0.18.0
CUDA 10.0
cuDNN 7.3.1
Ubuntu 16.04

Any ideas on what's going on?

kautenja kautenja · Accepted Answer · 2018-10-31T06:53:48

My assumption is that you're trying to compile directly from the master branch. I was running into the same issue earlier, but after checking out a published release (v1.12.0-rc2), I was able to compile TensorFlow without anymore issues. Try git checkout v1.12.0-rc2 then run through the ./configure and bazel compilation again. The master branch updates very frequently. My thought is that the build was broken at the moment we were trying to install.

When compiling Tensorflow with GPU support, CUDA_TOOLKIT_PATH unbound variable is encountered

1 Answers