error while running tensorflow on P100

Question

I am trying to run tf_cnn_benchmarks on P100 GPU. I am using TF-1.60.

I am getting the following message:

2018-03-23 11:02:22.043275: E tensorflow/core/common_runtime/executor.cc:645] Executor failed to create kernel. Invalid argument: Default MaxPoolingOp only supports NHWC on device type CPU [[Node: v/tower_0/cg/mpool0/MaxPool = MaxPoolT=DT_FLOAT, data_format="NCHW", ksize=[1, 1, 3, 3], padding="SAME", strides=[1, 1, 2, 2], _device="/job:localhost/replica:0/task:0/device:CPU:0"]] I0323 11:02:22.115472 47942823427200 tf_logging.py:116] Error reported to Coordinator: , Default MaxPoolingOp only supports NHWC on device type CPU [[Node: v/tower_0/cg/mpool0/MaxPool = MaxPoolT=DT_FLOAT, data_format="NCHW", ksize=[1, 1, 3, 3], padding="SAME", strides=[1, 1, 2, 2], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

I am confused. Why it is asking for stuff for CPU. I am using the following command:

srun python tf_cnn_benchmarks.py --num_gpus=1 --batch_size=32 --model=resnet50 --variable_update=parameter_server

Thanks

Answer 1

The problem is the format of your image data. You've set the data_format argument of tf.nn.max_pool to NHWC, which is the default value. But this isn't supported on the P100 GPU. I'd try setting the argument to NCHW or NCHW_VECT_C.

In the format description, N stands for the batch number, H stands for height, W stands for width, and C stands for channel. So if you set the argument to NCHW, you'll have to switch the components of your input image tensor.