TensorBoard debugger and gRPC max message size between client and server

2
votes

Trying to debug a TensorFlow model with the new TensorBoard debugger GUI. The communication between client (TensorFlow side) and server (TensorBoard side) fails with the following message:

grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.RESOURCE_EXHAUSTED, Received message larger than max (11373336 vs. 4194304))>

Apparently the issue is well known in general and there are ways tricks to modify the max message size in grpc. However, in TensorFlow this is transparent to the user given that I am using the tf_debug.TensorBoardDebugWrapperSession wrapper.

My question is how to increase the max message size so I can debug my model. I am using TensorFlow 1.6 with Python 3.6.

Thank you!

1
python-3.xdebuggingtensorflowgrpctensorboard
Have you looked at "channel arguments"? They are an intentionally-underdocumented "advanced" feature. Perhaps one of grpc.max_send_message_size and grpc.max_receive_message_size are arguments that you want to set? - Nathaniel Manista At Google
Like OP, I'm using TensorBoardDebugWrapperSession (but with TensorFlow 1.8) and not using grpc directly. I don't understand how to set these options. (I tried naively importing grpc and setting them, but this didn't work.). Also, nothing about my model is really unusually large, so I am surprised I would have to increase defaults like this. - gmr

1 Answers

0
votes

Can you try creating TensorBoardDebugWrapperSession or TensorBoardDebugHook with the keyword argument send_source=False as a workaround? The root cause is that large source file sizes and/or large number of source files causes the gRPC message to exceed the 4-MB message size limit.

The issue will be fixed in the next release of TensorFlow and TensorBoard.