Bidirectional Streaming using Tensorflow Serving

Question

I have a model that accepts an arbitrary-length stream of data and performs classification. I use Tensorflow Serving to listen to gRPC requests and perform the classification on a trained model.

Google Cloud Speech API has the "Streaming Speech Recognition" feature, that is available when using gRPC requests, which "allows you to stream audio to the Cloud Speech API and receive a stream [of] speech recognition results in real time as the audio is processed".

I believe this is possible due to the Bidirectional streaming RPC that is described in gRPC documentation whereby "the server and client could “ping-pong”: the server gets a request, then sends back a response, then the client sends another request based on the response, and so on".

So now I'm wondering whether I can achieve something similar to Google Cloud Speech API streaming recognition using Tensorflow Serving. The only reference I could find about this in the official docs of TF Serving (unless I missed something) was when describing possible future improvements: "Servables can be of any type and interface, enabling flexibility and future improvements such as: streaming results [...]".

Is it already possible to achieve this functionality (bidirectional streaming) using TF Serving? If so, how? If not, what would be the best way to go about extending TF Serving to add this feature?

Answer 1

It appears that this is in fact currently unavailable. I've filed a feature request on TensorFlow Serving GitHub repository and it was declined with the following response:

TF-Serving does not support this use-case out of the box at this time, but as you allude to it would be possible to leverage its extensiblity to do so without doing major surgery on the internals. We do not currently have any short-term plans to add it to the official distribution.

They were also unwilling to give any guidance:

Unfortunately this falls outside the scope of what we can support at this time, so you'll have to devise a solution and keep it in your own repo.