How do I export a graph to Tensorflow Serving so that the input is b64?

3
votes

I have a Keras graph with a float32 tensor of shape (?, 224, 224, 3) that I want to export to Tensorflow Serving, in order to make predictions with RESTful. Problem is that I cannot input tensors, but encoded b64 strings, as that is a limitation of the REST API. That means that when exporting the graph, the input needs to be a string that needs to be decoded.

How can I "inject" the new input to be converted to the old tensor, without retraining the graph itself? I have tried several examples [1][2].

I currently have the following code for exporting: 

image = tf.placeholder(dtype=tf.string, shape=[None], name='source')


signature = predict_signature_def(inputs={'image_bytes': image},
                                 outputs={'output': model.output})

I somehow need to find a way to convert image to model.input, or a way to get the model output to connect to image.

Any help would be greatly appreciated!

2
resttensorflowkerasbase64serving

2 Answers

2
votes

You can use tf.decode_base64:

image = tf.placeholder(dtype=tf.string, shape=[None], name='source')
image_b64decoded = tf.decode_base64(image)
signature = predict_signature_def(inputs={'image_bytes': image_b64decoded},
                                 outputs={'output': model.output})

EDIT:

If you need to use tf.image.decode_image, you can get it to work with multiple inputs using tf.map_fn:

image = tf.placeholder(dtype=tf.string, shape=[None], name='source')
image_b64decoded = tf.decode_base64(image)
image_decoded = tf.map_fn(tf.image.decode_image, image_b64decoded, dtype=tf.uint8)

This will work as long as the images have all the same dimensions, of course. However, the result is a tensor with completely unknown shape, because tf.image.decode_image can output a different number of dimensions depending on the type of image. You can either reshape it or use another tf.image.decode_* call so at least you have a known number of dimensions in the tensor.

0
votes

Creating an export_model may be an easier way. One example in tensorflow.org

  1. The Keras graph with a float32, shape (?, 224, 224, 3) tensor

model = ...

  1. Define a function to preprocess b64 image
def preprocess_input(base64_input_bytes):
    def decode_bytes(img_bytes):
        img = tf.image.decode_jpeg(img_bytes, channels=3)
        img = tf.image.resize(img, (224, 224))
        img = tf.image.convert_image_dtype(img, tf.float32)
        return img

    base64_input_bytes = tf.reshape(base64_input_bytes, (-1,))
    return tf.map_fn(lambda img_bytes:
                     decode_bytes(img_bytes),
                     elems=base64_input_bytes,                     
                     fn_output_signature=tf.float32)
  1. Export a serving model
serving_inputs = tf.keras.layers.Input(shape=(), dtype=tf.string, name='b64_input_bytes')
serving_x = tf.keras.layers.Lambda(preprocess_input, name='decode_image_bytes')(serving_inputs)
serving_x = model(serving_x)
serving_model = tf.keras.Model(serving_inputs, serving_x)
tf.saved_model.save(serving_model, serving_model_path)
  1. Serving
import requests
data = json.dumps({"signature_name": "serving_default", "instances": [{"b64_input_bytes": {"b64": b64str_1}}, {"b64_input_bytes": {"b64": b64str_2}}]})
headers = {"content-type": "application/json"}
json_response = requests.post('http://localhost:8501/v1/models/{model_name}:predict', data=data, headers=headers)
predictions = json.loads(json_response.text)['predictions']