Where should pre-processing and post-processing steps be executed when a TF model is served using TensorFlow serving?

44
votes

Typically to use a TF graph, it is necessary to convert raw data to numerical values. I refer to this process as a pre-processing step. For example, if the raw data is a sentence, one way is to do this is to tokenize the sentence and map each word to a unique number. This preprocessing creates a sequence of number for each sentence, which will be the input of the model.

We need also to post-process the output of a model to interpret it. For example, converting a sequence of numbers generated by the model to words and then building a sentence.

TF Serving is a new technology that is recently introduced by Google to serve a TF model. My question is that:

Where should pre-processing and post-processing be executed when a TF model is served using TensorFlow serving?

Should I encapsulate pre-processing and post-processing steps in my TF Graph (e.g. using py_fun or map_fn) or there is another TensorFlow technology that I am not aware of.

1
tensorflowtensorflow-serving
py_fun cannot be a solution as mentioned here: The tf.py_func() operation has the following known limitations: The body of the function (i.e. func) will not be serialized in a GraphDef. Therefore, you should not use this function if you need to serialize your model and restore it in a different environment. - MajidL
Did you ever resolve this? It doesn't appear as if tf.transform can support sophisticated tokenization yet. - Luke
From my understanding, there is no easy way to handle this issue. Ideally, you want to use Tensorflow ops (or adding necessary ones) to implement pre/post processing steps and use tf.transform to ship these ops along with your TF graph. However, as you may guess, adding a new TF OPs is not a trivial task and this adds a lot of limitation for implementing pre/post-processing steps. For sure you can always do pre/post processing outside graph, but this is not an ideal solution. - MajidL
This is my number one problem with using tensorflow-serving right now. I made an issue in tensorflow-serving on this topic. - Luke

1 Answers

5
votes

I'm running over the same issue here, even if I'm not 100% sure yet on how to use the wordDict variable (I guess you use one too to map the words with its ids), the main pre-process and post-process functions are defined here:

https://www.tensorflow.org/programmers_guide/saved_model

as export_outputs and serving_input_receiver_fn.

  • exports_outputs

Needs to be defined in EstimatorSpec if you are using estimators. Here is an example for a classification algorithm

  predicted_classes = tf.argmax(logits, 1)
  categories_tensor = tf.convert_to_tensor(CATEGORIES, tf.string)
  export_outputs = { "categories": export_output.ClassificationOutput(classes=categories_tensor) }
  if mode == tf.estimator.ModeKeys.PREDICT:
    return tf.estimator.EstimatorSpec(
        mode=mode,
        predictions={
            'class': predicted_classes,
            'prob': tf.nn.softmax(logits)
        },
        export_outputs=export_outputs)
  • serving_input_receiver_fn

It needs to be defined on before exporting the trained estimator model, it assumes the input is a raw string and parses your input from there, you can write your own function but I'm unsure whenever you can use external variables. Here is a simple example for a classification algorithm:

def serving_input_receiver_fn():
    feature_spec = { "words": tf.FixedLenFeature(dtype=tf.int64, shape=[4]) }
    return tf.estimator.export.build_parsing_serving_input_receiver_fn(feature_spec)()

  export_dir = classifier.export_savedmodel(export_dir_base=args.job_dir,
                                            serving_input_receiver_fn=serving_input_receiver_fn)

hope it helps.