Sagemaker Predict on local instance, JSON Error

Question

I'm trying to make transfer learning method on MXNet on Sagemaker instance. Train and serve start locally without any problem and I'm using that python code to predict:

def predict_mx(net, fname):
    with open(fname, 'rb') as f:
      img = image.imdecode(f.read())
      plt.imshow(img.asnumpy())
      plt.show()
    data = transform(img, -1, test_augs)
    plt.imshow(data.transpose((1,2,0)).asnumpy()/255)
    plt.show()
    data = data.expand_dims(axis=0)
    return net.predict(data.asnumpy().tolist())

I checked data.asnumpy().tolist() that is ok and pyplot draw images (firts is the original image, the second is the resized image). But net.predict raise an error:

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
<ipython-input-171-ea0f1f5bdc72> in <module>()
----> 1 predict_mx(predictor.predict, './data2/burgers-imgnet/00103785.jpg')

<ipython-input-170-150a72b14997> in predict_mx(net, fname)
     30     plt.show()
     31     data = data.expand_dims(axis=0)
---> 32     return net(data.asnumpy().tolist())
     33 

~/Projects/Lab/ML/AWS/v/lib64/python3.6/site-packages/sagemaker/predictor.py in predict(self, data)
     89         if self.deserializer is not None:
     90             # It's the deserializer's responsibility to close the stream
---> 91             return self.deserializer(response_body, response['ContentType'])
     92         data = response_body.read()
     93         response_body.close()

~/Projects/Lab/ML/AWS/v/lib64/python3.6/site-packages/sagemaker/predictor.py in __call__(self, stream, content_type)
    290         """
    291         try:
--> 292             return json.load(codecs.getreader('utf-8')(stream))
    293         finally:
    294             stream.close()

/usr/lib64/python3.6/json/__init__.py in load(fp, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    297         cls=cls, object_hook=object_hook,
    298         parse_float=parse_float, parse_int=parse_int,
--> 299         parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
    300 
    301 

/usr/lib64/python3.6/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    352             parse_int is None and parse_float is None and
    353             parse_constant is None and object_pairs_hook is None and not kw):
--> 354         return _default_decoder.decode(s)
    355     if cls is None:
    356         cls = JSONDecoder

/usr/lib64/python3.6/json/decoder.py in decode(self, s, _w)
    337 
    338         """
--> 339         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    340         end = _w(s, end).end()
    341         if end != len(s):

/usr/lib64/python3.6/json/decoder.py in raw_decode(self, s, idx)
    355             obj, end = self.scan_once(s, idx)
    356         except StopIteration as err:
--> 357             raise JSONDecodeError("Expecting value", s, err.value) from None
    358         return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I tried to json.dumps my data, and there is no problem with that.

Note that I didn't deployed the service on AWS yet, I want to be able to test the model and prediction locally before to make a larger train and to serve it later.

Thanks for your help

Answer 1

The call to net.predict is working fine.

It seems that you are using the SageMaker Python SDK predict_fn for hosting. After the predict_fn is invoked, the MXNet container will try to serialize your prediction to JSON before sending it back to the client. You can see code that does that here: https://github.com/aws/sagemaker-mxnet-container/blob/master/src/mxnet_container/serve/transformer.py#L132

The container is failing to serialize because net.predict does not return a serializable object. You can solve this issue by returning a list instead:

return net.predict(data.asnumpy().tolist()).asnumpy().tolist()

Another alternative is to use a transform_fn instead of prediction_fn so you can handle the output serialization yourself. You can see an example of a transform_fn here https://github.com/aws/sagemaker-python-sdk/blob/e93eff66626c0ab1f292048451c4c3ac7c39a121/examples/cli/host/script.py#L41

Answer 2

You have an issue with the deserialization of the data that's being passed from the notebook to the prediction environment (in docker), but I have not been able to reproduce this given the provided code. When using the MXNet estimator (e.g from sagemaker.mxnet import MXNet), you can implement the transform_fn in the entrypoint script to deserialize the data and make the prediction with the model. Use json.loads at the start of the function, as in the example below;

def transform_fn(net, data, input_content_type, output_content_type):
    """
    Transform a request using the Gluon model. Called once per request.
    :param net: The Gluon model.
    :param data: The request payload.
    :param input_content_type: The request content type.
    :param output_content_type: The (desired) response content type.
    :return: response payload and content type.
    """
    # we can use content types to vary input/output handling, but
    # here we just assume json for both
    parsed = json.loads(data)
    nda = mx.nd.array(parsed)
    output = net(nda)
    prediction = mx.nd.argmax(output, axis=1)
    response_body = json.dumps(prediction.asnumpy().tolist()[0])
    return response_body, output_content_type

You should inspect the value of data if you're still having issues with the json.loads command, and look carefully for issues related to encodings (i.e. string starting with \ which wouldn't be valid).

Note: You've also got different code in the function and the stack trace, so you might want to confirm you're running what you think you're running. And you mention that you haven't deployed (local or instance), but this is required for prediction.