Converting gRPC protobuf message to json runs for long

0
votes

We are trying to convert a gRPC protobuf message to finally be a json format object for processing in python. The data sent across from server in serialized format is around 35MB and there is around 15K records. But when we convert protobuf message into string (using MessageToString) it is around 135 MB and when we convert protobuf message into a JSON string (using MessageToJson) it is around 140MB. But the time taken for conversion is around 5 minutes for each. It does not add any value wherein we take so much time to convert data on the client side.

Any thoughts or suggestion or caveats that we are missing would be helpful. Thanks.

1
pythonjsonprotocol-buffersgrpc
Why don't you skip the JSON conversion? Python has support for protobuf: developers.google.com/protocol-buffers/docs/pythontutorial – Blender
@Blender : Thanks! We have a streaming gRPC api that is being consumed on the client side and once data is received, it will be processed using python(pandas) for creating a file for consumers in a particular format. The data that is being passed is nested and has around 15-20k records. That is where we are trying to get the data in json for processing. Hope this did not muddy the waters, which wasnt already! :) – Sandesh
Fixed the issue by only picking the fields that is needed when deserializing the data, rather than deserialize all the data returned from the server. – Sandesh

1 Answers

0
votes

Fixed the issue by only picking the fields that is needed when deserializing the data, rather than deserialize all the data returned from the server.