Python 3 - Google protobuf response - decode without .proto file

Question

I have an issue decoding the google protobuf response without .proto file, had implemented with proto file and working fine, but in this case the .proto file is not available.

Using python 3+ and from the tunnel getting this response

b'\x08\x00\x12\x88\x01\x08\xda\xc9\x06\x10\xb6\xc9\x03\x18\xa1\x8b\xb8\x01 \x00*\x00:\x00B\x00J\x00R\x00Z\x00b\x00j\x00r\x00z\x00\x80\x01\xe9\x9b\x8c\xb5\x99-\x90\x01d\x98\x01\xea\x9b\x8c\xb5\x99-\xa2\x01\x00\xaa\x01\x00\xb0\x01\x00\xb8\x01\x01\xc0\x0
1\x00\xd1\x01\x00\x00\x00\x00\x00\x00\x00\x00\xd9\x01\x00\x00\x00\x00\x00\x00\x00\x00\xe1\x01\x00\x00\x00\x00\x00\x00\x00\x00\xea\x01\x00\xf0\x01\x01\xf8\x01\x00\x80\x02\x00\x88\x02\x00\x90\x02\x00\x98\x02\x00\xa8\x02\x00\xb0\x02\x00\xb8\x02\x90N\xc0\x02\x00\xc8\x0
2\x00'

There is a way to decode the google ptobuf without .proto file and make it a dict?

my code to achieve this is below:

import pika

credentials = pika.PlainCredentials('demo', 'demo')

cp = pika.ConnectionParameters(
    host='127.0.0.1',
    port=5671,
    credentials=credentials,
    ssl=False,
)

connection = pika.BlockingConnection(cp)
channel = connection.channel()

def callback(ch, method, properties, body):
    print(" [x] Received %r" % body)

channel.basic_consume(callback, queue='demo_queu', no_ack=True)

print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()

need to achieve:

to get body and decode it in human readable

any idea will be appreciated

Your best bet is to reverse-engineer the .proto file; there is a tool in protoc for this, but I typically use protogen.marcgravell.com/decode to help me figure out the fields. Is this approach viable for you? There are APIs for working with proto data without a .proto, but you still need to figure out what each field means, so you might as well just completely reverse engineer it. The tricky part is usually: signed vs unsigned on integers — Marc Gravell♦
can you articulate why not? the protobuf format is ambiguous in many ways without a schema, so you kinda need to figure out the layout with some manual inspection, using tools like protoc (I think --decode-raw ?) or the link above. Without that... I'm not sure what you expect to do with the bits — Marc Gravell♦
@MarcGravell i found on google protobuf to decode the message without own ptoro file but with the default google messages, can not find the article but tried a lot of times and no success, is that possible to use google protobuf default message to decode, if yes, can i have an example? spend already some weeks on this issue and no result — aaa

aaa aaa · Accepted Answer · 2019-03-21T07:21:12

Finally found a workaround by my self, maybe is a primitive way but only this worked for me.

SOLUTION:

1. Did list with all the descriptors from .proto file

    here is .proto file generated for python 3 is too big cant paste content here

    https://ufile.io/2p2d6

    descriptors = [proto_file.descriptor_1, proto_file.descriptor_2]

2. Loop throw list and pass one by one

for d in descriptors:
    decoded_response = proto_file._reflection.ParseMessage(d, raw_response.body)

3. Check if decoded_response is not blank

   if decoded_response:
       # descriptor was found
       # response is decoded
   else:
       # no descriptor

4. After decoded response we go parse it into dict:

from protobuf_to_dict import protobuf_to_dict

decoded_response_to_dict = protobuf_to_dict(decoded_response)

This solution that spent weeks on it finally worked.

Python 3 - Google protobuf response - decode without .proto file

1 Answers