Java Protocol Buffers - Message sizes

Question

So, for the past few weeks I've been learning very simple network programming and protocol buffers. Right now, I have a Java client and a C# server that are communicating back and forth using the protocol buffers. It's all working fine, but to make it work on the client (Java) side I had to create my byte array with the exact size of the incoming message or else the parser would throw an error of "Protocol message contained an invalid tag (zero)"

After doing some research, I came to find out that the array I had created (1024bytes) for my DatagramPacket had tons of trailing zeros (since my incoming data from the server was 27bytes long), and that's why I now, as previously mentioned, have to create the array with the exact size of the incoming data.

As for the question, is there any way to find out the size of all of my proto "messages" in my .proto files? If there isn't some sort of static getSize(), is there a way I can calculate that just by the types of fields I have within the "message"?

My message I'm using right now contains 3 doubles, and now that I'm thinking about it, but I want a for sure answer from someone who knows what's going on, is it 27 because 8bytes per double and the 1byte per "tag" on each message field?

Answer 1

The root object in protobuf data is not self-terminating; it is designed to be appendable (with append===merge), so normally the library simply reads until it runs out of data. If you have spare zeros, it will fail to parse the next field header. There are two ways of addressing this:

• if you only want to send one message, simply close the outbound socket at the end of you message; the client should detect the end of the socket and compensate accordingly (note, you still don't want to use an oversized buffer unless you are using a length-limited stream wrapper)
• use some kind of "framing" protocol; the simplest of which is simply to prefix each message with the number of bytes in that message (note that in the general case this size is not fixed, but in the case off 3 doubles, each with a field-header of a field-number no-greater-than 16, then yes: it will be 27 bytes); you would then either create the buffer the right size (noting that repeated array allocations can be expensive), or more typically: use a length-limited stream wrapper, or a memory-backed in-memory stream

Answer 2

I believe your problem lies in your socket receive code. Having an array with trailing zeroes is not a problem, but when receiving you should check the number of bytes received (it is the return value of a receive call) and only consider the bytes of the buffer array from the beginning up to "bytes received".