Protobuf c# to java deserializing multiple messages

Question

I am trying to parse multiple protocol buffer messages in java which are generated in vb.net

I am using the excellent protobuf-net to stream multiple messages to java as per below:

ProtoBuf.Serializer.SerializeWithLengthPrefix(Of Msg)(postStream, msg, 
       ProtoBuf.PrefixStyle.Base128)

In Java, I am using following code to parse the messages

final byte[] buffer = new byte[4096];
for (int c = ins.read(buffer); c >= 0; c = ins.read(buffer)) {
  Msg msg = Msg.parseDelimitedFrom(new ByteArrayInputStream(buffer));
}

Problem is after first message is parsed, it throws error for parsing second time with following error:

com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either than the input has been truncated or that an embedded message misreported its own length.

Should the buffer size and message size be same? If yes, then how should I parse it, especially for large messages.

Removing [C#] and [vb.net] as the answer doesn't appear to need to refer to either of these. — Peter Lawrey
Hi, I have a very similar problem could you elaborate further on your solution. I have C# file that parses a file containing many proto messages and it uses the same code you pasted ProtoBuf.Serializer.SerializeWithLengthPrefix(Of Msg)(postStream, msg, ProtoBuf.PrefixStyle.Base128) — Ersen Osman

Peter Lawrey Peter Lawrey · Accepted Answer · 2013-06-02T21:17:18

The problem is that you need to read directly from the original stream, not a block at a time. (Even if you know every message is exactly 4096 bytes, you cannot be sure to read that much at once) I suggest you use

while(stream still open) {
    Msg msg = Msg.parseDelimitedFrom(ins);
}

Note: TCP is a streaming protocol, not a message protocol. You are only guaranteed to read one byte at a time, and any extra bytes you get is a bonus.

Protobuf c# to java deserializing multiple messages

1 Answers