Exceptions when reading protobuf messages in Java

Question

I am using protobuf now for some weeks, but I still keep getting exceptions when parsing protobuf messages in Java.

I use C++ to create my protobuf messages and send them with boost sockets to a server socket where the Java client ist listening. The C++ code for transmitting the message is this:

boost::asio::streambuf b;
std::ostream os(&b);

ZeroCopyOutputStream *raw_output = new OstreamOutputStream(&os);
CodedOutputStream *coded_output = new CodedOutputStream(raw_output);

coded_output->WriteVarint32(agentMessage.ByteSize());
agentMessage.SerializeToCodedStream(coded_output);

delete coded_output;
delete raw_output;

boost::system::error_code ignored_error;

boost::asio::async_write(socket, b.data(), boost::bind(
        &MessageService::handle_write, this,
        boost::asio::placeholders::error));

As you can see I write with WriteVarint32 the length of the message, thus the Java side should know by using parseDelimitedFrom how far it should read:

AgentMessage agentMessage = AgentMessageProtos.AgentMessage    
                                .parseDelimitedFrom(socket.getInputStream());

But it's no help, I keep getting these kind of Exceptions:

Protocol message contained an invalid tag (zero).
Message missing required fields: ...
Protocol message tag had invalid wire type.
Protocol message end-group tag did not match expected tag.
While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either than the input has been truncated or that an embedded message misreported its own length.

It is important to know, that these exceptions are not thrown on every message. This is only a fraction of the messages I receive the most work out just fine - still I would like to fix this since I do not want to omit the messages.

I would be really gratful if someone could help me out or spent his ideas.

Another interesting fact is the number of messages I receive. A total messages of 1.000 in 2 seconds is normally for my program. In 20 seconds about 100.000 and so on. I reduced the messages sent artificially and when only 6-8 messages are transmitted, there are no errors at all. So might this be a buffering problem on the Java client socket side?

On, let's say 60.000 messages, 5 of them are corrupted on average.

Maybe a silly question but is there any way you've left padding / oversized-buffers in the data, rather than trimming any surplus? — Marc Gravell♦
(this error could definitely be easily caused by spare zeros) — Marc Gravell♦
@Marc-Gravell: What would be an oversized buffers? Actually I don't get what you think might cause this. Maybe you could point out where I should look for this? Btw. I've added some couple of other exceptions I receive as well. — Konrad Reiche
Something else I've seen people do way too often - is use an Encoding to try to get the BLOB as a string, then decode. Guaranteed corruption there (should use base-64 or hex etc if you want a string, not a UTF/codepage/etc) — Marc Gravell♦
"Java client socket side" - well, I would more readily expect simply a bug in the stream processing code, especially if async/threaded. Could be either read or write, in that case... reducing the rate will obviously remove a lot of accidental collisions caused by incorrect async. — Marc Gravell♦

Jonny Dee Jonny Dee · Accepted Answer · 2011-07-02T22:49:17

I am not familiar with the Java API, but I wonder how Java deals with an uint32 value denoting the message length, because Java only has signed 32-bit integers. A quick look at the Java API reference told me an unsigned 32-bit value is stored within a signed 32-bit variable. So how is the case handled where an unsigned 32-bit value denotes the message length? Also, there seems to be support for varint signed integers in the Java implementation. They are called ZigZag32/64. AFAIK, the C++ version doesn't know about such encodings. So maybe the cause for your problem might be related with these things?

Exceptions when reading protobuf messages in Java

2 Answers