protobuf-net stream objects from disk

Question

Consider I have a very large collection (millions) of objects serialized according to the proto wire format. Is it possible to stream these items from the file? I tried serializing the objects as a List<T> and then deserializing a single T item, but it ended up only reading the very last item from the stream. I also tried serializing each instance individually to the stream with the same effect of upon deserialization, it only reading the last item.

I suspect the solution requires my to know the size of each serialized item and then reading that size from the stream and passing that span of bytes to the protobuf serializer for deserialization. I wanted to make sure there wasn't an easier mechanism, one that doesn't require knowledge of the length of each individual item which may be different for each instance of the object, to accomplish this task.

Another thought I had was including the size of each upcoming object as it's own object in the stream, for example:

0: meta-information for the first object, including type/length in bytes 1: object defined in 0 2: meta-information for the second object, including type/length in bytes 3: object defined in 2 4: ...etc

Version information: I'm currently using dotnet core 3.1 and protobuf-net version 2.4.4

Answer 1

In protobuf the root object is not terminated by default, with the intent being to allow "merge" === "append". This conflicts with he very common scenario you are describing. Fortunately, many libraries provide a mechanism to encode the length before the object for this reason. What you are looking for is the SerializeWithLengthPrefix and DeserializeWithLengthPrefix methods.

If the data already exists as flat appends, and cannot be rewritten: there are still ways to recover it, by using the reader API. A bit more complex, but I've recovered such data in the past for people when needed.