Protobuf Net and storing/retrieving data blobs to/from binary file

Question

I am trying to figure out whether I can use protobuf-net to store and retrieve the following serialized data structure:

I have about 200,000 objects of size 16 bytes (the object contains one long and two floating type values, 8 byte plus 2 * 4 bytes) each day that I like to store in a binary file. Retrieval of such objects will only be by full days, for example, I would want to request objects between April 1 2012 and April 6 2012 which are supposed to be read starting April 1 , then April 2,...April 6. One requirement is that access needs to be random, meaning, The file may contain data from 2010 to June 2012 but I may only want to retrieve elements between April 1 and April 6 2012 without having to read all elements from the start.

I currently store the data as contiguous byte arrays in the order of DateTimeTick but without regards to when a new day starts or ends. If I could use protbuf-net to stream the data as full-day "blobs" as IEnumerable that would be terrific. Is that possible? I was thinking to store an IEnumerable or List for each day serialized with protobuf-net but am not sure how to randomly access a particular list later on? Any ideas or suggestions? Thanks

Answer 1

That isn't a typical use-case for protobuf-net, and while I suspect it would be possible to use it that way, it wouldn't be my instinct to try to do so. If the key requirement is to split by day, then using multiple files would be an obvious choice. Alternatively, tweak your existing file format to include (per day) the date stamp and the size of the data for that day - then you can just skip forward in entire days using the FileStream's .Position property.

protobuf-net does have streaming and skipping APIs, or a raw "reader" API if you don't want to involve the core serializer, but: I'm not sure this is going to help you massively.

Frankly, since (in your current process) each chunk is fixed size, you could also use binary search (maybe using linear interpolation for the staring position) to just seek to the right time.