C++ Protocol Buffer, sending integer array

Question

I have an embedded C++ project where I'm reading a series of int32's from a hardware device, then packing them into an int array as part of large data structure, and then sending to a remote system over TCP/IP. So, I was using a simple data struct with a bunch of stuff defined and now I want to convert this to use Protocol Buffers. So, I was thinking of using a "repeated int32 data" as the element of my proto buff. But I want to avoid using a loop such as this:

int hardware_data[1000]; // An array that holds the data read from the hardware
for(int i=0; i< sizeof(hardware_data); i++ )
{
    proto.add_data( hardware_data[i] );
}

I'd much rather use an efficient method, such as making the proto buff just point to the existing hardware_data[] array (a zero copy method), or using memcpy from hardware_data into proto.data.

I understand how to setup the memcpy(), but how then does the proto buff know how many elements are in the proto.data "array"? Can I still use the proto.data_size() to get the number of elements? Is there an efficient way to move the data from my hardware read to the proto buff for sending? Is there a better way to do this?

Kerrik, I wasn't aware of the zero copy API. Here's my proto definition:

message hardware_data 
{
optional    Lob                     lob             = 1;
optional    int32                   taskSeqNum      = 2;
optional    int32                   secondsOfDay    = 3;
optional    float                   IQOutRateKhz    = 4;
optional    float                   IQBwKhz         = 5;
optional    int32                   tStart          = 6;
optional    int32                   tOffset         = 7;
optional    float                   collectionTime  = 8;
optional    int32                   numSamples      = 9;
optional    int32                   chunk           = 10;
optional    int32                   dimSize         = 11;
repeated    int32                   data            = 12 [packed=true];
}

I'm not sure how the zero copy would play into this proto buff definition.

sizeof operator returns the size of the data structure in bytes, so for 1000-item array of ints it is probably 4000 and your loop will index the array far past its actual end, thus causing undefined behavior. Use sizeof(hadware_data)/sizeof(hardware_data[0]) to calculate the number of items of the array. — CiaPan
Of course.... I only placed that as example code. Your comment, while correct, does nothing to answer the question of how to use the protocol buffer. — rbwilliams
@KerrekSB That's a low-level API that has nothing to do with what rbwilliams wants. — Kenton Varda
@KentonVarda: Ah, right, because the OP wants varint encoding. If the serialization speed is an issue, then perhaps the fixed-width integers are worth considering. — Kerrek SB

Kenton Varda Kenton Varda · Accepted Answer · 2015-01-08T05:19:52

On the wire, a packed repeated int32 is encoded as a series of varints. A varint is a variable-width encoding in which smaller values take less space. Of course, this isn't how the data is represented in your array, so embedding it into the message zero-copy isn't really possible.

In fact, though, you're currently doing two copies, and you can eliminate one of them. Instead of allocating int hardware_data[1000] directly, consider sticking the data directly into a google::protobuf::RepeatedField<int>. You can then make clever use of Swap() to move that data into a message without a copy:

RepeatedField<int> hardware_data;
hardware_data.Reserve(expected_size);
get_data_somehow(&hardware_data);

// later
proto.mutable_data()->Swap(&hardware_data);

After you've serialized the message, you may wish to additionally Swap() the field back, so that you can reuse the memory that was already reserved. (RepeatedField::Clear() will not free the underlying memory, just mark it for reuse.)

With all that said, serializing the message will still require copying the data as part of encoding it. Even if you changed the encoding to packed repeated fixed32 (which is actually encoded as 32-bit integers on the wire), there's no way to convince the library to use your memory directly.

C++ Protocol Buffer, sending integer array

1 Answers