Endianness safe copying binary data

Question

I'm trying to manipulate binary data with ArrayBuffer. If original ArrayBuffer looks like this:

var buffer = new ArrayBuffer(40);
var dv = new DataView(buffer);

var num = 0;
for(var i = 0; i < 40; i+=2) {
    dv.setUint16(i, num++);
}

Then ArrayBuffer has 20 length of Uint16Array. I want to prepend extra binary datas, not append. Let's say that I want to prepend 5 bytes of Uint8 like this:

ArrayBuffer[0] = 10;
ArrayBuffer[1] = 20;
ArrayBuffer[2] = 30;
ArrayBuffer[3] = 40;
ArrayBuffer[4] = 50;
ArrayBuffer[5 ~ 25] = Original Data

As I know, there is no prepending like method in ArrayBuffer, so I did it myself. This wasn't matter, but considering endianness was really made me crazy.

First 5 bytes of data was produced by me, so I could set the endianness manually, but the original data is coming from external, so data could be Little Endian and Big Endian both.

I'm making somekind of binary data module for Node.js and Web Browser both can be use, and basic usage is prepending extra binary data from Web Browser by using this module, and browser sent this to server, then server reads this binary data from this module too, and split to prepended and orignal.

But the problem is, if the browser, I mean client computer using little-endian, and server using big-endian, original data might not be read correctly, because they have different endianness.

I'm appending original data to new binary data like this. New binary data has different 5 bytes of data at 0 ~ 5 bytes. So I have to write after 5 bytes offset.

// write rest of data
var newBuffer = new ArrayBuffer(5 + origBuffer.byteLength);
var ndv = new DataView(newBuffer);
var dv = new DataView(origBuffer);    

for(var i = 0; i < origBuffer.byteLength; i++) {
    var offset = 5 + i;
    ndv.setUint8(offset, dv.getUint8(i));
}

Testing on my local machine is fine, because server and client both using same cpu, but in reallife, if they has different endianness, this module will not work correctly.

Is there a endianness safe way to copying binary data with ArrayBuffer & DataView? Or should I just forget about the endianness? Any advice will very appreciate it.

You haven't yet shown the code your server uses to read from the binary data it receives? — Bergi

Unknown Unknown · Accepted Answer · 2016-11-30T23:39:12

For bytes it doesn't matter as they don't have endianess, but if you get data wider than a byte (16/32-bits etc) you have to know the endianess in advance as there is no way to detect the original endianess (you could do some guessing depending on the data but...).

In most cases byte-streams are in so-called network order which basically is big-endian, and what DataView defaults to.

However, I would suggest to implement a test function before sending any data to client. For example, calling the server from client with a isLittleEndian() function which triggers the server to send back a 16-bit value like 0xff00. Then test in which order the bytes are received - something like:

function isLittleEndian(callback) {  // callback as request will be async
  // get 0xff00 from server as 2 byte ArrayBuffer here somehow...
  var test = new Uint8Array(arraybufferFromServer);
  callback({isLittleEndian: !test[0]}); // will be 0x00ff as little-endian
}

The result can then be used with a DataView using the little-endian flag for the various methods.

Note that this apply to the data stream and not necessarily the CPU architecture/endianess of the server (a little-endian server can still (and most likely) send data in big-endian/network order).

Endianness safe copying binary data

2 Answers