boost asio cross platform newline confusion

Question

I am using boost asio to write simple server/client to transmit binary data. Particularly, I am using async_write and async_read with ip:tcp::socket. Nothing fancy, really.

boost::asio::async_write(*mp_socket, boost::asio::buffer(data_ptr, data_size), 
                         boost::bind(&TcpSocket::m_handleWrite, this,
                                     boost::asio::placeholders::error,
                                     boost::asio::placeholders::bytes_transferred));

boost::asio::async_read(*mp_socket, boost::asio::buffer(mp_buffer, m_buffer_size),
                         boost::bind(&TcpSocket::m_handleRead, this,
                                     boost::asio::placeholders::error,
                                     boost::asio::placeholders::bytes_transferred));

I randomly generated binary data and send/receive in various platform pairs to test. The code works fine in same platform (such as all in linux, all in windows), but fails to work cross platform.

For example, the left is received data in windows, and the right is sent data in linux. The only difference is 0D0A (CRLF) in windows and 0A (LF) in linux. I think somewhere (in boost asio, or in winsock, or etc..), LF->CRLF conversion is happening.

So, is there a way to disable the conversion, as I am sending binary data? I was looking for some option in boost::asio configuration (such as using raw buffer, not stream buffer), but could not find it. Thank you for the help.

enter image description here

data in data_ptr

  size_t sz = rand() % 60000;
  char* p = (char*)malloc(sz + 4);
  uint32_t* p_header = reinterpret_cast<uint32_t*>(p);
  *p_header = htonl((uint32_t)sz);
  for (size_t i = 0; i < sz; ++i)
  {
    p[i + 4] = rand() % 255;
  }

Generally the buffer you provide is not modified by the async_write, or async_read calls (the overloads that take a ConstBufferSequence should guarantee this). The difference is likely happening where the buffers themselves are generated. Can you show that code? — Chad
Hello Chad, Can you clarify the question a little bit more? which 'buffer' did you mean? boost::asio::buffer, or char* mp_buffer, or the data buffer whose values are randomly populated? — xosp7tom

kenba kenba · Accepted Answer · 2014-04-10T08:32:24

I think that the issue is because async_read is "stream-oriented", see: Short reads and writes.

I assume that asio is using CRLF or LF as the stream terminators in Windows and Linux respectively and I guess that somewhere it's using a Windows terminator for a Linux stream or vice-versa.

Try changing your async_read call to:

mp_socket->async_read_some(boost::asio::buffer(mp_buffer, m_buffer_size),
                           boost::bind(&TcpSocket::m_handleRead, this,
                                       boost::asio::placeholders::error,
                                       boost::asio::placeholders::bytes_transferred));

I've used this method to read large binary files and verified the received files against their hash without any of the problems that you've encountered.

boost asio cross platform newline confusion

1 Answers