Consuming a HTTP stream without reading one byte at a time

Question

I have been trying to read data from the Twitter stream API using C#, and since sometimes the API will return no data, and I am looking for a near-realtime response, I have been hesitant to use a buffer length of more than 1 byte on the reader in case the stream doesn't return any more data for the next day or two.

I have been using the following line:

input.BeginRead(buffer, 0, buffer.Length, InputReadComplete, null); 
//buffer = new byte[1]

Now that I plan to scale the application up, I think a size of 1 will result in a lot of CPU usage, and want to increase that number, but I still don't want the stream to just block. Is it possible to get the stream to return if no more bytes are read in the next 5 seconds or something similar?

Eric J. Eric J. · Accepted Answer · 2012-12-12T00:09:53

Async Option

You can use a timer in the async callback method to complete the operation if no bytes are received for e.g. 5 seconds. Reset the timer every time bytes are received. Start it before BeginRead.

Sync Option

Alternatively, you can use the ReceiveTimeout property of the underlying socket to establish a maximum time to wait before completing the read. You can use a larger buffer and set the timeout to e.g. 5 seconds.

From the MSDN documentation that property only applies to a synchronous read. You could perform a synchronous read on a separate thread.

UPDATE

Here's rough, untested code pieced together from a similar problem. It will probably not run (or be bug-free) as-is, but should give you the idea:

private EventWaitHandle asyncWait = new ManualResetEvent(false);
private Timer abortTimer = null;
private bool success = false;

public void ReadFromTwitter()
{
    abortTimer = new Timer(AbortTwitter, null, 50000, System.Threading.Timeout.Infinite);

    asyncWait.Reset();
    input.BeginRead(buffer, 0, buffer.Length, InputReadComplete, null);
    asyncWait.WaitOne();            
}

void AbortTwitter(object state)
{
    success = false; // Redundant but explicit for clarity
    asyncWait.Set();
}

void InputReadComplete()
{
    // Disable the timer:
    abortTimer.Change(System.Threading.Timeout.Infinite, System.Threading.Timeout.Infinite);
    success = true;
    asyncWait.Set();
}

Consuming a HTTP stream without reading one byte at a time

1 Answers