Asynchronous event loop design and issues

5
votes

I'm designing event loop for asynchronous socket IO using epoll/devpoll/kqueue/poll/select (including windows-select).

I have two options of performing, IO operation:

Non-blocking mode, poll on EAGAIN

  1. Set socket to non-blocking mode.
  2. Read/Write to socket.
  3. If operation succeeds, post completion notification to event loop.
  4. If I get EAGAIN, add socket to "select list" and poll socket.

Polling mode: poll and then execute

  1. Add socket to select list and poll it.
  2. Wait for notification that it is readable writable
  3. read/write
  4. Post completion notification to event loop of sucseeds

To me it looks like first would require less system calls when using in normal mode, especially for writing to socket (buffers are quite big). Also it looks like that it would be possible to reduce the overhead over number of "select" executions, especially it is nice when you do not have something that scales well as epoll/devpoll/kqueue.

Questions:

  • Are there any advantages of the second approach?
  • Are there any portability issues with non-blocking operations on sockets/file descriptors over numerous operating systems: Linux, FreeBSD, Solaris, MacOSX, Windows.

Notes: Please do not suggest using existing event-loop/socket-api implementations

3
c++casynchronousnonblockingevent-loop

3 Answers

3
votes

I'm not sure there's any cross-platform problem; at the most you would have to use Windows Sockets API, but with the same results.

Otherwise, you seem to be polling in either case (avoiding blocking waits), so both approaches are fine. As long as you don't put yourself in a position to block (ex. read when there's no data, write when buffer's full), it makes no difference at all.

Maybe the first approach is easier to code/understand; so, go with that.

It might be of interest to you to check out the documentation of libev and the c10k problem for interesting ideas/approaches on this topic.

2
votes

The first design is the Proactor Pattern, the second is the Reactor Pattern

One advantage of the reactor pattern is that you can design your API such that you don't have to allocate read buffers until the data is actually there to be read. This reduces memory usage while you're waiting for I/O.

1
votes

from my experience with low latency socket apps:

for writes - try to write directly into the socket from writing thread (you need to obtain event loop mutex for that), if write is incomplete subscribe to write readiness with event loop (select/waitformultipleobjects) and write from event loop thread when socket gets writable

for reads - be always "subscribed" for read readiness for all sockets, so you always read from within event loop thread when the socket gets readable