write to TCP socket keeps returning EAGAIN

Question

I'm seeing a situation where I have a TCP socket (non-blocking) that I'm writing to, but under load it gets into a situation where it keeps returning EAGAIN. I know this because it stops processing, and I can attach a debugger and step through it. Stepping through, the write call returns an error and errno is set to EAGAIN every time (it busy-waits on EAGAIN...ignore that this is a bad idea :)

My understanding was that EAGAIN should only be returned on a write if the buffer is full, but I don't understand what would prevent it from draining and the write call eventually succeeding.

This is Ubuntu, Linux kernel 3.19.0-47-generic.

Ideas?

Answer 1

What is preventing it from draining is that the peer isn't reading as fast as you're writing. The peer's receive buffer fills, your send buffer fills, you can't write.

it busy-waits on EAGAIN...ignore that this is a bad idea

I can't ignore that. It's a bad idea. You're supposed to use select() in this specific circumstance to tell you when the socket becomes writable, not just loop mindlessly.

Answer 2

EAGAIN OR EWOULDBLOCK gets returned when the socket is in non-blocking mode.

You're probably initializing your socket somewhere with a call to fnctl and the O_NONBLOCK flag (or similar effect). Or you are using the MSG_DONTWAIT flag with the send call.

In any case, if you don't want to convert the socket back to blocking mode, just invoking the send in a loop should suffice. Regardless of how the socket was initialized (blocking or non-blocking), I always recommend using a loop for the scenario where send indicates only part of the buffer was handled.

int remaining = <bytes to send>;
int sent = 0;
const char* buffer = <data buffer to send>;
int success = 0;

while (remaining > 0  &&  !DoesTheAppNeedToExit())
{
     int result = send(s, buffer+sent, remaining, 0);
     if (result != -1)
     {
         // sent partial or all the remaining data
         sent += result;
         remaining -= result;
         if (remaining <= 0)
         {
             success = 1;
             break;
         }
     }
     else
     {
       int err = errno;
       if ((err == EAGAIN) || (err == EWOULDBLOCK))
       {
           timeval tv = {};
           fd_set fds = {};
           int selectresult;

           tv.tv_sec = 1; // or how ever long you want to wait
           FD_ZERO(&fds);
           FD_SET(0, &fds);
           selectresult = select(s+1, NULL, &fds, NULL, &tv);
           // recommended: check return value from select and check for fatal error. Or timeout handling.

           // socket is ready if IS_FDSET(s, &fds), but it doesn't hurt to just try the send again at this point
       }
       else
       {
           // unrecoverable error!
           close(s);
           s = -1;
           break;
       }
     }
}