How to update epoll events after epoll_wait?

Question

I have the following code excerpt (heavily redacted to remove unimportant details) which fails under a rare and particular set of circumstances.

struct epoll_event *events = calloc(MAXEVENTS+1, sizeof(struct epoll_event));
struct sockaddr_in in_addr;
socklen_t in_len = sizeof in_addr;

while(1)
  {
  int n = epoll_wait(efd,events, MAXEVENTS, -1);
  for(int i=0; i<n; i++)
    {
    struct epoll_event *evI = &events[i];
    uint64 u64 = evI->data.u64;
    int type = u64>>32, fd=u64, fdx = fd;

    if(type == -1)
      {
      while((fd = accept4(fdx, &in_addr, &in_len, SOCK_NONBLOCK|SOCK_CLOEXEC))>-1)
        {
        setNew_Connection(efd, fd);
        storeAddrPort(fd, &in_addr, &in_len);
        }
      }
    else
      {
      if(evI->events&(EPOLLERR|EPOLLHUP|EPOLLRDHUP)){closeConnection(fd);}
      if(evI->events&EPOLLOUT)  //process out data stuff
      else if(evI->events&EPOLLIN) //process in data stuff and possibly close a different connection.
      }
    }
  }

Listening sockets are differentiated by -1 in the upper part of evI->data.u64

setNew_Connection does the usual accepting stuff like adding the new socket to epoll etc

EPOLLET is used.

Now it all works brilliantly except under the following circumstances it fails because events is only updated in the epoll_wait so a connection closure does not affect the n events until after returning to the top of the while(1) loop.

epoll_wait unblocks with 3 events queued in the events struct table.
First event (n=0), is incoming data after which code decides to close a connection (e.g. file descriptor 8) as it is no longer needed.
2nd event (n=1) is an incoming new connection. accept4 assigns fd:8 as it has recently become available. setNew_Connection adds it to the epoll list.
3rd event is incoming data for the connection closed in step 2. i.e. fd:8 but it is no longer valid as the original fd:8 connection was closed and the current fd:8 is for a different connection.

I hope I have explained the problem adequately. The issue is that queued events in the events table are not updated when a connection is closed until the code returns to epoll_wait. How can I code around this problem?

you can mark the connection to be closed, and do a closing loop after the events handler. — Ôrel

poby poby · Accepted Answer · 2015-10-28T20:30:39

Orel gave me the answer but I thought I would post the complete code solution. Instead of

close(fd)

I use

shutdown(fd,SHUT_RDWR);
FDS[FDSL++] = fd;

The shutdown prevents anymore data being read or written but doesn't actually close the socket. FDS[FDSL++] = fd; stores the fd so that later after the n events are done, it can be closed with while(FDSL)close(FDS[--FDSL];

int FDS[MAXEVENTS],FDSL=0;

struct epoll_event *events = calloc(MAXEVENTS+1, sizeof(struct epoll_event));
struct sockaddr_in in_addr;
socklen_t in_len = sizeof in_addr;

while(1)
  {
  int n = epoll_wait(efd,events, MAXEVENTS, -1);
  for(int i=0; i<n; i++)
    {
    struct epoll_event *evI = &events[i];
    uint64 u64 = evI->data.u64;
    int type = u64>>32, fd=u64, fdx = fd;

    if(type == -1)
      {
      while((fd = accept4(fdx, &in_addr, &in_len, SOCK_NONBLOCK|SOCK_CLOEXEC))>-1)
        {
        setNew_Connection(efd, fd);
        storeAddrPort(fd, &in_addr, &in_len);
        }
      }
    else
      {
      if(evI->events&(EPOLLERR|EPOLLHUP|EPOLLRDHUP)){closeConnection(fd);}
      if(evI->events&EPOLLOUT)  //process out data stuff
      else if(evI->events&EPOLLIN) //process in data stuff and possibly close a different connection.
      }
    }

  while(FDSL)close(FDS[--FDSL];
  }

How to update epoll events after epoll_wait?

1 Answers