3
votes

Suppose I have simple nio based java server. For example (simplified code):

while (!self.isInterrupted()) {
  if (selector.select() <= 0) {
    continue;
  }

  Iterator<SelectionKey> iterator = selector.selectedKeys().iterator();
  while (iterator.hasNext()) {
    SelectionKey key = iterator.next();
    iterator.remove();
    SelectableChannel channel = key.channel();

    if (key.isValid() && key.isAcceptable()) {
      SocketChannel client = ((ServerSocketChannel) channel).accept();
      if (client != null) {
        client.configureBlocking(false);
        client.register(selector, SelectionKey.OP_READ);
      }
    } else if (key.isValid() && key.isReadable()) {
      channel.read(buffer);
      channel.close();
    }
  }
}

So, this is simple single threaded non blocking server.

Problem reside in following code.

channel.read(buffer);
channel.close();

When I closing channel in same thread (thread that accept connection and reading data) all works fine. But I got a problem when connection closed in another thread. For example

((SocketChannel) channel).read(buffer);
executor.execute(new Runnable() {
   public void run() {
     channel.close();
   }
});

In this scenario I ended up with socket in state TIME_WAIT on server and ESTABLISHED on client. So connection is not closing gracefully. Any ideas what's wrong? What I missed?

5
What the host OS? This may be an implementation feature/bug as the package specs assert that these are all thread-safe constructs.alphazero
Host machine under Mac OS X 10.5.6. Today I'm repeat tests under windows 7 and it's work just fine. Seems like you are right.Denis Bazhenov
What happens if you close channel.socket() before closing the channel?Nuoji
For me it's all the same (at least on windows). If I call channel.socket().close(), channel becomes closed (with gracefull client disconnect).Denis Bazhenov
Behavior is identical in 1.4, 1.5 and 1.6, at least on my Mac, so I suggest following my advice below and do it on the same thread. That strategy also has the virtue of simplifying other features you're likely to want to add to your server.Nuoji

5 Answers

1
votes

TIME_WAIT means the OS has received a request to close the socket, but waits for possible late communications from the client side. Client apparently didn't get the RST, since it's still thinks it's ESTABLISHED. It's not Java stuff, it's OS. RST is apparently delayed by OS -- for whatever reason.

Why it only happens when you close it in another thread -- who knows? May be OS believes that closes in another thread should wait for original thread exit, or something. As I said, it's OS internal mechanics.

0
votes

I don't see why it would make a difference unless the close is throwing an exception. If it were you wouldn't see the exception. I suggest putting the close in a catch(Throwable t) and print out the exception (assuming there is one)

0
votes

You know, after a bit more careful testing I cannot reproduce you results on my Mac.

While it is true that the connection remains in TIME_WAIT for something around 1 minute after close on the server side, it closes immediately on the client side (when I connect to it using a telnet client to test).

This is the same regardless of on what thread I close the channel. What machine are you running on and what version of java?

0
votes

It may have something to do with the problem mentioned here. If it really is the behaviour of the BSD / OS X poll() method I do think you're out of luck.

I think I would mark this code as non-portable due to - as I understand it - a bug in BSD / OS X.

0
votes

You have a major problem in your example.

With Java NIO, the thread doing the accept() must only be doing the accept(). Toy examples aside you are probably using Java NIO because of anticipated high number of connections. If you even think about doing the read in the same thread as the selects, the pending unaccepted selects will time out waiting for the connection to be established. By the time this one overwrought thread gets around to accepting the connection, the OS's on either side will have given up and the accept() will fail.

Only do the absolute minimum in the selection thread. Any more and you will just being rewriting the code until you do only the minimum.

[In response to comment]

Only in toy examples should the reading be handled on the main thread.

Try to handle:

  • 300+ simultaneous connection attempts.
  • Each connection once established sends 24K bytes to a single server - i.e. a small web page, a tiny .jpg.
  • Slow down each connection slightly ( the connection is being established over a dialup, or the network is having a high-error/retry rate) - so the TCP/IP ACK takes longer than ideal (out of your control OS level thing)
  • Have some of your test connections, send a single bytes every 1 milliseconds. (this simulates a client that is having its own high load condition, so is generating the data at a very slow rate.) The thread has to spend almost the same amount of effort processing a single bytes as it does 24K bytes.
  • Have some connections be cut with no warning ( connection lost issues ).

As a practical matter, the connection needs to be established within 500ms -1500ms before the attempting machine drops the connection.

As a result of all these issues, a single thread will not be able to get all the connections set up fast enough before the machine on the other end gives up the connection attempt. The reads must be in a different thread. period.

[Key Point] I forgot to really be clear about this. But the threads doing the reading will have their own Selector. The Selector used to establish the connection should not be used to listen for new data.

Addition (in response to Gnarly's contention that no I/O actually occurs during the java call to read the stream.

Each layer has a defined buffer size. Once that buffer is full, the IO is halted. For example, TCP/IP buffers have between 8K-64K buffers per connection. Once the TCP/IP buffer fills, the receiving computer tells the sending computer to stop. If receiving computer does not process the buffered bytes fast enough the sending computer will drop the connection.

If the receiving computer is processing the buffered bytes, the sender will continue to stream the bytes, while the java io read call is being made.

Furthermore, realize that the first byte to arrive triggers the "bytes available to be read" on the selector. There is no guarantee as to how many have arrived.

The buffer sizes defined in the java code have no relationship to the buffer size of the OS.