Netty client generates lots of TIME_WAIT socket states

Question

I have written a netty client code to send some processed data to multiple clients. After running for 3-4 hours I exhaust all sockets and no more connections possible. Also when I check the socket states in the OS a large number of sockets are in TIME_WAIT state.

public class NettyClient {

private static LogHelper logger = new LogHelper(NettyClient.class);

private static EventLoopGroup workerGroup = new NioEventLoopGroup();

private static Bootstrap nettyClient = new Bootstrap()
        .group(workerGroup)
        .channel(NioSocketChannel.class)
        .option(ChannelOption.SO_KEEPALIVE, true)
        .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 5000);

private URL url;
private RequestVo Req;
private ChannelFuture chFuture;
private Object ReportData;
private JAXBContext jbContext;
private static final int CHANNEL_READ_TIMEOUT = 5;


public NettyClient() {
    // TODO Auto-generated constructor stub
}

public NettyClient(RequestVo Req, JAXBContext jbCtx,Object data) {
    this.Req = Req;
    this.ReportData = data;
    this.jbContext = jbCtx;
}

public void sendRequest() {

    logger.debug("In sendRequest()");
    //ChannelFuture chFuture = null;
    try {
        this.url = new URL(Req.getPushAddress());
        //add handlers
        nettyClient.handler(new ChannelInitializer<SocketChannel>() {

            @Override
            public void initChannel(SocketChannel ch) {
                ch.pipeline()
                  .addLast("timeout",
                    new ReadTimeoutHandler(CHANNEL_READ_TIMEOUT, TimeUnit.SECONDS));

                ch.pipeline()
                  .addLast("codec", new HttpClientCodec());

                ch.pipeline()
                  .addLast("inbound",
                     new NettyClientInBoundHandler(Req, jbContext, ReportData));
            }
        });

        //make a connection to the Client
        int port = url.getPort() == -1? url.getDefaultPort():url.getPort();
        chFuture = nettyClient.connect(url.getHost(), port);
        chFuture.addListener(new NettyClientConnectionListener(this.Req.getRequestId()));
    } catch (Exception e) {
        logger.error("Exception: Failed to connect to Client ", e);
    } finally {

    }
}
}

Here are the methods from ChannelInBoundHandler Class

@Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception
{
    Map<String, String> props = new HashMap<String, String>();

    if(msg instanceof HttpResponse) {
        logger.debug("channelRead()");
        HttpResponse httpRes = (HttpResponse) msg;
        HttpResponseStatus httpStatus = httpRes.status();
        props.put(REQUEST_ID, this.Request.getRequestId());
        props.put(CLIENT_RESPONSE_CODE, String.valueOf(httpStatus.code()));
        JmsService.getInstance(DESTINATION).sendTextMessage(props, "");
        logger.debug("channelRead() HttpResponse Code: " + httpStatus.code());
        ctx.close();
    }
}

@Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception
{
    Map<String, String> props = new HashMap<String, String>();

    logger.error("exceptionCaught()", cause);
    if(cause instanceof ReadTimeoutException) {
        //If read-timeout, send back the response
        props.put(REQUEST_ID, this.Request.getRequestId());
        props.put(CLIENT_RESPONSE_CODE,
                  String.valueOf(HttpResponseStatus.REQUEST_TIMEOUT.code()));
        JmsService.getInstance(DESTINATION).sendTextMessage(props, "");
        ctx.close();
    }
    else {
        logger.error("Exception: ", cause);
    }
}

Any idea what is wrong in the code would greatly help me. Thanks

Yes, I do close the channel, after read timeout exception or when response is received. — Giridhar Kandachar
Might it be possible to reuse an open connection instead of opening a new one each time? — Kjartan
You're creating outbound connections faster than their local ports can be reclaimed. You should try to conserve outbound connections, by using HTTP keepalive where possible. — user207421

Kjartan Kjartan · Accepted Answer · 2019-04-24T06:45:26

I'm not familiar with netty, but I think I can explain part of your problem, and hopefully help you along the way:

When you make use of a port and then close it, the port will not automatically be available for use by other processes at once. Instead, it will go into the TIME_WAIT state for a certain period of time. For Windows, I believe this will be 240 seconds (four minutes).

I'd guess that your code is slowly using up all the available ports on your system, due to the release of ports from the TIME_WAIT state is happening too slowly.

It's not entirely clear to me where the actual port numbers are coming from (are they auto-generated by url.getDefaultPort() perhaps?), but perhaps you can find some way to reuse them? If you can keep one or more open connections and somehow reuse these, then you might be able to decrease the frequency of requests for new ports enough for the closed ports to go out of their TIME_WAIT state.

Netty client generates lots of TIME_WAIT socket states

1 Answers