Why RST was Sent When No Close() or Shutdown() Called?

2
votes

We are using LoadRunner to do press test on a tcp-based multi-threaded server application. Recently we encounter an annoying issue:

The clients get error reporting -- "10054 - connection reset by peer" after some cycles. By capturing packets from client, we found clients receive RST from server.

What strange is we don't call any close() or shutdown() from server side (for test purpose), the client still gets the RST packet and 10054 error?!!

Below are the packets I caught:

  1. 5395 77.335317000 192.168.11.232 192.168.11.30 TCP 62 27124→10000 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 SACK_PERM=1
  2. 5656 77.356679000 192.168.11.30 192.168.11.232 TCP 60 10000→27124 [SYN, ACK] Seq=0 Ack=1 Win=14600 Len=0 MSS=1460
  3. 5657 77.356684000 192.168.11.232 192.168.11.30 TCP 54 27124→10000 [ACK] Seq=1 Ack=1 Win=65535 Len=0
  4. 5776 77.359276000 192.168.11.232 192.168.11.30 TCP 1514 [TCP segment of a reassembled PDU]
  5. 5778 77.359286000 192.168.11.232 192.168.11.30 HTTP/XML 332 POST /AddressBookController.php HTTP/1.1
  6. 5901 77.368227000 192.168.11.30 192.168.11.232 TCP 60 10000→27124 [RST] Seq=1 Win=0 Len=0

Here are my questions, any help would be greatly appreciated!

Q1. Why would the server send RST to the client when it doesn't even call any close() or shutdown() to the socket from server side?

Q2. [DUPLICATE Q1, Please Skip]In what condition will a RST be sent by the server when both the client and server haven't called any close() or shutdown?

Q3. Is it possible for me to recompile the kernel and print some log in the TCP stack? I want to know what happened when it sends a RST packet.

[UPDATE]
Increasing the listen(BACKLOG) FROM BackLog=100 to BackLog=1000, will dramatically decreases the errors from around 50/hour to 1~2/10hours. Why?

[UPDATE2]
1. This is a gSOAP server which accepts soap request and returns around 1~2 Kilobytes of data to clients.
2. We checked the packets from client's side, there is no FIN flag from server which proves server never calls any close() or shutdown().
3. We were using LoadRunner doing press test with 400 USERs make parallel soap requests, the frequency is 80 thousand requests per hour average.
4. Contents of Packet 4 are nothing abnomal, below are part of it

POST /AddressBookController.php HTTP/1.1
Host: 192.168.11.30:10000
User-Agent: gSOAP/2.7
Content-Type: text/xml; charset=utf-8
Content-Length: 1470
Connection: close
Cookie: sessionId=a23b15dcc14bb1d8f9f6bf3c4ed728d90c4c3939143866821152918211
SOAPAction: ""

<?xml version="1.0" encoding="UTF-8"?>
......
2
linuxmultithreadingsocketstcp
What was in packet 4? – user207421

2 Answers

0
votes

Q1. Why would the server send RST to the client while it doesn't even call any close() or shutdown() to the socket?

Because the server had decided to close the connection but the client kept sending. Or, the server closed its socket while there was still unread data.

Q2. In what condition will a RST be sent by the server when both the client and server haven't called any close() or shutdown?

That's just the same question all over again.

Q3. Is it possible for me to recompile the kernel and print some log in the TCP stack? I want to know what happened when it sends a RST packet.

No, you want to know what happened before it sent the RST packet.

The problem seems to be with your client. It should certainly close the connection at some point.

EDIT In this case it appears you have sent something odd (non-HTTP), in packet #4, which has caused the server to close the connection. What is it?

0
votes

Refering your "packet 4" from (the formerly relevant) RFC 2616:

HTTP/1.1 defines the "close" connection option for the sender to signal that the connection will be closed after completion of the response. For example,

Connection: close

So you seem to get what you requested.