TCP Retransmission continues even after reset RST flag came up

Question

From my understanding about TCP reset(RST) flag is set whenever a segment received is not intended for the current connection and this results in aborting the current TCP session. But the wireshark capture pasted below doesn't seems to follow this theory. Basically end A which has initiated RESET ( frame#466 ) itself tries to Re-transmit the TCP frames over the same TCP session and also kept on responding any subsequent new connection request [SYN] from end B with [ RST,ACK ] and this behaviour repeats itself for 5 times and 3 way handshake again succeeds only during the 6th attempt (frame#486).

464 04:35.1 50000 > enpc [PSH, ACK] Seq=31894 Ack=7454 Win=5345 Len=105             
465 04:35.2 enpc > 50000 [ACK] Seq=7454 Ack=31999 Win=32127 Len=0               
466 04:35.2 50000 > enpc [RST] Seq=31999 Win=0 Len=0                
467 04:35.4 [TCP Retransmission] 50000 > enpc [PSH, ACK] Seq=31894 Ack=7454 Win=5345 Len=105                
468 04:36.1 [TCP Retransmission] 50000 > enpc [PSH, ACK] Seq=31894 Ack=7454 Win=5345 Len=105                
469 04:37.5 [TCP Retransmission] 50000 > enpc [PSH, ACK] Seq=31894 Ack=7454 Win=5345 Len=105                
470 04:40.3 [TCP Retransmission] 50000 > enpc [PSH, ACK] Seq=31894 Ack=7454 Win=5345 Len=105                
471 04:45.9 [TCP Retransmission] 50000 > enpc [PSH, ACK] Seq=31894 Ack=7454 Win=5345 Len=105                
472 04:57.1 [TCP Retransmission] 50000 > enpc [PSH, ACK] Seq=31894 Ack=7454 Win=5345 Len=105                
473 05:17.1 fg-fps > 50000 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=2 SACK_PERM=1                
474 05:17.1 50000 > fg-fps [RST, ACK] Seq=1 Ack=1 Win=0 Len=0               
475 05:17.5 fg-fps > 50000 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=2 SACK_PERM=1                
476 05:17.5 50000 > fg-fps [RST, ACK] Seq=1 Ack=1 Win=0 Len=0               
477 05:18.0 fg-fps > 50000 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=2 SACK_PERM=1                
478 05:18.0 50000 > fg-fps [RST, ACK] Seq=1 Ack=1 Win=0 Len=0               
479 05:19.5 [TCP Retransmission] 50000 > enpc [PSH, ACK] Seq=31894 Ack=7454 Win=5345 Len=105                
480 05:23.2 fg-gip > 50000 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=2 SACK_PERM=1                
481 05:23.2 50000 > fg-gip [RST, ACK] Seq=1 Ack=1 Win=0 Len=0               
482 05:23.7 fg-gip > 50000 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=2 SACK_PERM=1                
483 05:23.7 50000 > fg-gip [RST, ACK] Seq=1 Ack=1 Win=0 Len=0               
484 05:24.2 fg-gip > 50000 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=2 SACK_PERM=1                
485 05:24.2 50000 > fg-gip [RST, ACK] Seq=1 Ack=1 Win=0 Len=0               
486 05:29.7 dyniplookup > 50000 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=2 SACK_PERM=1               
487 05:29.7 50000 > dyniplookup [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1 WS=2             
488 05:29.7 dyniplookup > 50000 [ACK] Seq=1 Ack=1 Win=65536 Len=0               
489 05:29.7 dyniplookup > 50000 [PSH, ACK] Seq=1 Ack=1 Win=65536 Len=105                
490 05:29.7 50000 > dyniplookup [ACK] Seq=1 Ack=106 Win=5840 Len=0              
491 05:29.7 dyniplookup > 50000 [PSH, ACK] Seq=106 Ack=1 Win=65536 Len=105

My Question:

Why end A kept on re-transmitting the packets over a connection where RST was generated from it's own end?

Perhaps the initiator who sent initially ACK didn't expect ACK for another packet (different sequence number), so it sends RST and then sends again the package in growing time intervals until the receiver sends back the right message (which it does not, so the initiator sends RST again and again). Looks like a problem in the channel, perhaps a failing cable or difference in interface modes (half-duplex vs. full-duplex, etc) — Alfabravo
@Alfabravo : Why would it try to re-transmit the packet when connection moved to CLOSED state once RST is sent ? — amit
RST does not mean closed. RST means "let's try this again". For a closing connection, the sequence is FIN->, <-ACK, <-FIN, ACK->. And usually, RST means "let's wait a random (increasing) time for each new try so we avoid collisions and stuff". — Alfabravo
@Alfabravo: Once RST is sent connection can either remain in "same state" or will move to "CLOSED" state and atleast it moved to "CLOSED" state on B's end because it tries to re-initiate the 3 way handshake again with A. — amit
Yes, I was not accurate. stackoverflow.com/questions/7735618/tcp-rst-packet-details — Alfabravo

Mark Hillick Mark Hillick · Accepted Answer · 2012-05-10T13:59:49

There are many reasons why a RST might be sent. The reset flag is used when a TCP segment arrives that is not intended for a current open connection or listening port. For example, if the TCP port is closed, the TCP stack on the system will respond with a RST.

Typically when a system sends a TCP reset, it will have the ack flag also set as it's acknowledging a connection attempt. In your case, there's no ack flag, which (from memory) as per the RFC is only done when there's no established connection, which there is in your case, which would suggest to me that it's a firewall or seem intermediary device (not part of the TCP connection) that sends the reset. Therefore, the server A will legitimately then still think the connection is alive.

Frames 467-472 are standard TCP retransmission behaviour (from system A) with the time between connection attempts doubling with the server finally appearing to give up after the last attempt in frame 472. These frames are retransmissions of frame 464, which would appear to indicate that frame 465 was not received by system B. I'm guessing you took the capture on system B?

I believe system A only moved to CLOSED after frame 473 was sent, which will explain why we then see the three-way handshake initiated from system B.

From your trace, it's hard to say much more without more knowledge of the network.

HTH!

TCP Retransmission continues even after reset RST flag came up

1 Answers