rsyslogd client not closing the TCP connection when server rsyslogd goes down

Question

I have configured rsyslogd on a remote server to send the logs from a client machine with rsyslogd using TCP protocol. After configuring and restarting the rsyslogd daemon on both client and server I am able to send the packets to the server and all works fine. But later when I restart the rsyslogd on the server the client is still sending the packets to the old TCP connection. Hence the client retries for 16 times and fails to send the packet. After the retry for sending the next packet the client is creating a new connection and the communication works fine there onward.

When I restart the rsyslogd on server using the tcpdump i captured the packets and we can see that the server sends flag[F] to client and the client acknowledges it as well. But when we send the next packet it is not crating a new connection.

restart rsyslog on server: Server side tcpdump:

*09:54:50.012933 IP x.x.x.101.514 > y.y.y.167.37141: Flags [F.], seq 1, ack 31, win 229, length 0

09:54:50.013050 IP y.y.y.167.37141 > x.x.x.101.514: Flags [.], ack 2, win 115, length 0*

For the very next packet send from client the server sends flag[R] but client keeps retrying for 16 times:

tcpdump from server:

*03:55:11.811611 IP y.y.y.167.37141 > x.x.x.101.514: Flags [P.], seq 31:61, ack 2, win 115, length 30

03:55:11.811647 IP x.x.x.101.514 > y.y.y.167.37141: Flags [R], seq 1863584583, win 0, length 0

03:55:12.014158 IP y.y.y.167.37141 > x.x.x.101.514: Flags [P.], seq 31:61, ack 2, win 115, length 30

03:55:12.014189 IP x.x.x.101.514 > y.y.y.167.37141: Flags [R], seq 1863584583, win 0, length 0*

<this repeated 6 times on sever>

at same time on client we do not see the response reaching from server:

09:55:11.811077 IP y.y.y.167.37141 > x.x.x.101.514: Flags [P.], seq 31:61, ack 2, win 115, length 30

09:55:12.013639 IP y.y.y.167.37141 > x.x.x.101.514: Flags [P.], seq 31:61, ack 2, win 115, length 30

09:55:12.421627 IP y.y.y.167.37141 > x.x.x.101.514: Flags [P.], seq 31:61, ack 2, win 115, length 30

<this retied 16 times>

Now after the 16 retry (took ~13 min) if we send a new packet it is sending correctly. Here we see a new session is getting created:

*10:16:43.873325 IP y.y.y.167.39859 > x.x.x.101.514: Flags [S], seq 1000783963, win 14600, options [mss 1460,nop,wscale 7], length 0

10:16:43.873658 IP x.x.x.101.514 > y.y.y.167.39859: Flags [S.], seq 231452091, ack 1000783964, win 29200, options [mss 1460,nop,wscale 7], length 0

10:16:43.873740 IP y.y.y.167.39859 > x.x.x.101.514: Flags [.], ack 1, win 115, length 0

10:16:43.873904 IP y.y.y.167.39859 > x.x.x.101.514: Flags [P.], seq 1:31, ack 1, win 115, length 30

10:16:43.874084 IP x.x.x.101.514 > y.y.y.167.39859: Flags [.], ack 31, win 229, length 0*

Does any one faced such issue? Can any one tell why server is not closing the connection when client sends flag[F]. Do we have any configuration parameter in rsyslogd to create a new session when server sends flag[F]?

Answer 1

Why client is sending data after receiving FIN and ACKed it?
TCP connection termination is a 4 way handshake, which means once a client received FIN from server, it acknowledges it and sends all remaining data to server before sending another FIN to server and wait for it's ACK to complete the hand-shake and fully close the connection.

Logs you have provided shows that, the connection is half-open when the server restarted (which it should have not done, before connection is full-close). and that's why the client is sending remaining data before completing handshake.

What is the correct way of abrupt termination?
When an endpoint is about to abruptly terminate a connection, while already some data is in transfer, it should send RST packet instead of FIN.

Why the RST packet sent by server after restart is not received in client?
It must have been discarded as already the connection is half-open with FIN packet received earlier, or it must have been discarded by client firewall for potential TCP Reset attack