I've set up a TCP network load balancer, as described here: https://cloud.google.com/load-balancing/docs/network. I need to balance traffic from anywhere on the internet to my backend VMs, running a custom application listening to a non-standard TCP port.
Everything seems to work initially, but after about 10 seconds the connected clients are disconnected, reporting the error "An existing connection was forcibly closed by the remote host.". For debugging I allow my backend VMs to have public IPs and when connecting to any of them directly, bypassing the load balancer, everything works and there's no disconnect.
As I understand it, this load balancer setup I'm using should be pass through: Once the backend VM is selected, the TCP connection should essentially be with the back end VM and the load balancer no longer involved. The backend VMs are certainly not terminating the connection forcibly - as far as the backends are concerned, the connection persists after the client disconnect and time out later. The timeout settings described for other google cloud load balancers don't seem to apply to External TCP/UDP Network Load Balancing.
What am I missing?