Google Cloud HTTP(S) load balancer does not cancel connection with backend

Question

I have a Google Kubernetes Engine cluster, inside several pods with NodePorts, and all is exposed via an Ingress, which creates an HTTP load balancer (LB). I am using custom domain with Google managed SSL certificate for the LB.

My backend is an HTTP server written in Go, using its "net/http" package. It is using self-signed certificate for mTLS with LB (Google's HTTP LB accepts any certificate for mTLS).

Everything works fine, except for one case, and that is when a client creates an HTTP 1.1 connection with the LB and then cancels the request. This cancels the connection between the client and the LB, but LB holds an open connection with my backend until server's timeout.

My use-case requires requests to be opened even for hours, so my server has huge timeout values. Business logic inside the request is correctly using the request's Context and takes into account if the request is canceled by the client.

Everything works as expected if the client makes an HTTP2 request and cancels it i.e. the whole connection down to my backend is canceled.

Here is an example Go handler that simulates a cancelable long-running task:

func handleLongRunningTask(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    t := time.Now()

    select {
    case <-ctx.Done():
        log.Println("request canceled")
    case <-time.After(30 * time.Second):
        log.Println("request finished")
    }
    log.Printf("here after: %v\n", time.Since(t))

    w.WriteHeader(http.StatusOK)
}

The case <-ctx.Done(): is never called for canceled HTTP 1.1 requests.

For easy testing I am using curl and Ctrl+C; this works:

curl -v --http2 'https://example.com/long_running_task'

and this does not:

curl -v --http1.1 'https://example.com/long_running_task'

It does not matter if the NodePort is HTTPS or HTTP2, the LB has exactly the same behaviour regarding requests canceled by clients.

I tried compiling the server with Go 1.14.4 and 1.13.12 and the results are the same.

Is this a bug in Kubernetes, Ingress, Google Kubernetes Engine, Google's HTTP Load Balancer, Go's HTTP server? Or is it something with HTTP 1.1 that I am missing? What can be wrong and how can I fix this?

...it is not possible to know the HTTP version in the backend, so I could reject all HTTP 1.1 requests. LB is always using the same HTTP version when communicating with its backends, no matter the client's HTTP version.

yyyyahir yyyyahir · Accepted Answer · 2020-06-04T16:26:40

From your description it looks like the issue might be between the GFE and the backends, since GFE might hold the connections for reuse.

My take is that you're seeing this variation between protocol version because how both handle connection persistence.

For HTTP2, the connection will be open until one of the parties send a termination signal and the earliest takes preference. But for HTTP1.1, it might be prolonged until an explicit connection header is sent, specifying the termination:

An HTTP/1.1 server MAY assume that a HTTP/1.1 client intends to maintain a persistent connection unless a Connection header including the connection-token "close" was sent in the request. If the server chooses to close the connection immediately after sending the response, it SHOULD send a Connection header including the connection-token close.

This might explain why HTTP1.1 follows the same timeout configuration as the LB and HTTP2 doesn't.

I'd recommend trying to actively send termination headers whenever you want to terminate a connection. An example taken from Github:

func (m *MyHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    log.Printf("HTTP request from %s", r.RemoteAddr)
    // Add this header to force to close the connection after serving the request.
    w.Header().Add("Connection", "close")
    fmt.Fprintf(w, "%s", m.hostname)
}

Additionally, there seem to be some success stories switching your cluster to be VPC Native, as it takes out of the equation the kube-proxy connection management.

Finally, it might be that you're in a very specific situation that is worth being evaluated separately. You might want to try to send some replication steps to the GKE team using Issue Tracker.

I hope this helps.

Google Cloud HTTP(S) load balancer does not cancel connection with backend

1 Answers