How to set Service Load Balancer request timeouts on GKE

1
votes

I have a Service on GKE of type LoadBalancer that points to a GKE deployment running nginx. My nginx has all of the timeouts set to 10 minutes, yet HTTP/HTTPS requests that have to wait on processing before receiving a response get cutoff with 500 errors after 30 seconds. My settings:

http {
    proxy_read_timeout 600s;
    proxy_connect_timeout 600s;
    keepalive_timeout 600s;
    send_timeout 600s;
}

Apparently there are default settings of 30 seconds in the LoadBalancer somewhere.

After pouring through documentation, I've only found a step-through at Google that outlines setting an Ingress with back-end service Load Balancer with a timeout but can't find how to do that on a Service that's Type=LoadBalancer for use with GKE. I've also reviewed all of the Kubernetes documentation for versions 1.7+ (we're on 1.8.7-gke.1) and nothing about setting a timeout. Is there a setting I can add to my yaml file to do this?

If it helps I found the following for AWS, which appears to be what I would need to have on GKE:

  annotations:
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
2
kubernetesgoogle-cloud-platformgoogle-kubernetes-engine
Hi, check the update of the answer – GalloCedrone

2 Answers

2
votes

So far you cannot do it from the YAML file.

There is a open feature request at the moment that I advise you to subscribe and to follow:

They were already discussing regarding this change in 2016: issue.

"Specific use case: GCE backends are provisioned with a default timeout of 30 seconds, which is not sufficient for some long requests. I'd like to be able to control the timeout per-backend."

However I would suggest you to check this part of the Google Cloud Documentation talking specifically regarding configurable response timeout.

UPDATE Check the issue because they are making progresses

I see there was a v1.0.0 release 18 days ago. Was this the completion of the major refactoring you talked about @nicksardo ? Is it possible yet to configure how long a connection can be idle before the LB closes it? UPDATE The issue mentioned above is now closed and documentation for setting the timeout (and other backend service settings) is available here: https://cloud.google.com/kubernetes-engine/docs/how-to/configure-backend-service

0
votes

As of April 2021 you can do this via GKE/GCE configuration. Here are the instructions. Essentially you create a BackendConfig resource similar to this:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: my-backendconfig
spec:
  timeoutSec: 40
  connectionDraining:
    drainingTimeoutSec: 60

(kubectl apply -f my-backendconfig.yaml)

and then connect it to your GKE service resource with an annotation:

apiVersion: v1
kind: Service
metadata:
  name: my-service
  labels:
    purpose: bsc-config-demo
  annotations:
    cloud.google.com/backend-config: '{"ports": {"80":"my-backendconfig"}}'
    cloud.google.com/neg: '{"ingress": true}'
spec:
  type: ClusterIP
  selector:
    purpose: bsc-config-demo
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080

(kubectl apply -f my-service.yaml)

If you prefer, the BackendConfig resource (and Service) can be placed in a namespace with a metadata namespace designation in your yaml.

metadata:
  namespace: my-namespace