Changing Kubernetes' node-proxy tcp keepalive time

15
votes

How do I properly change the TCP keepalive time for node-proxy?

I am running Kubernetes in Google Container Engine and have set up an ingress backed by HTTP(S) Google Load Balancer. When I continuously make POST requests to the ingress, I get a 502 error exactly once every 80 seconds or so. backend_connection_closed_before_data_sent_to_client error in Cloud Logging, which is because GLB's tcp keepalive (600 seconds) is larger than node-proxy's keepalive (no clue what it is).

The logged error is detailed in https://cloud.google.com/compute/docs/load-balancing/http/.

Thanks!

2
proxykubernetesload-balancinggoogle-kubernetes-engine

2 Answers

2
votes

You can use the custom resource BackendConfig that exist on each GKE cluster to configure timeouts and other parameters like CDN here is the documentacion

An example from here shows how to configure on the ingress

That is the BackendConfig definition:

apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: my-bsc-backendconfig
spec:
  timeoutSec: 40
  connectionDraining:
    drainingTimeoutSec: 60

And this is how to use on the ingress definition through annotations

apiVersion: v1
kind: Service
metadata:
  name: my-bsc-service
  labels:
    purpose: bsc-config-demo
  annotations:
    beta.cloud.google.com/backend-config: '{"ports": {"80":"my-bsc-backendconfig"}}'
spec:
  type: NodePort
  selector:
    purpose: bsc-config-demo
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
-1
votes

just for the sake of understanding, when you use Google solution to load-balance and manage your Kubernetes Ingress, you will have GLBC pods running in kube-system namespace.

You can check it out with :

kubectl -n kube-system get po

These pods are intended to route the incoming traffic from the actual Google Load Balancer.

I think that the timeouts should be configured there, on GLBC. You should check what annotations or ConfigMap GLBC can take to be configured, if any.

You can find details there :

Personally I prefer to use the Nginx Ingress Controller for now, and it has necessary annotations and ConfigMap support.

See :