GKE RPC Health Check with multiple ports and different protocols under the same backend service

Question

I am trying to spin up a third-party service that accepts connections in 4 different ports:

x-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: x-deployment
  labels:
    app: x
...
ports:
  - containerPort: 8000 # HttpGraphQLServer
  - containerPort: 8001 # WebSocketServer
  - containerPort: 8020 # JsonRpcServer
  - containerPort: 8030 # HttpIndexingServer
livenessProbe:
  tcpSocket:
    port: 8020

x-service.yaml

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: x-rpc-config
spec:
  healthCheck:
    checkIntervalSec: 7
    timeoutSec: 3
    healthyThreshold: 2
    unhealthyThreshold: 2
    type: HTTP2
    port: 8020
---
apiVersion: v1
kind: Service
metadata:
  name: x-service
  annotations:
    beta.cloud.google.com/backend-config: '{"default": "x-rpc-config"}'
spec:
  selector:
    app: x
  ports:
    - name: graphql
      port: 8000
      targetPort: 8000
    - name: subscription
      port: 8001
      targetPort: 8001
    - name: indexing
      port: 8030
      targetPort: 8030
    - name: jrpc
      port: 8020
      targetPort: 8020
  type: NodePort

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: backend-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: backend-dev-ip-address
    networking.gke.io/managed-certificates: backend-certificate
spec:
  rules:
    - host: x.dev.domain.io
      http:
        paths:
          - path: /rpc
            backend:
              serviceName: x-service
              servicePort: 8020
          - path: /idx
            backend:
              serviceName: x-service
              servicePort: 8030
          - path: /ws
            backend:
              serviceName: x-service
              servicePort: 8001
          - path: /*
            backend:
              serviceName: x-service
              servicePort: 8000

By default the GKE LoadBalancer runs the Health Check on HTTP:80, if I spin up the backend-service (x-service.yaml) without the BackendConfig (x-rpc-config), it is able to detect only 2 healthy backend-services, both with HTTP ports: 8000 and 8030). However the backend-services listen to ports: 8020 (RPC) and 8030 (WS) are not considered healthy. I believe it happens because of the protocol type, so I've created the BackendConfig (x-rpc-config) to run a TPC Health Check instead, using HTTP2 protocol for port 8020 - which is where the livenessProbe is pointing to.

The pods and services are created properly, but the Load Balancer still fails to detect them as healthy services. The console simply shows the following warning:

Some backend services are in UNHEALTHY state

The goal is to open up the port 8020 (RPC) but also keep the 8000 (HTTP) working. Is it possible? Do I need another type of Load Balancer or it is just a config issue?

I could not find any example of HealthCheck config for multiple ports with different protocols under the same service. It is probably an anti-pattern?

Thanks in advance.

Could you elaborate more about third-party service that accepts? What GKE version are you using. Did you configured firewall? Thats all configuration youve applied in your cluster? Its standard cluster or private? Did you configure network in any way? — PjoterS
Sure @PjoterS, the third-party service is a graph-node (hub.docker.com/r/graphprotocol/graph-node), it launches 4 servers. I am interested on the RPC and HTTP servers - ports 8020 and 8000 respectively. I am using GKE v1.17.13-gke.600, and network config is the default setup for a ingress with global static ip. The only change I've made though was to set HTTP2 protocol for the load balancer that points to port 8020 from the x-service, and I could see that the x-rpc-config was properly applied to the Health Check setup of the Load Balancer. — fforbeck
Did you consider to use HTTP(S) LB with custom port number health checks and NEG ? — PjoterS
Great to hear that. Could you provide your solution as an answer? It might help other users who encounter similar issue. — PjoterS

fforbeck fforbeck · Accepted Answer · 2020-11-25T13:24:38

Solution

Instead of using an Ingress, which will launch a HTTP/HTTPs Load Balancer on GCP by default, I've changed the Service to work as a LoadBalancer with a custom HTTP2 health check config. By default this configuration will spin up a TCP Load Balancer on GCP. For instance:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: rpc-config
spec:
  healthCheck:
    checkIntervalSec: 10
    timeoutSec: 3
    healthyThreshold: 2
    unhealthyThreshold: 2
    type: HTTP2
    port: 8020
---
apiVersion: v1
kind: Service
metadata:
  name: x-service
  annotations:
    cloud.google.com/app-protocols: '{"rpc-a":"HTTP2", "rpc-b":"HTTP2", "rpc-c":"HTTP2"}'
    beta.cloud.google.com/backend-config: '{"default": "rpc-config"}'
spec:
  selector:
    app: x-node
  ports:
    - name: rpc-a
      port: 5001
      protocol: TCP
      targetPort: 5001
    - name: rpc-b
      port: 8020
      protocol: TCP
      targetPort: 8020
    - name: rpc-c
      port: 8000
      protocol: TCP
      targetPort: 8000
  type: LoadBalancer

The next step is to enable the SSL for the TCP LB. I saw GCP has the SSL Proxy LB, that might solve it. Just need to figure out the proper configuration for that, I could not find it in their docs.

GKE RPC Health Check with multiple ports and different protocols under the same backend service

1 Answers