Internal gRPC routing times out after injecting Istio Sidecar

Question

I am currently trying to deploy an Istio Service Mesh in our Kubernetes cluster on GKE. When I inject the Istio sidecar using kubectl apply -f <(istioctl kube-inject -f <service-name>.yaml the downstream services are not able to connect using gRPC. The issue does not present itself when the Istio sidecar is running in an HTTP service.

The configuration file for the gRPC service is as follows:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: <service name>
  namespace: default
  labels:
    app: <service-name>
    version: v1
spec:
  minReadySeconds: 30
  replicas: 1
  template:
    metadata:
      labels:
        app: <service-name>
    spec:
      containers:
      - name: <service-name>
        image: gcr.io/project/service-name
        ports:
        - containerPort: 5011
          name: grpc
        resources:
          requests:
            memory: "10Mi"
            cpu: "100m"
          limits:
            memory: "100Mi"
            cpu: "250m"
        readinessProbe:
          tcpSocket:
            port: 5011
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 5011
          initialDelaySeconds: 15
          periodSeconds: 20
---
apiVersion: v1
kind: Service
metadata:
  name: <service-name>
  namespace: default
spec:
  selector:
    app: <service-name>
  ports:
  - port: 5011
    targetPort: 5011
    name: grpc

Istio is running in the downstream service and does not show any outgoing requests to the upstream service. If I redeploy the upstream service without Istio, the gRPC call succeeds and the logs show an outgoing request.

Brendan Chan Brendan Chan · Accepted Answer · 2018-08-14T14:25:48

I managed to fix this. The issue was that the upstream gRPC service was making a request to Datastore and Istio wasn't configured to let traffic to Google Cloud services out of the mesh, so the request was timing out. I solved this by running

helm template install/kubernetes/helm/istio <the flags you used to install Istio> 
--set global.proxy.includeIPRanges="<cluster IP ranges>" 
-x templates/sidecar-injector-configmap.yaml | kubectl apply -f -

Internal gRPC routing times out after injecting Istio Sidecar

1 Answers