Istio causing frequent disconnection in grpc stream

Question

I am trying to setup a grpc stream from outside world into istio cluster through istio ingress. I am able to establish the connection, but I am seeing a connection reset every 60sec.

Container logs tell "rpc error: code = Unavailable desc" before breaking

Looked into ingress and envoy logs. nothing much helpful. attached them below.

INGRESS LOGS
[2020-03-06T12:14:10.221Z] "- - -" 0 - "-" "-" 2679 2552 9993 - "-" "-" "-" "-" "10.244.0.93:5448" outbound|5448||grpc-broker.x-infra.svc.cluster.local 10.244.0.116:58094 10.244.0.116:443 10.222.2.9:37864 <xxxxxx DNS NAME xxxxxxxx> -

ENVOY LOGS
[2020-03-06T12:16:28.331Z] "- - -" 0 - "-" "-" 12021 2733 50282 - "-" "-" "-" "-" "127.0.0.1:5448" inbound|5448|tcp-broker|grpc-broker.x-infra.svc.cluster.local 127.0.0.1:56816 10.244.0.93:5448 10.244.0.116:34782 outbound_.5448_._.grpc-broker.x-infra.svc.cluster.local -

Should we add anything extra to grpc stream to work?

Cluster is default mTLS enabled, Source and destination pods are deployment and not stateful set.

Hi, what version of istio do You have? Do You have timeout configured anywhere? Many grpc API's have default 60s timeout. — Piotr Malec
I used the latest 1.4.5 version. I have not configured any timeouts myself, but the istio might have auto set it in some configs. Any idea where I can check the API timeout? — yog raj

Piotr Malec Piotr Malec · Accepted Answer · 2020-03-11T13:13:20

From the istio side to make sure that istio is not shutting down the connection.:

This could be prevented by idleTimeout setting in DestinationRule.

According to istio documentation about idleTimeout:

The idle timeout for upstream connection pool connections. The idle timeout is defined as the period in which there are no active requests. If not set, there is no idle timeout. When the idle timeout is reached the connection will be closed. Note that request based timeouts mean that HTTP/2 PINGs will not keep the connection alive. Applies to both HTTP1.1 and HTTP2 connections.

So If You make DestinationRule like this:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: grpc-iddletimeout-policy
spec:
  host: grpcservice.servicenamespace.svc.cluster.local
  trafficPolicy:
    connectionPool:
      http:
        idleTimeout: 2m

This should close the any HTTP/2 connection from Istio envoy proxy side after being idle for 2 minutes for grpcservice in servicenamespace namespace.

Hope it helps.

Istio causing frequent disconnection in grpc stream

1 Answers