golang gRPC 503 on CORS preflight using Istio grpc-web proxy

Question

I'm trying to set up a go-based gRPC server in Kubernetes, and making requests via a grpc-web client. The client first runs a CORS preflight request, which seem to make it to the gRPC server. The server is returning a 503 on these option requests. I can use the go gRPC client to connect directly to the server, so I think if the CORS request goes through, the rest will work fine.

• Here's the log messages from the server for the OPTIONS request:

2020/06/02 23:46:42 http2: decoded hpack field header field ":authority" = "localhost:32443"
2020/06/02 23:46:42 http2: decoded hpack field header field ":path" = "/helloworld.Greeter/SayHello"
2020/06/02 23:46:42 http2: decoded hpack field header field ":method" = "OPTIONS"
2020/06/02 23:46:42 http2: decoded hpack field header field ":scheme" = "http"
2020/06/02 23:46:42 http2: decoded hpack field header field "user-agent" = "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"
2020/06/02 23:46:42 http2: decoded hpack field header field "accept" = "*/*"
2020/06/02 23:46:42 http2: decoded hpack field header field "accept-language" = "en-US,en;q=0.5"
2020/06/02 23:46:42 http2: decoded hpack field header field "accept-encoding" = "gzip, deflate"
2020/06/02 23:46:42 http2: decoded hpack field header field "pragma" = "no-cache"
2020/06/02 23:46:42 http2: decoded hpack field header field "cache-control" = "no-cache"
2020/06/02 23:46:42 http2: decoded hpack field header field "x-forwarded-for" = "192.168.86.23"
2020/06/02 23:46:42 http2: decoded hpack field header field "x-forwarded-proto" = "http"
2020/06/02 23:46:42 http2: decoded hpack field header field "x-request-id" = "1485ede7-ae5f-94fd-8922-0d1726ee6c44"
2020/06/02 23:46:42 http2: decoded hpack field header field "x-envoy-internal" = "true"
2020/06/02 23:46:42 http2: decoded hpack field header field "x-forwarded-client-cert" = "By=spiffe://cluster.local/ns/default/sa/default;Hash=2cd071712de3d01ef26a994f2bafebb79081fa7ff96d79bc31ee28cd4f32f8d2;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
2020/06/02 23:46:42 http2: decoded hpack field header field "x-b3-traceid" = "4125f45329ec3d40f845b6801863416a"
2020/06/02 23:46:42 http2: decoded hpack field header field "x-b3-spanid" = "42e83011657bf466"
2020/06/02 23:46:42 http2: decoded hpack field header field "x-b3-parentspanid" = "f845b6801863416a"
2020/06/02 23:46:42 http2: decoded hpack field header field "x-b3-sampled" = "1"
2020/06/02 23:46:42 http2: Framer 0xc0001a01c0: wrote RST_STREAM stream=19 len=4 ErrCode=INTERNAL_ERROR

• Here's the log messages from the server when connecting directly to the pod with the grpc-client:

2020/06/03 00:05:59 http2: Framer 0xc0001a01c0: read HEADERS flags=END_HEADERS stream=35 len=103
2020/06/03 00:05:59 http2: decoded hpack field header field ":method" = "POST"
2020/06/03 00:05:59 http2: decoded hpack field header field ":scheme" = "http"
2020/06/03 00:05:59 http2: decoded hpack field header field ":path" = "/helloworld.Greeter/SayHello"
2020/06/03 00:05:59 http2: decoded hpack field header field ":authority" = "192.168.102.135:50051"
2020/06/03 00:05:59 http2: decoded hpack field header field "content-type" = "application/grpc"
2020/06/03 00:05:59 http2: decoded hpack field header field "user-agent" = "grpc-go/1.30.0-dev"
2020/06/03 00:05:59 http2: decoded hpack field header field "te" = "trailers"
2020/06/03 00:05:59 http2: decoded hpack field header field "grpc-timeout" = "1000m"
2020/06/03 00:05:59 http2: decoded hpack field header field "x-forwarded-proto" = "http"
2020/06/03 00:05:59 http2: decoded hpack field header field "x-request-id" = "dacfa197-e319-9e96-bcbe-963121341275"
2020/06/03 00:05:59 http2: decoded hpack field header field "x-envoy-expected-rq-timeout-ms" = "1000"
2020/06/03 00:05:59 http2: decoded hpack field header field "x-b3-traceid" = "6c709591324ca3724f66ea0f60673a9c"
2020/06/03 00:05:59 http2: decoded hpack field header field "x-b3-spanid" = "4f66ea0f60673a9c"
2020/06/03 00:05:59 http2: decoded hpack field header field "x-b3-sampled" = "1"
2020/06/03 00:05:59 http2: Framer 0xc0001a01c0: read DATA flags=END_STREAM stream=35 len=12 data="\x00\x00\x00\x00\a\n\x05world"
2020/06/03 00:05:59 Received: world
2020/06/03 00:05:59 http2: Framer 0xc0001a01c0: wrote WINDOW_UPDATE len=4 (conn) incr=12
2020/06/03 00:05:59 http2: Framer 0xc0001a01c0: wrote PING len=8 ping="\x02\x04\x10\x10\t\x0e\a\a"
2020/06/03 00:05:59 http2: Framer 0xc0001a01c0: wrote HEADERS flags=END_HEADERS stream=35 len=2
2020/06/03 00:05:59 http2: Framer 0xc0001a01c0: wrote DATA stream=35 len=18 data="\x00\x00\x00\x00\r\n\vHello world"
2020/06/03 00:05:59 http2: Framer 0xc0001a01c0: wrote HEADERS flags=END_STREAM|END_HEADERS stream=35 len=2
2020/06/03 00:05:59 http2: Framer 0xc0001a01c0: read PING flags=ACK len=8 ping="\x02\x04\x10\x10\t\x0e\a\a"

• For the server, I'm using the helloworld (https://github.com/grpc/grpc-go/tree/master/examples/helloworld)
• For the client, I generated the grpc-web files by running the following from within the helloworld directory with the .proto file: protoc --js_out=import_style=commonjs:. --grpc-web_out=import_style=commonjs,mode=grpcwebtext:. helloworld.proto
• The code executing the client:

    var c = new GreeterClient('http://localhost:32443');
    c.sayHello(new HelloRequest("world"), {}, (err, response) => {
      if (err != null) {
        alert(err.name, err.message);
      } else {
        console.log(response);
      }
    });

• The gateway & virtual service:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: grpc-helloworld
spec:
  host: grpc-helloworld
  subsets:
  - name: v1
    labels:
      version: v1
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: vs
spec:
  hosts:
  - "*"
  gateways:
  - gateway
  http:
  - match:
    - port: 80
    route:
    - destination:
        host: grpc-helloworld
        port:
          number: 50051
        subset: v1
    corsPolicy:
      allowOrigin:
        - "*"
      allowMethods:
        - POST
        - GET
        - OPTIONS
        - PUT
        - DELETE
      allowHeaders:
        - grpc-timeout
        - content-type
        - keep-alive
        - user-agent
        - cache-control
        - content-type
        - content-transfer-encoding
        - custom-header-1
        - x-accept-content-transfer-encoding
        - x-accept-response-streaming
        - x-user-agent
        - x-grpc-web
      maxAge: 1728s
      exposeHeaders:
        - custom-header-1
        - grpc-status
        - grpc-message
      allowCredentials: true

• The service:

apiVersion: v1
kind: Service
metadata:
  name: grpc-helloworld
  labels:
    app: grpc-helloworld
spec:
  ports:
  - name: grpc-web
    port: 50051
  selector:
    app: grpc-helloworld

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grpc-helloworld
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grpc-helloworld
  template:
    metadata:
      labels:
        app: grpc-helloworld
        version: v1
    spec:
      containers:
      - name: grpc-helloworld
        image: grpc-helloworld:1.0
        env:
        - name: GRPC_GO_LOG_SEVERITY_LEVEL
          value: "info"
        - name: GRPC_GO_LOG_VERBOSITY_LEVEL
          value: "2"
        - name: GODEBUG
          value: "http2debug=2"
        ports:
        - containerPort: 50051
          name: grpc-web

Answer 1

In Istio 1.6 at least, the allowOrigin directive I was using wasn't working. I changed it to allowOrigins:

allowOrigins:
  - regex: ".*"