Go GRPC client disconnect terminates Go server

Question

Bit of a newb to both Go and GRPC, so bear with me.

Using go version go1.14.4 windows/amd64, proto3, and latest grpc (1.31 i think). I'm trying to set up a bidi streaming connection that will likely be open for longer periods of time. Everything works locally, except if I terminate the client (or one of them) it kills the server as well with the following error:

Unable to trade data rpc error: code = Canceled desc = context canceled

This error comes out of this code server side

func (s *exchangeserver) Trade(stream proto.ExchageService_TradeServer) error {

    endchan := make(chan int)
    defer close(endchan)

    go func() {
        for {
            req, err := stream.Recv()
            if err == io.EOF {
                break
            }
            if err != nil {
                log.Fatal("Unable to trade data ", err)
                break
            }

            fmt.Println("Got ", req.GetNumber())
        }

        endchan <- 1
    }()

    go func() {
        for {
            resp := &proto.WordResponse{Word: "Hello again "}
            err := stream.Send(resp)
            if err != nil {
                log.Fatal("Unable to send from server ", err)
                break
            }

            time.Sleep(time.Duration(500 * time.Millisecond))
        }

        endchan <- 1
    }()

    <-endchan
    return nil
}

And the Trade() RPC is so simple it isn't worth posting the .proto. The error is clearly coming out of the Recv() call, but that call blocks until it sees a message, like the client disconnect, at which point I would expect it to kill the stream, not the whole process. I've tried adding a service handler with HandleConn(context, stats.ConnStats) and it does catch the disconnect before the server dies, but I can't do anything with it. I've even tried creating a global channel that the serve handler pushes a value into when HandleRPC(context, stats.RPCStats) is called and only allowing Recv() to be called when there's a value in the channel, but that can't be right, that's like blocking a blocking function for safety and it didn't work anyway.

This has to be one of those real stupid mistakes that beginner's make. Of what use would GPRC be if it couldn't handle a client disconnect without dying? Yet I have read probably a trillion (ish) posts from every corner of the internet and noone else is having this issue. On the contrary, the more popular version of this question is "My client stream stays open after disconnect". I'd expect that issue. Not this one.

SwiftD SwiftD · Accepted Answer · 2020-09-09T23:49:45

Im not 100% sure how this is supposed to behave but I note that you are starting separate receive and send goroutines up at the same time. This might be valid but is not the typical approach. Instead you would usually receive what you want to process and then start a nested loop to handle the reply .

See an example of typical bidirectional streaming implementation from here: https://grpc.io/docs/languages/go/basics/

func (s *routeGuideServer) RouteChat(stream pb.RouteGuide_RouteChatServer) error {
    for {
        in, err := stream.Recv()
        if err == io.EOF {
            return nil
        }
        if err != nil {
            return err
        }
        key := serialize(in.Location)
                ... // look for notes to be sent to client
        for _, note := range s.routeNotes[key] {
            if err := stream.Send(note); err != nil {
                return err
            }
        }
    }
}

sending and receiving at the same time might be valid for your use case but if that is what you are trying to do then I believe your handling of the channels is incorrect. Either way, please read on to understand the issue as it is a common one in go.

You have a single channel which only blocks until it receives a single message, once it unblocks the function ends and the channel is closed (by defer).

You are trying to send to this channel from both your send and receive loop.

When the last one to finish tries to send to the channel it will have been closed (by the first to finish) and the server will panic. Annoyingly, you wont actually see any sign of this as the server will exit before the goroutine can dump its panic (no clues - probably why you landed here)

see an example of the issue here (grpc code stripped out): https://play.golang.org/p/GjfgDDAWNYr Note: comment out the last pause in the main func to stop showing the panic reliably (as in your case)

So one simple fix would probably be to simply create two separate channels (one for send, one for receive) and block on both - this however would leave the send loop open necessarily if you don't get a chance to respond so probably better to structure like the example above unless you have good reason to pursue something different.

Another possibility is some sort server/request context mix up but I'm pretty sure the above will fix - drop an update with your server setup code if your still having issues after the above changes

Go GRPC client disconnect terminates Go server

1 Answers