gRPC Unary Stream from Python server -> C# Client: "Stream removed"

Question

I have a unary streaming service from a Python server (1.8.4) to C# client (1.8.3).

When I signal the Python to shutdown (TERM15) the shutdown method in the code listing below is called, the intention of which is to terminate the RPCs gracefully, and shut the server down.

This works when I'm running a server on localhost, an RpcException with the expected status is raised: Status(StatusCode=Cancelled, Detail="Completed")

However, when running the server in Kubernetes(GKE) and terminating the pod I receive: Grpc.Core.RpcException: Status(StatusCode=Unknown, Detail="Stream removed") in the client. The python server is behind an haproxy ingress controller and google-cloud-endpoint proxy, but in my understanding neither of these components should affect the connection.

Can anybody think what could be causing the different statuses in the RpcException on the client?

class StreamsService(StreamsServicer):
    def MyHandler(self, request, context):
        while not stop_event.isSet():
            try:
                yield update_q.get(True, 0.1)
            except queue.Empty:
                continue

        context.set_code(grpc.StatusCode.CANCELLED)
        context.set_details("Completed")

def shutdown(subscriber_service: StreamsService,
             executor: futures.ThreadPoolExecutor,
             server: grpc.Server, exit_code):    

    logger.info("Stopping stream handlers")
    for stop_event in subscriber_service.stop_events:
        stop_event.set()

    logger.info("Stopping executor")
    executor.shutdown()

    logger.info("Stopping server")
    ev: threading.Event = server.stop(grace=10)  # allows RPCs to terminate gracefully

    logger.info("Waiting for server to stop gracefully")
    ev.wait()

    logger.info("Stopping process with exit code {}".format(exit_code))
    sys.exit(exit_code)

(ps. cross-posted from https://groups.google.com/forum/#!topic/grpc-io/Ge6hcUUhXzo)

MarkNS MarkNS · Accepted Answer · 2018-01-26T16:27:59

Turns out this was a problem with the google-cloud-endpoint/NGINX container being terminated before the gRPC server container in the same pod had a chance to close it's connections gracefully.

If anyone is remotely interested in knowing more, there's a conversation I had with myself over here: https://groups.google.com/forum/#!topic/google-cloud-endpoints/FyfdvD6xS1Q

gRPC Unary Stream from Python server -> C# Client: "Stream removed"

1 Answers