JAVA GRPC server side stream keeps increasing max-inbound-message-size

Question

What are the best practices of streaming lots of data with GRPC? I am sending a request to a GRPC server that will stream back data. The data send back can be a in the order of 100 protobuf messages or can be a couple of 100.000 protobuf messages.

service CrossEngineSelector {
    rpc QueryDB (QueryRequest) returns (stream QueryResponse) {}
}

The server is a simple implementation that sends the protobuf messages.

@Override
public void queryBD(QueryRequest request, StreamObserver<QueryResponse> responseObserver) {

      Iterables.partition( dataLoader.getData(), 1000).forEach(batch -> {
          responseObserver.onNext(QueryResponse.newBuilder().addAllRows(batch).build());
      });
      responseObserver.onCompleted();
}

The client side uses a blockingStub that calls this method (generated code by protobuf):

public Iterator<QueryResponse> queryDB(QueryRequest request) {
            return ClientCalls.blockingServerStreamingCall(this.getChannel(),
                   CrossEngineSelectorGrpc.getQueryEnginesMethod(),
                   this.getCallOptions(), request);
}

Once the client calls this method I just iterate over the QueryResponse.

All of this works fine for streams that only send back a small amount of messages. Once I try to stream 100.000 messages the max-inbound-message-size keeps increasing and I end up with an error: RESOURCE_EXHAUSTED: Compressed gRPC message exceeds maximum size 4194304: 4196022 bytes read

My current fix for this is to set the max-inbound-message-size very high +1Gb. This is a hardcoded value, so it doesn't scale. The client has no idea how many messages the server will return. I might run into use cases were even the 1Gb max-inbound-message-size will not be enough.

I hope I am making an implementation mistake. I hope there is a way of resetting the message size for every stream (onNext()) from the server or is it normal it keeps increasing the message size?

I would assume that a single responseObserver.onNext(QueryResponse.newBuilder().addAllRows(batch).build()); sends a couple of Mb and that this would be considered as the message size and not the entire stream for as long as it runs.

I am using Micronaut for both server and client.

creamsoup creamsoup · Accepted Answer · 2020-02-18T19:41:32

The maxInboundMessageSize is for protecting receiver going out of memory if malicious peer sends very large payload to attack. The real problem here is the 1000 chunks can be larger than 4MB (default maxInboundMessageSize).

you can fix this in two ways,

fix the sender to limit by size of the payload, not by # of rows.
or, the receiver have large enough maxInboundMessageSize to handle normal 1000 rows worth of data.

the later method is essentially what you implemented. It should be lower than 1GB+ though. gRPC won't use the maxInboundMessageSize amount of memory as I mentioned above this is purely for protecting. (added later)

JAVA GRPC server side stream keeps increasing max-inbound-message-size

1 Answers