grpc-okhttp-3 java.lang.OutOfMemoryError: Failed to allocate a 8204 byte allocation with 3880 free bytes and 3KB until OOM

Question

I am trying to upload a file of size more than 300 MB using StreamObserver grpc it gives the following exception. I am uploading it in chunks of 512 KB.

grpc-okhttp-3 java.lang.OutOfMemoryError: Failed to allocate a 8204 byte allocation with 3880 free bytes and 3KB until OOM at okio.Segment.(Segment.java:63) ~[na:0.0] at okio.SegmentPool.take(SegmentPool.java:48) ~[na:0.0] at okio.Buffer.writableSegment(Buffer.java:1211) ~[na:0.0] at okio.Buffer.writeByte(Buffer.java:1069) ~[na:0.0] at okio.RealBufferedSink.writeByte(RealBufferedSink.java:124) ~[na:0.0] at io.grpc.okhttp.internal.framed.Http2.writeMedium(Http2.java:773) ~[na:0.0] at io.grpc.okhttp.internal.framed.Http2.access$600(Http2.java:47) ~[na:0.0] at io.grpc.okhttp.internal.framed.Http2$Writer.frameHeader(Http2.java:578) ~[na:0.0] at io.grpc.okhttp.internal.framed.Http2$Writer.dataFrame(Http2.java:498) ~[na:0.0] at io.grpc.okhttp.internal.framed.Http2$Writer.data(Http2.java:493) ~[na:0.0] at io.grpc.okhttp.AsyncFrameWriter$9.doRun(AsyncFrameWriter.java:148) ~[na:0.0] at io.grpc.okhttp.AsyncFrameWriter$WriteRunnable.run(AsyncFrameWriter.java:220) ~[na:0.0] at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[na:0.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133) ~[na:0.0] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607) ~[na:0.0] at java.lang.Thread.run(Thread.java:761) ~[na:0.0]

BufferedInputStream bInputStream = new BufferedInputStream(new FileInputStream(file));

try {

int bufferSize = 512; // 512k
byte[] buffer = new byte[bufferSize];
long size = 0, originalSize = file.length();
int chunk;
do {
chunk = bInputStream.read(buffer);
size += chunk;
if (chunk > 0) {
ByteString byteString = null;
if (chunk < 512) {
byte[] remaining = Arrays.copyOfRange(buffer, 0, chunk);
byteString = ByteString.copyFrom(remaining);
 } else {
byteString = ByteString.copyFrom(buffer);
}
putRequestStreamObserver.onNext(FileUploadRequest.newBuilder()
.setFileChunk(FileChunk.newBuilder()
.setChunk(byteString)
.build())
.build());

logger.info("New Chunk!");

i have android:hardwareAccelerated="false" and android:largeHeap="true" in my manifest.

Eric Anderson Eric Anderson · Accepted Answer · 2020-06-01T16:37:22

Since the StreamObserver API is asynchronous, it can return before the message is sent. If you send too quickly, they will buffer locally.

You need to observe flow control/backpressure. You should create a ClientResponseObserver to be provided the ClientCallStreamObserver and call setOnReadyHandler() within beforeStart(). You'd then want to observe isReady() and avoid sending when it becomes false. See the manualflowcontrol example; you can ignore the disableAutoInboundFlowControl()/request() pieces; that is for receiving, not sending.

grpc-okhttp-3 java.lang.OutOfMemoryError: Failed to allocate a 8204 byte allocation with 3880 free bytes and 3KB until OOM

1 Answers