We are trying to upload multiple files i.e. 100 or more than that to Google Cloud Storage, but we get the
below error randomly.
Average File Size: 30MB - 40 MB
08:57:15.488 [ForkJoinPool-1-worker-26] INFO
demo.storage.GCPStorageComponent - Error writing request body to
server com.google.cloud.storage.StorageException: Error writing
request body to server
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.write(HttpStorageRpc.java:581)
at com.google.cloud.storage.BlobWriteChannel$1.run(BlobWriteChannel.java:51)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
at com.google.cloud.storage.BlobWriteChannel.flushBuffer(BlobWriteChannel.java:47)
at com.google.cloud.BaseWriteChannel.close(BaseWriteChannel.java:160)
at demo.storage.GCPStorageComponent.uploadFile(GCPStorageComponent.java:56)
at demo.storage.StorageRepository.uploadFile(StorageRepository.java:56)
at demo.restful.UploadRepository.uploadFileToBucket(UploadRepository.java:132)
at demo.restful.UploadRepository.processClipResults(UploadRepository.java:56)
at demo.restful.UploadRepository.lambda$getClipModelCallableList$0(UploadRepository.java:85)
at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.io.IOException: Error writing request body to server
at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3536)
at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3519)
at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:55)
at com.google.api.client.util.IOUtils.copy(IOUtils.java:94)
at com.google.api.client.http.AbstractInputStreamContent.writeTo(AbstractInputStreamContent.java:72)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:80)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.write(HttpStorageRpc.java:563)
... 17 more
Below is our code, which we are using to upload files. We are uploading files in chunk of 1 MB using multi-part approach.
public void uploadFile(String bucketName, String bucketFileName, Path localFilePath)
throws IOException, StorageException {
BlobInfo blobInfo = BlobInfo.newBuilder(bucketName, bucketFileName).build();
ByteBuffer buffer = ByteBuffer.allocate(10_000_000); //i.e. 1 MB
try (WriteChannel writer = storage.writer(blobInfo);
ByteChannel inChannel = Files.newByteChannel(localFilePath);) { // write to bucket
while (inChannel.read(buffer) > 0) {
buffer.flip();
writer.write(buffer);
buffer.clear();
}
} catch (com.google.cloud.storage.StorageException ex) {
LOGGER.info(ex.getMessage());
ex.printStackTrace();
}
}
Edit: Access is being done from within Google Cloud
An Github Issue has been opened for same
