IO Exception when writing files to Google Cloud Storage

Question

I am trying to write files from Google App Engine to Google Cloud, and I use backend to process a request which writes a bunch of files to google cloud storage. But it seems that I get IOException in approximately about 5 minutes (I tried different file size, sleep the thread for 30s for each file etc, and only the time seems to be consistent).

Then I tested the following code:

         String testString = "1,1,1,1,1,1,1,1"; 
         for (int i = 0; i < 200; i++) {
            log.info("Dumping file " + i);
            String fileName = "test/test" + i + ".csv";
            FileService fs = FileServiceFactory.getFileService();
            GSFileOptionsBuilder optionsBuilder = new GSFileOptionsBuilder()
                .setBucket(BUCKET_NAME)
                .setKey(fileName)
                .setAcl("public_read");
            AppEngineFile writableFile = fs.createNewGSFile(optionsBuilder.build());
            FileWriteChannel writeChannel = fs.openWriteChannel(writableFile, true);
            PrintWriter out = new PrintWriter(Channels.newWriter(writeChannel, "UTF8"));
            for (int j = 0; j < 10; j++) { 
                out.println(testString);
            }
            out.close();
            writeChannel.closeFinally();
            Thread.sleep(30000);
        }

And I get IOException as below:

java.io.IOException
    at com.google.appengine.api.files.FileServiceImpl.translateException(FileServiceImpl.java:617)
    at com.google.appengine.api.files.FileServiceImpl.makeSyncCall(FileServiceImpl.java:590)
    at com.google.appengine.api.files.FileServiceImpl.create(FileServiceImpl.java:498)
    at com.google.appengine.api.files.FileServiceImpl.createNewGSFile(FileServiceImpl.java:153)

In the google cloud storage bucket, 11 files are written and are good (each has 160 bytes), and the request fails on creation of the twelfth file. For my real application, it will fail on writeChannel.closeFinally().

java.io.IOException
    at com.google.appengine.api.files.FileServiceImpl.translateException(FileServiceImpl.java:617)
    at com.google.appengine.api.files.FileServiceImpl.makeSyncCall(FileServiceImpl.java:590)
    at com.google.appengine.api.files.FileServiceImpl.close(FileServiceImpl.java:551)
    at com.google.appengine.api.files.FileServiceImpl.close(FileServiceImpl.java:441)
    at com.google.appengine.api.files.FileWriteChannelImpl.closeFinally(FileWriteChannelImpl.java:81)

btw. I am sure I give full control access to the google service account of my GAE application in order to write to the corresponding bucket.

Thanks very much!

What 10MB size limit? There is no size limit for the files in Google Storage or Blobstore. — Stuart Langley
@StuartLangley : The limit per request is specified here. I basically want to transfer the data in google app engine to google cloud storage, but I kept getting IO Exceptions after ~40MB of data written (in multiple chunks). I have updated the question with some error stacktrace. Thanks for your help! — Jack Guo
There is no 10MB limit - there is a 32MB per call limit. Have you added your applications service account to the Google Storage bucket so that the app can write to the bucket? — Stuart Langley
@StuartLangley : Thanks for clarifying this. I tried a few tests and narrows down the problem a bit, and modify the question accordingly. I am sure I give my GAE application FULL_CONTROL privilege by "setacl" using gsutil (some files are written in google cloud storage). Now the problem seems to be something like a deadline exceeded exception. Thanks! — Jack Guo

Stuart Langley Stuart Langley · Accepted Answer · 2012-05-24T17:50:56

The files API is still experimental - I think you're now just running into general issues with the service because it's still experimental.

I know that there is some very active work happening in improving the reliability of this service so for now you should probably code defensively and implement retries when you see transiently errors like this.

IO Exception when writing files to Google Cloud Storage

1 Answers