8
votes

I'm trying to setup a Google Cloud Storage file upload by following the sample from google with the GcsExampleServlet.java . I've complete all the step but when I deploy the project to aggengine and I try to upload a simple text in GCS , it fail with this log :

com.google.appengine.tools.cloudstorage.NonRetriableException: com.google.appengine.tools.cloudstorage.NonRetriableException: com.google.appengine.api.appidentity.AppIdentityServiceFailureException: 
The AppIdentity service threw an unexpected error. Details: 
at com.google.appengine.tools.cloudstorage.RetryHelper.doRetry(RetryHelper.java:120)
at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:166)
at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:156)
at com.google.appengine.tools.cloudstorage.GcsServiceImpl.createOrReplace(GcsServiceImpl.java:70)
at com.appart.storage.server.GcsExampleServlet.doPost(GcsExampleServlet.java:88)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
(...)

Still, there is nothing complicate in the code... In web.xml I've configured the servlet :

<servlet>
  <servlet-name>GcsExample</servlet-name>
    <servlet-class>
      com.example.server.GcsExampleServlet
    </servlet-class>
</servlet>
<servlet-mapping>
  <servlet-name>GcsExample</servlet-name>
  <url-pattern>/gcs/*</url-pattern>
</servlet-mapping>

Here is the servlet GcsExampleServlet.java ( exactly the same as in google sample ) :

@SuppressWarnings("serial")
public class GcsExampleServlet extends HttpServlet {

  private final GcsService gcsService = GcsServiceFactory.createGcsService(new RetryParams.Builder()
      .initialRetryDelayMillis(10)
      .retryMaxAttempts(10)
      .totalRetryPeriodMillis(15000)
      .build());     
  //...
  @Override
  public void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException 
  {
    GcsOutputChannel outputChannel =
    gcsService.createOrReplace(getFileName(req), GcsFileOptions.getDefaultInstance());
    copy(req.getInputStream(), Channels.newOutputStream(outputChannel));
  }

  private void copy(InputStream input, OutputStream output) throws IOException {
    try {
      byte[] buffer = new byte[BUFFER_SIZE];
      int bytesRead = input.read(buffer);
      while (bytesRead != -1) {
        output.write(buffer, 0, bytesRead);
        bytesRead = input.read(buffer);
      }
    } finally {
      input.close();
      output.close();
    }
  }
}

Here is my upload.html file :

<form action="/upload.html" enctype="text/plain" method="get" name="putFile" id="putFile">
   <div>
        Bucket: <input type="text" name="bucket" />
        File Name: <input type="text" name="fileName" />
        <br /> File Contents: <br />
        <textarea name="content" id="content" rows="3" cols="60"></textarea>
        <br />
        <input type="submit" onclick='uploadFile(this)' value="Upload Content" />
   </div>
</form>
<script>

  function uploadFile() {
    var bucket = document.forms["putFile"]["bucket"].value;
    var filename = document.forms["putFile"]["fileName"].value;
    if (bucket == null || bucket == "" || filename == null || filename == "") {
      alert("Both Bucket and FileName are required");
      return false;
    } else {
      var postData = document.forms["putFile"]["content"].value;
      document.getElementById("content").value = null;

      var request = new XMLHttpRequest();
      request.open("POST", "/gcs/" + bucket + "/" + filename, false);
      request.setRequestHeader("Content-Type", "text/plain;charset=UTF-8");
      request.send(postData);
    }
  }
</script>

I've enabled Billing, created a bucket but the AppIdentity error still appear. I've no Oauth, the cloud storage API is enabled, the appengine account used to upload has write access to the bucket. I even tried a

gsutil acl ch -u [email protected]:WRITE gs://ctrlxbucket

To be sure that the user has write access to my bucket.

Please help me to figure out what this error mean, I'm stick here since days :(

Thanks a lot

PS : If you just have some working sample of GCS ( not the google one ), I'll be happy too since there is not a lot of stuff around that topic.

1
I am having similar issue for any AppEngine project created recently to talk to GCS. In python AppEngine code I am getting InternalError that is related to get token code for GCS access. The problem seems to be missing credentials in any new AppEngine app e.g. if you look at Api & auth -> Credentials in Dev Console, it might not have "Compute Engine and AppEngine" client ids for service account. I have a legacy app, created last year, that is working fine though.Abhijit Kalamkar
@Abhijit Kalamkar +1. The same for me. I have got my old project which is performing well. But 2 days ago I created new project with the same code and this error occurred. I also noticed that Compute Engine credentials disappeared on new project but I guess this is not case because CtrlX has got [email protected] account mentioned in this comment. I guess the solution is in some AppEngine configuration.Juniper
@Juniper Fixed the problem, saw this note on cloud.google.com/appengine/docs/python/googlecloudstorageclient/… Note: In some unusual circumstances, you may not be able to add your service account using the Add member form. If this happens, you must add your app's Service Account Number to the bucket ACL. For more information, see Give permissions to your bucket or objects.Abhijit Kalamkar
@AbhijitKalamkar Sound interesting ! I'm trying your solution, but in the doc, when gsutil retrieve the bucket ACL, it seems to be an xml. But I retrieve a JSON. Do you have the same JSON ACL file ?CtrlX
@AbhijitKalamkar Actually, it worked ( with the JSON instead of XML as said in the doc ) ! I have to add { "email": "[email protected]", "entity": "[email protected]", "role": "WRITER" } to the myACL.txt and myDefAcl.txt Can you please create an answer instead of a comment so I can accept it ? ( try to give the set provided in the doc for generating the ACL ) Thanks againCtrlX

1 Answers

1
votes

Instead of [email protected] try using [email protected] (replace XXX with your application's project id - taken from "Google APIs Console Project Number" in the" Administration/Application Settings" of the admin console).