Having trouble uploading large file (using resumable upload) into google cloud storage bucket

Question

So I've been grappling with this issue for some time now. I'm trying to upload a file (~43mb) to my google storage bucket using curl. The following is used to create the request.

curl -vv -X POST --data-binary @/Users/hg0719/Desktop/G7-Enterprise.ipa \
                                  -H "Authorization: Bearer `gcloud auth print-access-token`" \
                                  -H "Content-Type: application/octet-stream" \
                                  -H "Content-Length: *" \
                                  "https://www.googleapis.com/upload/storage/v1/b/dexflight-v0.appspot.com/o?uploadType=resumable&name=ipa/G7-Enterprise.ipa"

Note that I'm referring to this documentation: https://cloud.google.com/storage/docs/uploading-objects

I'm also able to successfully upload small files to the bucket (where the uploadType in the url is set to "media"), but I've had no luck trying to use resumable uploads (uploadType=resumable).

I've also tried directly including the Gcloud access token in the Authorization header, but still produces the same exact output.

Which produces this error message:

Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 216.58.217.202...
* TCP_NODELAY set
* Connected to www.googleapis.com (216.58.217.202) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=*.googleapis.com
*  start date: Aug 23 10:30:37 2019 GMT
*  expire date: Nov 21 10:30:37 2019 GMT
*  subjectAltName: host "www.googleapis.com" matched cert's "*.googleapis.com"
*  issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fc76d805000)
> POST /upload/storage/v1/b/dexflight-v0.appspot.com/o?uploadType=resumable&name=ipa/G7-Enterprise.ipa HTTP/2
> Host: www.googleapis.com
> User-Agent: curl/7.54.0
> Accept: */*
> Authorization: Bearer `gcloud auth print-access-token`
> Content-Type: application/octet-stream
> Content-Length: *
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 400 
< content-type: text/html; charset=UTF-8
< referrer-policy: no-referrer
< content-length: 1555
< date: Thu, 05 Sep 2019 22:47:49 GMT
* HTTP error before end of send, stop sending
< 
* HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)

Mike Schwartz Mike Schwartz · Accepted Answer · 2019-09-06T00:04:14

I suggest you try using gsutil -d to do a resumable upload, and looking at the debug output, which will show the API requests/responses:

gsutil -d cp bigfile gs://your-bucket/bigfile

You can then compare what gsutil did with the request you're attempting to construct via curl.

Having trouble uploading large file (using resumable upload) into google cloud storage bucket

1 Answers