Can't upload file to GCS using gsutil

4
votes

Copying file://InstreamImpression.csv.gz [Content-Type=application/octet-stream]...

AccessDeniedException: 401 Login Requiredfe13d1e0fb408639_4...: 46.75 MB/46.77 MB

CommandException: 1 files/objects could not be transferred.

Seems like whole object is being transferred but gives 401 error at the end. And it's been happening for a while.

Ran "gcloud auth login" couple times. But still the same error

I am able to upload the files from different machine.

Any idea?

2
google-cloud-storagegsutil
Are you using a service account? It's not available via the Cloud SDK yet but you can try googleapis.com/storage/v1/b/prerelease/o/… in the meantime. - Travis Hobrla
I think what's happening here is that gcloud auth login gives you a token for the gcloud client ID, but you are using a standalone version (even like the prerelease I linked earlier) that uses the gsutil client ID. When your access token needs refreshing, this fails. The solution is to use the gcloud-distributed gsutil only, or to uninstall gcloud and run gsutil config with the standalone gsutil version. This is definitely painful and something the gsutil devs want to improve soon. - Travis Hobrla

2 Answers

3
votes

It was a weird case.

The file was on windows server under "D:" drive and I was running gsutil tool from there.

i.e. D:>gsutil -m cp xyz.csv gs:\somebucket\

I didn't have enough permission on D: drive

But as soon as I run the same command from "C:" drive it worked fine

i.e C:>gsutil -m cp "D:\xyz.csv" gs:\somebucket\

0
votes

This may be an issue with how gsutil/boto handles the OS path separators on Windows, as referenced here. This should eventually get merged into the sdk tools, but until then the following should work:

Go to
google-cloud-sdk\platform\gsutil\third_party\boto\boto\pyami\config.py

and replace the line:

for path in os.environ['BOTO_PATH'].split(':'):

with:

for path in os.environ['BOTO_PATH'].split(os.path.pathsep):

Next, go to
google-cloud-sdk\bin\bootstrapping\gsutil.py

replace the lines that use ':'

if boto_config:
      boto_path = ':'.join([boto_config, gsutil_path])
    elif boto_path:
      # this is ':' for windows as well, hardcoded into the boto source.
      boto_path = ':'.join([boto_path, gsutil_path])
    else:
      path_parts = ['/etc/boto.cfg',
                    os.path.expanduser(os.path.join('~', '.boto')),
                    gsutil_path]
      boto_path = ':'.join(path_parts)

with

if boto_config:
      boto_path = os.path.pathsep.join([boto_config, gsutil_path])
    elif boto_path:
      # this is ':' for windows as well, hardcoded into the boto source.
      boto_path = os.path.pathsep.join([boto_path, gsutil_path])
    else:
      path_parts = ['/etc/boto.cfg',
                    os.path.expanduser(os.path.join('~', '.boto')),
                    gsutil_path]
      boto_path = os.path.pathsep.join(path_parts)

Reload cmd and the error should go away.