gsutil copy to storage failing

5
votes

i'm working in a instance at us-central1-a zone and I can't copy a ~200GB file.

i've tried :

  • gsutil -m cp -L my.log my.file gs://my-bucket/
  • gsutil -m cp -L my.second.log my.file gs://my-bucket2/

And after several "catch ups" I get the following error:

CommandException: Some temporary components were not uploaded successfully. Please retry this upload. CommandException: X files/objects could not be transferred.

Any clues?

Thanks

2
google-cloud-storagegoogle-compute-engine

2 Answers

4
votes

This is a message you'll see if gsutil's parallel composite uploads feature fails to upload at least one of the pieces of the file.

A couple of questions...

  1. Have you already tried performing this upload again, after you saw this message?

  2. If this error persists, could you please provide the stack trace from gsutil -d cp...

If you're consistently seeing this error and need an immediate fix (if this is a bug with parallel uploads), you can set parallel_composite_upload_threshold=0 in the GSUtil section your boto config to disable parallel uploads.

1
votes

I had the same experience using gsutil. I fixed by installing the crcmod.

First run the command you have issues with using the debug flag, for example:

gsutil -d -m cp gs://<path_to_file_in_bucket>

In the output I can see:

CommandException: Downloading this composite object requires integrity checking with CRC32c, but your crcmod installation isn't using the module's C extension, so the hash computation will likely throttle download performance. For help installing the extension, please see "gsutil help crcmod".

To download regardless of crcmod performance or to skip slow integrity checks, see the "check_hashes" option in your boto config file.

NOTE: It is strongly recommended that you not disable integrity checks. Doing so could allow data corruption to go undetected during uploading/downloading.

You can follow the instructions here from google to install crcmod for your specific os: https://cloud.google.com/storage/docs/gsutil/addlhelp/CRC32CandInstallingcrcmod