Google cloud gsutil fails inconsistently due to crcmode not compiled

Question

I'm running my build with google cloudbuild and I have a step that copies cached files from a google storage bucket.
the step is using google's gsutil image from "gcr.io/cloud-builders/gsutil".
the build runs automatically on every "push" to the github repo.
this is the step:

steps:
  - name: 'gcr.io/cloud-builders/gsutil'
    id: 'Loading gradle cache'
    entrypoint: 'sh'
    args:
      - '-c'
      - |
        fileName=file.tar.gz
        if gsutil -q stat gs://cache_${PROJECT_ID}/${fileName}; then
          echo 'Gradle cache found'
          cd /
          gsutil -m cp gs://cache_${PROJECT_ID}/${fileName} ${fileName}
          tar -xpzf ${fileName}
        else
          echo 'Gradle cache not found'
        fi
    volumes:
      - name: 'gradle_cache'
        path: /root/.gradle
    waitFor: ['-']
    timeout: 60s

The problem is that sometime it passes and sometime it fails with the following due to crcmod not compiled:

CommandException:
Downloading this composite object requires integrity checking with CRC32c,
but your crcmod installation isn't using the module's C extension, so the
hash computation will likely throttle download performance. For help
installing the extension, please see "gsutil help crcmod".

To download regardless of crcmod performance or to skip slow integrity
checks, see the "check_hashes" option in your boto config file.
CommandException: 1 file/object could not be transferred.

This error started happening 4-5 days ago. any idea how to solve this?

Thanks

Mike Schwartz Mike Schwartz · Accepted Answer · 2020-07-13T00:12:34

See gsutil help crcmod. You need to install a compiled crcmod if you want to download composite objects. (It's also possible to disable checksumming instead, but that's a bad idea - you would not detect data corruption.)

Google cloud gsutil fails inconsistently due to crcmode not compiled

1 Answers