Checksums with gsutil rsync

1
votes

I am downloading a large number of public data files from Google Cloud Storage using gsutil rsync. Occasionally the download fails for a few files. To ensure that I have all of the requested files, I run gsutil a second time with checksums turned on. During the second run, gsutil reports that it is computing checksums for fewer files than were downloaded. I have attached some sample output below. In this case it downloaded 29 files during the first rsync, but only reported that it was computing checksums for 16 files during the second rsync.

Is gsutil not computing the checksums and doing the rsync for some of the files, or is it simply not reporting that it is doing the checksums?

Ken

mix> gsutil -m rsync -R -P gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX /csrpc1/NEXRAD/level2/2017/201702/20170201/KHGX
Building synchronization state...
Starting synchronization
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131210000_20170131215959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131190000_20170131195959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131230000_20170131235959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131200000_20170131205959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131220000_20170131225959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201040000_20170201045959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201090000_20170201095959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201020000_20170201025959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201010000_20170201015959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201030000_20170201035959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201180000_20170201185959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201130000_20170201135959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201060000_20170201065959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201100000_20170201105959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201110000_20170201115959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201120000_20170201125959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201170000_20170201175959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201150000_20170201155959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201160000_20170201165959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201050000_20170201055959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201000000_20170201005959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201080000_20170201085959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201070000_20170201075959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201140000_20170201145959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201200000_20170201205959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201190000_20170201195959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201210000_20170201215959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201220000_20170201225959.tar...
Copying gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201230000_20170201235959.tar...
- [29/29 files][387.3 MiB/387.3 MiB] 100% Done                                  
Operation completed over 29 objects/387.3 MiB.                                   
mix> gsutil -m rsync -R -P -c gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX /csrpc1/NEXRAD/level2/2017/201702/20170201/KHGX
Building synchronization state...
Starting synchronization
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131190000_20170131195959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131200000_20170131205959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131210000_20170131215959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170131220000_20170131225959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201000000_20170201005959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201010000_20170201015959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201020000_20170201025959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201030000_20170201035959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201040000_20170201045959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201050000_20170201055959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201160000_20170201165959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201170000_20170201175959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201180000_20170201185959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201190000_20170201195959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201200000_20170201205959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201210000_20170201215959.tar...
Computing CRC32C for gs://gcp-public-data-nexrad-l2/2017/02/01/KHGX/NWS_NEXRAD_NXL2DPBL_KHGX_20170201210000_20170201215959.tar...
mix>
1
google-cloud-storagegsutil
Please shorten your log to the relevant lines. This screenful of output seems very repetitive to me - rollstuhlfahrer

1 Answers

2
votes

gsutil rsync -c only computes checksums if the size of the source and destination match. This saves time because there's no need to compute the checksum to determine the size-matching files need to be downloaded.