GSutil RSync - Change Max File path limit to match Windows

1
votes

I have a large project that I need to backup using GSutil. I keep getting "Error 3" with files nearing the windows file path limit. Is there anyway to get this working?

I Tried recreating the issue by syncing a diffrent local file with a long path to a diffrent folder. The same error occured.

(console output) https://i.imgur.com/pV8Rj4r.png

I am running Windows 10. The command i ran is : gsutil -m rsync -r -d -n Test Target

This is the path which failed:

C:\Users\temp\Desktop\Test\TestPath\LongTestTestTestTestTest Test - LongTestTestTestTestTest Test - LongTestTestTestTestTest Test - LongTestTestTestTestTest Test -\LongTestTestTestTestTest Test\LongTestTestTestTes\SO MUCH LONGER PATH 2\A BIT LONGER AGAIN\LONG FILE NAME,2_1225688.txt

(windows path) https://i.imgur.com/LO2d9UL.png

The issue I have is Windows allows these files to exist but gsutil is unable to sync them. If I shorten the path by a few characters the issue goes away. The Issue occurs during the listing phase It occurs both in the soruce and the target folder.

1
google-cloud-storagegsutil
It's hard to diagnose without the more specific / detailed error message. If you want to avoid posting that info on the public forum you could try sending your question to [email protected]. - Mike Schwartz
@MikeSchwartz Thanks for the reply! I updated the post with relevant information. - Narovana
That filename is shorter than the max allowed GCS object name, and all the characters are allowed. Are you sure that's the full windows path, and not just the final pathname component? - Mike Schwartz
@MikeSchwartz , Ah yes the path i posted was relative to the desktop, I edited in the full path aswell. - Narovana
If I'm not mistaken, Error 3 means path not found. Are you sure the path you're using with gsutil is the same path as you're using with whatever Windows command you used to look at the file? I'm wondering if it's problem with a typo in the path name. - Mike Schwartz

1 Answers

1
votes

I found a useful discussion on Serverfault.com and this simple solution worked on my Windows Server 2016:

Replace

gsutil -m rsync -r C:\ gs://...

by

gsutil -m rsync -r \\?\C:\ gs://...