I want to download a large dataset from URL "https" directly to google cloud storage(Bucket).
I have used gsutil stream command. via google cloud shell
curl -L https://../../..velodyne.zip | gsutil cp - gs://###/##/velodyne.zip
the result is
We've detected excessive egress network traffic in your Google Cloud Shell session. Discontinue or decrease this traffic or your session will be terminated.
This is because you're downloading to your local environment (the cloud shell) with curl and then sending to GCS.
The way to do this is to use the Storage Transfer Service.
There is one option of giving the transfer service a URL list, and the service itself will download all the data to the bucket.
It will download the file from your local machine then use that as input to your gsutil command. You won't be able to do it using Cloud Shell, you would probably hit a "No Space left on device" if your file is too large.
You also mentioned in comment to Jofre's post that you've tried the Storage Transfer Service but it failed. Please share the error so we can see how it failed then maybe we can help you.
