Copy files from remote server to google cloud bucket

5
votes

How would I copy files from a remote server to a google bucket? For example,

gcloud compute scp username@server:/path/to/file gs://my-bucket

This method gives the error: All sources must be local files when destination is remote.

Additionally, gsutil only provides support for cp and not scp.

Thanks in advance!

2
google-cloud-platformgoogle-cloud-storagegcloudgsutil

2 Answers

4
votes

You can also directly execute gsutil command on your GCE VM (Most VM images have Cloud SDK preinstalled). For example:

gcloud compute ssh user@server --zone my_zone \
  --command='gsutil cp path/to/my_file gs://MY_BUCKET'

Note that for this to work your service account associated with VM must have appropriate access scope to GCS. If you run

gcloud beta compute instances describe my_instance --zone my_zone \
  --format="value(serviceAccounts.scopes)"

It will show list of scopes set for VM service account. Make sure you have https://www.googleapis.com/auth/cloud-platform or https://www.googleapis.com/auth/devstorage.full_control or https://www.googleapis.com/auth/devstorage.read_write. If not you can use set-scopes beta command to reset them or go to console and edit VM in question.

-1
votes

gsutil supports streaming data, so you can try to pipe the data in.

Not 100% certain this works with arbitrary binary data (and I'm on a phone so I can't test it)

gcloud compute ssh username@server --command='cat /path/to/file' | gsutil cp - gs://my-bucket