Copy Files from S3 bucket to Google Cloud Storage

0
votes

Trying to pull the data from AWS S3 to Google Cloud Storage by using Storage Transfer.

I have S3 bucket "Access key ID" and "Secret access key" and able to copy from gsutil (from laptop) but it is throwing permission error from Storage transfer.

Not understand how it is working with gsutil but not with storage transfer.

Error Message:

Invalid access key. Make sure the access key for your S3 bucket is correct, or set the bucket permissions to Grant Everyone.

Is there any way to use gsutil in compute engine to copy files from S3 to GCS?

Thanks,

1
amazon-s3google-cloud-storage
Storage transfer should work. Have you considered enabling S3 bucket logs and CloudTrail to see if any of the requests are arriving? And of course verified that your access key and secret have no copy/paste errors? – Michael - sqlbot
I am not the owner of the S3 bucket to enable S3 bucket logs. I checked with S3 bucket owner and he confirmed that S3 bucket have proper permissions and I am able to copy with gsutil by adding key and secret in boto file. Key and Secret are correct, I cross checked. – user374374
There may be a permission that storage transfer needs that gsutil doesn't need, such as s3:GetObjectAcl. (That's a guess.) Without access to the user and bucket policy, it's tricky to troubleshoot. But, yes, you can use gsutil in Google Compute Engine, or in EC2, or anywhere you have a server with bandwidth. It won't make any difference where you run it, as long as it is geographically close to both buckets. – Michael - sqlbot
Thanks Michael. In compute engine where I need to pass S3 bucket Key and Secrete information, I couldn't find boto file. When I tested in windows laptop I entered key and secrete key information in boto file and tested. – user374374

1 Answers

1
votes

Created .boto file in home directory with the following lines. gsutil is able to connect to aws S3 bucket. 

[Credentials]
aws_access_key_id = <key>
aws_secret_access_key = <secret key>

Thanks,