Transfer From S3 to Google Storage - Incorrect Key

Question

I've been trying for past couple of hours to setup a transfer from S3 to my google storage bucket.

The error that i keep getting, when creating the transfer is: "Invalid access key. Make sure the access key for your S3 bucket is correct, or set the bucket permissions to Grant Everyone."

Both the access key and the secret are correct, given that they are currently in use in production for S3 full access.

Couple of things to note:

CORS-enabled on S3 bucket
Bucket policy only allows authenticated AWS users to list/view its contents
S3 requires signed URLs for access

Bucket Policy:

{
    "Version": "2008-10-17",
    "Id": "Policy234234234",
    "Statement": [
        {
            "Sid": "Stmt234234",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:GetObjectAcl",
                "s3:RestoreObject",
                "s3:GetObjectVersion",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:PutObjectVersionAcl",
                "s3:PutObjectAcl",
                "s3:GetObject",
                "s3:PutObject",
                "s3:GetObjectVersionAcl"
            ],
            "Resource": "arn:aws:s3:::mybucket/*"
        },
        {
            "Sid": "2",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity xyzmatey"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::mybucket/*"
        },
        {
            "Sid": "3",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::mybucket"
        }
    ]
}

CORS Policy

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>http://www.mywebsite.com</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedMethod>DELETE</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>*</AllowedHeader>
    </CORSRule>
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedHeader>AUTHORIZATION</AllowedHeader>
    </CORSRule>
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>HEAD</AllowedMethod>
        <AllowedHeader>AUTHORIZATION</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

Any idea where i have gone wrong?

EDIT: I've setup the gsutil tool on a google compute instance and did a copy with the same AWS keys on the exact bucket. Worked like a charm..

Your bucket policy doesn't include "s3:ListBucket". I am guessing that the transfer service might need that in order to get a list of objects to transfer. Try adding it to the list? Of course, that wouldn't explain how gsutil manages to copy the bucket, so that may be wrong. — Brandon Yarbrough
Hey brandon, i added the policy you mentioned above. Same results: Invalid key. Will see if i get a grasp on the support staff of google for this one. Thanks. — Mysteryos

Jacob Criner Jacob Criner · Accepted Answer · 2016-02-05T17:57:23

I'm one of the devs on Transfer Service.

You'll need to add "s3:GetBucketLocation" to your permissions.

It would be preferable if the error you received was more specifically about your ACLs, however, rather than an invalid key. I'll look into that.

EDIT: Adding more info to this post. There is documentation which lists this requirement: https://cloud.google.com/storage/transfer/

Here's a quote from the section on "Configuring Access":

"If your source data is an Amazon S3 bucket, then set up an AWS Identity and Access Management (IAM) user so that you give the user the ability to list the Amazon S3 bucket, get the location of the bucket, and read the objects in the bucket." [Emphasis mine.]

EDIT2: Much of the information provided in this answer could be useful for others, so it will remain here, but John's answer actually got to the bottom of OP's issue.

Transfer From S3 to Google Storage - Incorrect Key

5 Answers