How to access Google Cloud Storage bucket using aws-cli

Question

I have an access with both aws and Google Cloud Platform.

Is this possible to do the following,

1. List Google Cloud Storage bucket using aws-cli
2. PUT a CSV file to Google Cloud Storage bucket using aws-cli
3. GET an object(s) from Google Cloud Storage bucket using aws-cli

Answer 1

It is possible. Per GCP documentation

The Cloud Storage XML API is interoperable with ... services such as Amazon Simple Storage Service (Amazon S3)

To do this you need to enable Interoperability in the Settings screen in the Google Cloud Storage console. From there you can creates a storage access key.

Configure the aws cli with those keys. IE aws configure.

You can then use the aws s3 command with the --endpoint-url flag set to https://storage.googleapis.com.

For example:

MacBook-Pro:~$ aws s3 --endpoint-url https://storage.googleapis.com ls
2018-02-09 14:43:42 foo.appspot.com
2018-02-09 14:43:42 bar.appspot.com
2018-05-02 20:03:08 etc.appspot.com

 aws s3 --endpoint-url https://storage.googleapis.com cp test.md s3://foo.appspot.com
 upload: ./test.md to s3://foo.appspot.com/test.md  

Answer 2

I had a requirement to copy objects from GC storage bucket to S3 using AWS Lambda.

Python boto3 library allows listing and downloading objects from GC bucket.

Below is sample lambda code to copy "sample-data-s3.csv" object from GC bucket to s3 bucket.

import boto3
import io

s3 = boto3.resource('s3')

google_access_key_id="GOOG1EIxxMYKEYxxMQ"
google_access_key_secret="QifDxxMYSECRETKEYxxVU1oad1b"

gc_bucket_name="my_gc_bucket"


def get_gcs_objects(google_access_key_id, google_access_key_secret,
                     gc_bucket_name):
    """Gets GCS objects using boto3 SDK"""
    client = boto3.client("s3", region_name="auto",
                          endpoint_url="https://storage.googleapis.com",
                          aws_access_key_id=google_access_key_id,
                          aws_secret_access_key=google_access_key_secret)

    # Call GCS to list objects in gc_bucket_name
    response = client.list_objects(Bucket=gc_bucket_name)

    # Print object names
    print("Objects:")
    for blob in response["Contents"]:
        print(blob)    

    object = s3.Object('my_aws_s3_bucket', 'sample-data-s3.csv')
    f = io.BytesIO()
    client.download_fileobj(gc_bucket_name,"sample-data.csv",f)
    object.put(Body=f.getvalue())

def lambda_handler(event, context):
    get_gcs_objects(google_access_key_id,google_access_key_secret,gc_bucket_name) 

You can loop through blob to download all objects from GC bucket.

Hope this helps someone who wants to use AWS lambda to transfer objects from GC bucket to s3 bucket.

Answer 3

~$ aws configure
AWS Access Key ID [****************2ZL8]:
AWS Secret Access Key [****************obYP]:
Default region name [None]: us-east-1
Default output format [None]:
~$aws s3 ls --endpoint-url=<east-region-url>
2019-02-18 12:18:05 test
~$aws s3 cp test.py s3://<bucket-name>  --endpoint-url=<east-region-url>
~$aws s3 mv s3://<bucket-name>/<filename> test1.txt --endpoint-url=<east-region-url>

Answer 4

Unfortunately this is not possible,

Could you maybe update your question to why you want to do this, maybe we know of an alternative solution to your question?