How do I set the S3 Request Payer option when using the aws-sdk-v2?

Question

I am trying to use the aws-sdk-go-v2 to retrieve some data from an S3 bucket. In order to do so I need to be able to set the Request Payer option, however, since I am new to using the SDK, I have no idea how to do so.

I've tried setting this as an env variable AWS_REQUEST_PAYER=requester, but scanning the source code for this golang SDK quickly, I couldn't find that it would be picked up by the SDK as an option.

Using the SDK as directed also fails with an Unauthorized response:

import (
    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/feature/s3/manager"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

type awsArchive struct {
    bucket string
    config config.Config
    client *s3.Client
}

func (s *awsArchive) download(uri string, dest string) error {

    downloader := manager.NewDownloader(s.client)
    paginator := s3.NewListObjectsV2Paginator(s.client, &s3.ListObjectsV2Input{
        Bucket: &s.bucket,
        Prefix: &uri,
    })

    for paginator.HasMorePages() {
        page, err := paginator.NextPage(context.TODO())
        if err != nil {
            return err
        }
        for _, obj := range page.Contents {
            if err := downloadToFile(downloader, dest, s.bucket, aws.ToString(obj.Key)); err != nil {
                return err
            }
        }
    }
    return nil
}

func downloadToFile(downloader *manager.Downloader, targetDirectory, bucket, key string) error {
    // Create the directories in the path
    file := filepath.Join(targetDirectory, key)
    if err := os.MkdirAll(filepath.Dir(file), 0775); err != nil {
        return err
    }

    // Set up the local file
    fd, err := os.Create(file)
    if err != nil {
        return err
    }
    defer fd.Close()

    // Download the file using the AWS SDK for Go
    fmt.Printf("Downloading s3://%s/%s to %s...\n", bucket, key, file)
    _, err = downloader.Download(context.TODO(), fd, &s3.GetObjectInput{Bucket: &bucket, Key: &key})

    return err
}

Error: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: ..., HostID: ..., api error AccessDenied: Access Denied

Would anyone be able to provide me with an example of using the Golang SDK to get S3 files from a Requestor Pays enabled bucket please, i.e. the equivalent of:

aws s3 sync --request-payer requester source_bucket destination_folder

augustkang augustkang · Accepted Answer · 2021-05-03T03:47:03

It seems you can use field named 'RequestPayer' in the GetObjectInput struct. Found it from pkg document.

From the link:

type GetObjectInput struct {

    // The bucket name containing the object. When using this action with an access
    ...
    Bucket *string
    // Key of the object to get.
    // This member is required.
    Key *string
    ...
    ...
    // Confirms that the requester knows that they will be charged for the request.
    // Bucket owners need not specify this parameter in their requests. For information
    // about downloading objects from requester pays buckets, see Downloading Objects
    // in Requestor Pays Buckets
    // (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html)
    // in the Amazon S3 Developer Guide.
    RequestPayer types.RequestPayer

You can refer to 'RequestPayer' definition here.

How do I set the S3 Request Payer option when using the aws-sdk-v2?

1 Answers