How to use GetObject when key has leading slashes

Question

I need to get an object from an S3 bucket using the GetObject family of functions from the Go AWS SDK, where the object’s key possibly starts with one or more slashes. However, it seems the SDK removes those leading slashes, thus changing the key.

I created the bucket and put some data in like this:

$ aws s3 mb <TEST BUCKET>
$ aws s3 cp <SOME FILE> s3://<TEST BUCKET>//leadingslash

The following code shows that ListObjects correctly return the key with a leading slash, however when run the log shows the GET request is done without the leading slash.

package main

import (
    "log"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3"
)

func main() {
    bucket := "<TEST BUCKET>"
    region := "<TEST BUCKET REGION>"
    config := (&aws.Config{Region: &region}).WithLogLevel(aws.LogDebugWithHTTPBody)
    s3svc := s3.New(session.New(config))

    listInput := s3.ListObjectsInput{
        Bucket: &bucket,
    }
    listOutput, err := s3svc.ListObjects(&listInput)
    if err != nil {
        log.Fatalf("Failed to list objects: %v", err)
    } else {
        log.Printf("Good: %v", listOutput)
    }

    for _, object := range listOutput.Contents {
        getInput := s3.GetObjectInput{
            Bucket: &bucket,
            Key:    object.Key,
        }
        getOutput, err := s3svc.GetObject(&getInput)
        if err != nil {
            log.Fatalf("Failed to HEAD object: %v", err)
        } else {
            log.Printf("Good: %v", getOutput)
        }
    }
}

The call to GetObject with debug logging enabled shows that the SDK performs the following request:

GET /leadingslash HTTP/1.1

This lacks the leading slash and returns a 404 error.

How should I go about getting such objects with the Go SDK ? I do not have control over the objects’ keys.

I have tried URL-escaping the key before passing it to GetObject but the percent signs get escaped and the key changes.

I use Go 1.9 linux/amd64 and SDK 1.12.62.

Leading slashes in the keys is fundamentally incorrect, because conceptually the / in the URL is in the position before the first character in the object key, not the first character of the key itself (much like a file in the root directory of a server is actually named foo rather than /foo, the former being the name, the latter being the path)... but most of the SDKs have unfortunately masked this by swallowing the leading slash when developers incorrectly supply it. You may have to hack the SDK or roll your own signing code, if you can't get the erroneous incorrect keys corrected. — Michael - sqlbot

Victor Schubert Victor Schubert · Accepted Answer · 2018-01-24T16:27:07

Following @michael-sqlbot’s advice, this was solved by using custom logic to build the request :

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "time"

    "github.com/aws/aws-sdk-go/aws/credentials"
    "github.com/aws/aws-sdk-go/aws/signer/v4"
    "github.com/aws/aws-sdk-go/private/protocol/rest"
)

const (
    BUCKET = "<TEST BUCKET>"
    KEY    = "/leadingslashkey"
    REGION = "<TEST BUCKET REGION>"
)

func main() {
    credentials := credentials.NewEnvCredentials()
    signer := v4.NewSigner(credentials)
    request, err := http.NewRequest(
        http.MethodGet,
        fmt.Sprintf("https://%s.s3.%s.amazonaws.com/%s", BUCKET, REGION, rest.EscapePath(KEY, false)),
        nil,
    )
    if err != nil {
        println(err.Error())
        return
    }
    header, err := signer.Sign(
        request,
        nil,
        "s3",
        REGION,
        time.Now(),
    )
    if err != nil {
        println(err, err.Error())
        return
    }
    fmt.Printf("%#v\n", header)
    fmt.Printf("%#v\n", request)
    client := http.Client{}
    response, err := client.Do(request)
    if err != nil {
        println(err, err.Error())
        return
    }
    fmt.Printf("%#v\n", response)
    out, _ := ioutil.ReadAll(response.Body)
    println(string(out))
}

How to use GetObject when key has leading slashes

2 Answers