I need to get an object from an S3 bucket using the GetObject
family of functions from the Go AWS SDK, where the object’s key possibly starts with one or more slashes. However, it seems the SDK removes those leading slashes, thus changing the key.
I created the bucket and put some data in like this:
$ aws s3 mb <TEST BUCKET>
$ aws s3 cp <SOME FILE> s3://<TEST BUCKET>//leadingslash
The following code shows that ListObjects
correctly return the key with a leading slash, however when run the log shows the GET
request is done without the leading slash.
package main
import (
"log"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
func main() {
bucket := "<TEST BUCKET>"
region := "<TEST BUCKET REGION>"
config := (&aws.Config{Region: ®ion}).WithLogLevel(aws.LogDebugWithHTTPBody)
s3svc := s3.New(session.New(config))
listInput := s3.ListObjectsInput{
Bucket: &bucket,
}
listOutput, err := s3svc.ListObjects(&listInput)
if err != nil {
log.Fatalf("Failed to list objects: %v", err)
} else {
log.Printf("Good: %v", listOutput)
}
for _, object := range listOutput.Contents {
getInput := s3.GetObjectInput{
Bucket: &bucket,
Key: object.Key,
}
getOutput, err := s3svc.GetObject(&getInput)
if err != nil {
log.Fatalf("Failed to HEAD object: %v", err)
} else {
log.Printf("Good: %v", getOutput)
}
}
}
The call to GetObject
with debug logging enabled shows that the SDK performs the following request:
GET /leadingslash HTTP/1.1
This lacks the leading slash and returns a 404 error.
How should I go about getting such objects with the Go SDK ? I do not have control over the objects’ keys.
I have tried URL-escaping the key before passing it to GetObject
but the percent signs get escaped and the key changes.
I use Go 1.9 linux/amd64 and SDK 1.12.62.
/
in the URL is in the position before the first character in the object key, not the first character of the key itself (much like a file in the root directory of a server is actually namedfoo
rather than/foo
, the former being the name, the latter being the path)... but most of the SDKs have unfortunately masked this by swallowing the leading slash when developers incorrectly supply it. You may have to hack the SDK or roll your own signing code, if you can't get the erroneous incorrect keys corrected. – Michael - sqlbot