listing objects in aws bucket

Question

I was trying to print all the objects in a bucket but I am getting an error.

Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 301, AWS Service: Amazon S3, AWS Request ID: 758A7CBF1A29FD74, AWS Error Code: PermanentRedirect, AWS Error Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint., S3

At the moment I only have the following code :

public class S3Download {

    /**
     * @param args
     */
    public static void main(String[] args) {
        AmazonS3 s3 = new AmazonS3Client(new ClasspathPropertiesFileCredentialsProvider());
        Region usWest2 = Region.getRegion(Regions.US_WEST_2);
        s3.setRegion(usWest2);
        String bucketName = "apireleasecandidate1";

        ListObjectsRequest listObjectRequest = new ListObjectsRequest().withBucketName(bucketName);
        ObjectListing objectListing;

        do{
            objectListing = s3.listObjects(listObjectRequest);
            for(S3ObjectSummary objectSummary : objectListing.getObjectSummaries()){
                System.out.println(" - " + objectSummary.getKey() + " " + "(size = " +
            objectSummary.getSize() + ")");
            }
            listObjectRequest.setMarker(objectListing.getNextMarker());
        }while(objectListing.isTruncated());
    }

}

I found this solution on amazon's website.

Does anyone know what I am missing?

pangiole pangiole · Accepted Answer · 2014-06-05T12:38:18

For Scala developers, here it is recursive function to execute a full scan and map of the contents of an AmazonS3 bucket using the official AWS SDK for Java

import com.amazonaws.services.s3.AmazonS3Client
import com.amazonaws.services.s3.model.{S3ObjectSummary, ObjectListing, GetObjectRequest}
import scala.collection.JavaConversions.{collectionAsScalaIterable => asScala}

def map[T](s3: AmazonS3Client, bucket: String, prefix: String)(f: (S3ObjectSummary) => T) = {

  def scan(acc:List[T], listing:ObjectListing): List[T] = {
    val summaries = asScala[S3ObjectSummary](listing.getObjectSummaries())
    val mapped = (for (summary <- summaries) yield f(summary)).toList

    if (!listing.isTruncated) mapped.toList
    else scan(acc ::: mapped, s3.listNextBatchOfObjects(listing))
  }

  scan(List(), s3.listObjects(bucket, prefix))
}

To invoke the above curried map() function, simply pass the already constructed (and properly initialized) AmazonS3Client object (refer to the official AWS SDK for Java API Reference), the bucket name and the prefix name in the first parameter list. Also pass the function f() you want to apply to map each object summary in the second parameter list.

For example

map(s3, bucket, prefix)(s => println(s))

will print all the files

val tuple = map(s3, bucket, prefix)(s => (s.getKey, s.getOwner, s.getSize))

will return the full list of (key, owner, size) tuples in that bucket/prefix

val totalSize = map(s3, "bucket", "prefix")(s => s.getSize).sum

will return the total size of its content (note the additional sum() folding function applied at the end of the expression ;-)

You can combine map() with many other functions as you would normally approach by Monads in Functional Programming

listing objects in aws bucket

2 Answers