Iterating over Objects in an AWS S3 Bucket

Question

The script I have is working, but the bucket I am scanning over is massive and times out after a while. What can I do to make this more efficient or start from a specific location?

import boto3

s3 = boto3.resource('s3')
b = s3.Bucket('my_bucket')

for obj in b.objects.all():
   # Open the file, run some RegEx to find some data. If it's found, output to a log file

The first problem I have is the size of the bucket. It's about 1.5 million objects. I have my code opening up text files looking for some RegEx and if there's a match on the RegEx then it outputs the Object name and what was found.

After running the script for about an hour, it makes it about 40k objects in before throwing an error:

requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))

or

object at 0x109e82d50>: Failed to establish a new connection: [Errno 60] Operation timed out',))

The search items it's searching through are alphabetical, so we'll say it makes it through the "E" section and then times out. I want start with objects starting with "F".

Some performance considerations: docs.aws.amazon.com/AmazonS3/latest/dev/… — jarmod
Have you tried Amazon Athena? It is intended to quickly analyze large-scale datasets on S3: aws.amazon.com/athena — Khalid T.

John Rotenstein John Rotenstein · Accepted Answer · 2017-01-12T21:37:54

If you have a large number of objects in your Amazon S3 bucket, then objects.all() is not an efficient iteration method, since it tries to load them all into memory simultaneously.

Instead, use list_objects_v2() to page through the objects in groups of 1000. Then, call it again with the ContinuationToken that was returned.

You will effectively need a for loop calling list_objects_v2() and another for loop within that which loops through each object.

Iterating over Objects in an AWS S3 Bucket

2 Answers