AWS Lambda (Python) Fails to unzip and store files in S3

Question

Project currently maintains S3 bucket which holds a large zip size 1.5 GB containing .xpt and .sas7dbat files. Unzipped file size is 20 GB.

Trying to unzip file and push the same folder structure to S3

Following code works for a small zip files but fails for large Zip file (1.5GB) :

for obj in bucket.objects.all():
    #file_name = os.path.abspath(obj.key) # get full path of files
    key = urlparse(obj.key.encode('utf8'))
    obj = client.get_object(Bucket='my-zip-bucket', Key=obj.key)

    with io.BytesIO(obj["Body"].read()) as tf:
        # rewind the file
        tf.seek(0)

        with zipfile.ZipFile(tf, mode='r') as zipf:
            for file in zipf.infolist():
                fileName = file.filename
                putFile = client.put_object(Bucket='my-un-zip-bucket-', Key=fileName, Body=zipf.read(file))
                putObjects.append(putFile)

Error : Memory Size: 3008 MB Max Memory Used: 3008 MB

I would like to validate :

AWS-Lambda is not a suitable solution for large files ?
Should I use different libraries / approach rather than reading everything in memory

Ganondorfz Ganondorfz · Accepted Answer · 2018-10-03T00:05:30

There is a serverless solution using AWS Glue! (I nearly died figuring this out)

This solution is two parts:

A lambda function that is triggered by S3 upon upload of a ZIP file and creates a GlueJobRun - passing the S3 Object key as an argument to Glue.
A Glue Job that unzips files (in memory!) and uploads back to S3.

See my code below which unzips the ZIP file and places the contents back into the same bucket (configurable).

Please upvote if helpful :)

Lambda Script (python3) that calls a Glue Job called YourGlueJob

import boto3
import urllib.parse

glue = boto3.client('glue')

def lambda_handler(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
    print(key)    
try:
    newJobRun = glue.start_job_run(
        JobName = 'YourGlueJob',
        Arguments = {
            '--bucket':bucket,
            '--key':key,
        }
        )
    print("Successfully created unzip job")    
    return key  
except Exception as e:
    print(e)
    print('Error starting unzip job for' + key)
    raise e

AWS Glue Job Script to unzip the files

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME','bucket','key'],)

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

import boto3
import zipfile
import io
from contextlib import closing

s3 = boto3.client('s3')
s3r = boto3.resource('s3')

bucket = args["bucket"]
key = args["key"]

obj = s3r.Object(
    bucket_name=bucket,
    key=key
)

buffer = io.BytesIO(obj.get()["Body"].read())
z = zipfile.ZipFile(buffer)
list = z.namelist()
for filerr in list:
    print(filerr)
    y=z.open(filerr)
    arcname = key + filerr
    x = io.BytesIO(y.read())
    s3.upload_fileobj(x, bucket, arcname)
    y.close()
print(list)


job.commit()

AWS Lambda (Python) Fails to unzip and store files in S3

3 Answers