Aws lambda audio features extraction ( Not enough storage -Layers )

2
votes

We have IOT sensors that uploads wav files into S3 Bucket.

We want to be able to extract sound features from each file that is getting uploaded (create obj event) with aws lambda

For that we need:

  • python librosa or pyAudio analysis package + numpy and scipy. (~ 240mb unzziped)
  • ffmpeg (~ 70mb unzziped)

As you can see there is no way to put them all together in same lambda package (250mb uncompressed max). And im getting an error when not including the ffmpeg in the layers when gathering the wav file: 

[ERROR] FileNotFoundError: [Errno 2] No such file or directory: 'ffprobe': 'ffprobe'

which is related to ffmpeg.

We are looking for implementation recommendation, we thought about:

  1. Putting the ffmpeg file in s3 and getting it every single invoke ( without having to put it in the layers. ( if it is even possible)

  2. Chaining two lambdas: 1 for processing the input file through ffmpeg and puting the output file in abother bucket > 2 function invoked and extracting features from the processed data. ( using SNS / chaining mechanism) ( if it is even possible)

  3. Move to EC2 where there we will have a problem with concurrent invokation accuring when two files uploads at the same time.

there has to be and easier way, ill be glad to hear for other opinions before diving into implementation, Thank you all!

1
amazon-web-servicesaudioffmpegaws-lambdalayer
Do you really need the files to be processed immediately? Is there a problem if one file is delayed while another is being processed? Are you also likely to be hitting the 15-minute Lambda duration limit anyway, for large audio files? - John Rotenstein
Nope, I don't need a real-time processing .delay can be even few hours. About the native delay-the file is only 10 seconds, the feature extraction is pretty basic. What do you recommend mate? - Amit Neuhaus

1 Answers

2
votes

The scenario appears to be:

  • Files come in at random times
  • The files need to be processed, but not in real-time
  • The required libraries are too big for an AWS Lambda function

Suggested architecture:

  • Configure an Amazon S3 Event to send a message to an Amazon SQS queue when a file arrives
  • Configure an Amazon CloudWatch Event to trigger an AWS Lambda function at regular intervals (eg 1 hour)
    • The Lambda function checks whether there are messages in the queue
    • If there are messages, it launches an Amazon EC2 instance with a User Data script that installs and starts the processing system
  • The processing system will:
    • Grab a message from the queue
    • Process the message (without the limitations of Lambda)
    • Delete the message
    • If there are no messages left in the queue, it will terminate the EC2 instance

This can be very cost-effective because Amazon EC2 Linux instances are charged per-second. You can run several workers in parallel to process the messages (but be careful when writing the termination code, to ensure that all workers have finished processing messages). Or, if things are not time-critical, just choose the smallest usable Instance Type and single-thread it since larger instances cost more anyway (so they are no better from a cost-efficient standpoint).

Make sure you put monitoring in place to ensure that messages are being processed. Implement a Dead Letter Queue in Amazon SQS to catch messages that are failing to process and put a CloudWatch Alarm on the DLQ to notify you if things seem to be going wrong.