Serverless issue with AWS Lambda and Open CV in Python

Question

I am developing a microservice to analyze an image uploaded to an S3 AWS Bucket. I am using Serverless framework. I am using virtualenv to install the dependencies with PIP and serverless-python-requirements plugin to deploy these dependencies to the Lambda function.

However I am having an error when I deploy the microservice because of a missing .so file. The error I get is

Unable to import module 'handlers.image': libgthread-2.0.so.0: cannot open shared object file: No such file

My serverless.yml file is

service: sls-covid

provider:
  name: aws
  runtime: python3.8
  profile: testuser
  stage: ${opt:stage, 'staging'}
  environment: ${file(environment.yml):${self:provider.stage}}
  region: ${self:provider.environment.REGION}

  iamRoleStatements:
    # para poder leer y escribir en el bucket
    - Effect: "Allow"
      Action:
        - "s3:*"
      Resource: "*"

custom:
  pythonRequirements:
    dockerizePip: non-linux


package:
  individually: true
  exclude:
    - node_modules/**
    - env/**
    - package.json
    - package-lock.json

functions:

  analyzeImage:
    # para analizar la imagen cuando se almacena en S3
    handler: handlers.image.analyze
    events:
      - s3:
          bucket: ${self:service}-${self:provider.stage}-images
          event: s3:ObjectCreated:*
          rules:
            - prefix: uploads/

plugins:
  - serverless-python-requirements

The relevant code is:

import json
import logging
import boto3
from pydicom import dcmread
import numpy as np
# here the code stops working
import cv2
from pydicom.filebase import DicomBytesIO

logger = logging.getLogger()
logger.setLevel(logging.INFO)

s3 = boto3.client('s3')


def analyze(event, context):

    print("=== event: ", event)

    # code to read uploaded image from bucket and create a ds object 

    # code below is never executed since the error appears when the cv2 library is loaded  

    img = cv2.resize(ds.pixel_array, (224, 224))

And my requirements.txt contents is as follows:

numpy==1.18.4
opencv-python==4.2.0.34
pandas==1.0.3
pydicom==1.4.2
python-dateutil==2.8.1
pytz==2018.7
six==1.14.0

My question is. How can I properly upload opencv dependencies to my lambda function using the plugin? Should I do this in another way?

Thanks!

jellycsc jellycsc · Accepted Answer · 2020-05-20T23:07:37

Based on the doc, you need to include those manually using dockerExtraFiles option.

======================= UPDATE =======================

Packaging Python dependencies and building deployment package for AWS Lambda can be quite tricky. After digging around, here are the things that you need to do to make it work.

Step 1: Add a `Dockerfile`

FROM lambci/lambda:build-python3.8
RUN yum -y install libXext libSM libXrender

Step 2: Modify `serverless.yml`

service: sls-covid

provider:
  name: aws
  runtime: python3.8
  profile: testuser
  stage: ${opt:stage, 'staging'}
  environment: ${file(environment.yml):${self:provider.stage}}
  region: ${self:provider.environment.REGION}

  iamRoleStatements:
    # para poder leer y escribir en el bucket
    - Effect: "Allow"
      Action:
        - "s3:*"
      Resource: "*"

custom:
  pythonRequirements:
    dockerizePip: non-linux
    ########### BEGIN ###########
    dockerFile: Dockerfile
    dockerExtraFiles:
      - /lib64/libgthread-2.0.so.0
      - /lib64/libglib-2.0.so.0
      - /lib64/libSM.so.6
      - /lib64/libICE.so.6
      - /lib64/libXrender.so.1
      - /lib64/libXext.so.6
      - /lib64/libX11.so.6
      - /lib64/libuuid.so.1
      - /lib64/libxcb.so.1
      - /lib64/libXau.so.6
    ########### END ###########


package:
  individually: true
  exclude:
    - node_modules/**
    - env/**
    - package.json
    - package-lock.json

functions:

  analyzeImage:
    # para analizar la imagen cuando se almacena en S3
    handler: handlers.image.analyze
    events:
      - s3:
          bucket: ${self:service}-${self:provider.stage}-images
          event: s3:ObjectCreated:*
          rules:
            - prefix: uploads/

plugins:
  - serverless-python-requirements

Step 3: Hello world testing

$ sls invoke -f analyzeImage --log
null
--------------------------------------------------------------------
START RequestId: 00ccd940-cf8a-46ed-8671-65e597f997a0 Version: $LATEST
=== event:  {}
END RequestId: 00ccd940-cf8a-46ed-8671-65e597f997a0
REPORT RequestId: 00ccd940-cf8a-46ed-8671-65e597f997a0  Duration: 1.41 ms   Billed Duration: 100 ms Memory Size: 1024 MB    Max Memory Used: 121 MB Init Duration: 916.32 ms

Hope it helps.

Serverless issue with AWS Lambda and Open CV in Python

2 Answers

Step 1: Add a Dockerfile

Step 2: Modify serverless.yml

Step 3: Hello world testing

Step 1: Add a `Dockerfile`

Step 2: Modify `serverless.yml`