Package dlib in AWS Lambda using Serverless and python 3.7

0
votes

I am trying to deploy an AWS Lambda function using Serverless framework with python3.7 runtime and I need to package dlib as a dependency and import inside the lambda function. Any ideas what's the easiest way to make dlib work on python3.7 and package successfully using the Serverless framework? TIA...

UPDATE: I already added dlib==19.9.0 to my requirements.txt, I am using the Serverless plugin serverless-python-requirements and I deploy using bitbucket pipeline based on an ubuntu image with python & node base. I also install cmake in the pipeline script since dlib needs it to compile as I've seen.

The pipeline fails with:

Container 'Build' exceeded memory limit.

and the dlib compiling stops at 77% with the following recurring deprecation warnings until the pipeline exceeds memory and stops with failure:

 Scanning dependencies of target dlib
.
.

 [ 77%] Building CXX object CMakeFiles/dlib_python.dir/src/other.cpp.o
    In file included from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/cast.h:16,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/attr.h:13,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/pybind11.h:43,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/../dlib/python/pybind_utils.h:6,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/../dlib/python.h:6,
                     from /tmp/pip-install-nx1hok9_/dlib/tools/python/src/basic.cpp:3:
    /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/detail/internals.h:82:34: warning: ‘int PyThread_create_key()’ is deprecated [-Wdeprecated-declarations]
         decltype(PyThread_create_key()) tstate = 0; // Usually an int but a long on Cygwin64 with Python 3.x
                                      ^
    In file included from /usr/local/include/python3.7m/pystate.h:11,
                     from /usr/local/include/python3.7m/traceback.h:8,
                     from /usr/local/include/python3.7m/Python.h:119,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/detail/common.h:111,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/pytypes.h:12,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/cast.h:13,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/attr.h:13,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/pybind11.h:43,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/../dlib/python/pybind_utils.h:6,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/../dlib/python.h:6,
                     from /tmp/pip-install-nx1hok9_/dlib/tools/python/src/basic.cpp:3:
    /usr/local/include/python3.7m/pythread.h:95:17: note: declared here
     PyAPI_FUNC(int) PyThread_create_key(void) Py_DEPRECATED(3.7);
                     ^~~~~~~~~~~~~~~~~~~
    In file included from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/cast.h:16,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/attr.h:13,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/pybind11.h:43,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/../dlib/python/pybind_utils.h:6,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/../dlib/python.h:6,
                     from /tmp/pip-install-nx1hok9_/dlib/tools/python/src/basic.cpp:3:
    /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/detail/internals.h:82:34: warning: ‘int PyThread_create_key()’ is deprecated [-Wdeprecated-declarations]
         decltype(PyThread_create_key()) tstate = 0; // Usually an int but a long on Cygwin64 with Python 3.x
                                      ^
    In file included from /usr/local/include/python3.7m/pystate.h:11,

enter code here

2
amazon-web-servicesaws-lambdapython-3.7serverless-frameworkdlib
Don't know about serverless framework, but I usually create lambda layers with my dependencies. For that i use the lambci/lambda docker tool. - Marcin

2 Answers

1
votes

Marcin's suggestion works but is somewhat tedious. Fortunately, here comes the serverless framework to the rescue. The following example uses Python 3.8, but can easily switch it to 3.7.

Prerequisites:

serverless.yml

service: dlib-example

provider:
  name: aws
  runtime: python3.8

functions:
  dlib:
    handler: handler.main
    layers:
      - {Ref: PythonRequirementsLambdaLayer}

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: non-linux
    layer: true

requirements.txt

dlib==19.19.0

handler.py

import dlib


def main(event, context):
    print(dlib.__version__)


if __name__ == "__main__":
    main('', '')

Then use sls deploy to build dependencies (in docker container) and deploy to AWS using CloudFormation.

Test

Run sls invoke -f dlib --log and you will get something like this:

null
--------------------------------------------------------------------
START RequestId: 9fba7253-2f3b-425f-a0b7-9ee3dfaec13b Version: $LATEST
19.19.0
END RequestId: 9fba7253-2f3b-425f-a0b7-9ee3dfaec13b
REPORT RequestId: 9fba7253-2f3b-425f-a0b7-9ee3dfaec13b  Duration: 1.66 ms   Billed Duration: 100 ms Memory Size: 1024 MB    Max Memory Used: 62 MB  Init Duration: 227.31 ms

Cheers!

0
votes

ok I solved this by increasing the pipeline size to 2x and it succeeded