Native compile python libraries for serverless with serverless-python-requirements

Question

I'm trying to compile numpy for use in AWS Lambda functions from my windows machine.

I've been following this simple example of serverless-python-requirements for a lambda to print a numpy array.

I run serverless deploy from within a virtualenv in my local directory ../numpy-test

(numpy-test) C:\Users\...\numpy-test>serverless deploy
Serverless: Generated requirements from C:\Users\...\numpy-test\requirements.txt in C:\Users\...\numpy-test\.serverless\requirements.txt...
Serverless: Installing requirements from C:\Users\...\numpy-test\.serverless\requirements\requirements.txt ...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Injecting required Python packages to package...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (42.86 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...

My requirements.txt file simply contains

numpy==1.13.1

and my YML file looks like:

service: numpy-test 

frameworkVersion: ">=1.1.0 <2.0.0"

plugins:
  - serverless-domain-manager
  - serverless-python-requirements
custom:
  pythonRequirements:
    dockerizePip:non-linux

  stage: ${opt:stage, self:provider.stage}
  domains:
    prod: ...
    staging: ...
    dev: ...

When I attempt to test the function after upload with serverless invoke -f numpy --log I get the following error

    "errorMessage": "Unable to import module 'handler'"
}
--------------------------------------------------------------------
START RequestId: 24584f5c-5145-4694-b21b-7c6e4700f985 Version: $LATEST
Unable to import module 'handler':
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

Original error was: cannot import name 'multiarray'

When I download the dependencies directly from AWS lambda, it looks like the package is compiled in windows.

I have Docker installed and as far as I can tell it's working properly. I haven't used it before so that's why I'm trying to use this solution as opposed to learning a lot of new commands in Docker.

What is going wrong?

Milan Cermak Milan Cermak · Accepted Answer · 2019-02-04T13:25:17

I'm not sure what exactly is going on in your case, but if you want to use numpy in Lambda, have a look at Lambda Layers and the official (AWS) layer with numpy and scipy.

To use it, add a layers section to the function in your serverless.yml

functions:
  myFunction:
    layers:
      - arn:aws:lambda:us-east-1:668099181075:layer:AWSLambda-Python36-SciPy1x:2

You might need to change the ARN depending on the AWS region you're targeting.

Native compile python libraries for serverless with serverless-python-requirements

1 Answers