Numpy lambda error using serverless

Question

I'm on mac OSX and deploying a python lambda on AWS.

I have created a local env source venv/bin/activate following these instructions.

https://serverless.com/blog/serverless-python-packaging/

I have installed all of the packages

$ pip install numpy

Requirement already satisfied: numpy in ./venv/lib/python3.5/site-packages (1.14.2)

then i deploy the package using

pip freeze > requirements.txt

serverless deploy

error when running on the lambda

START RequestId: ################### Version: $LATEST Unable to import module 'main': Missing required dependencies ['numpy']

Also note: my code is not calling numpy, it's calling quandl and quandl is calling numpy

requirements.txt

asn1crypto==0.24.0
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
cryptography==2.2.2
idna==2.6
inflection==0.3.1
more-itertools==4.1.0
ndg-httpsclient==0.4.4
numpy==1.14.2
pandas==0.22.0
pyasn1==0.4.2
pycparser==2.18
pyOpenSSL==17.5.0
python-dateutil==2.7.2
pytz==2018.4
Quandl==3.3.0
requests==2.18.4
six==1.11.0
urllib3==1.22

Running the same code on an ec2. looks like its numpy is having an issue calling it.

I added the below to the python file

import os
import sys

CWD = os.path.dirname(os.path.realpath(__file__))
sys.path.insert(0, os.path.join(CWD, "lib"))
# end magic four lines

error

Traceback (most recent call last):
  File "__main__.py", line 11, in <module>
    import quandl
  File "/home/ubuntu/bots/ssali/quandl/__init__.py", line 7, in <module>
    from .model.database import Database
  File "/home/ubuntu/bots/ssali/quandl/model/database.py", line 18, in <module>
    import quandl.model.dataset
  File "/home/ubuntu/bots/ssali/quandl/model/dataset.py", line 5, in <module>
    from .data import Data
  File "/home/ubuntu/bots/ssali/quandl/model/data.py", line 1, in <module>
    from quandl.operations.data_list import DataListOperation
  File "/home/ubuntu/bots/ssali/quandl/operations/data_list.py", line 1, in <module>
    from quandl.model.data_list import DataList
  File "/home/ubuntu/bots/ssali/quandl/model/data_list.py", line 2, in <module>
    from .data_mixin import DataMixin
  File "/home/ubuntu/bots/ssali/quandl/model/data_mixin.py", line 1, in <module>
    import pandas as pd
  File "/home/ubuntu/bots/ssali/pandas/__init__.py", line 19, in <module>
    "Missing required dependencies {0}".format(missing_dependencies))
ImportError: Missing required dependencies ['numpy']

Lambda only supports python 2.7 and 3.6 can you try with 3.6 instead of 3.5 first? — Jacques Kvam
I think the issue is that panda can't see numpy as its in a folder out side of the "panda" folder — Chris Evans
I think the real issue has to do with the numpy package that gets installed on osx, but I wanted to verify that it wasn't 3.5 before I wrote a longer answer. — Jacques Kvam

Jacques Kvam Jacques Kvam · Accepted Answer · 2018-04-22T05:28:23

I think there were two issues here:

AWS Lambda only support Python 2.7 and 3.6 so we should use 3.6 instead of 3.5
Packages like Numpy that need to be compiled need to be built for Linux. If you are on Windows or OSX, these packages need to be installed through Docker. Serverless contains a convenient configuration for this. Make sure the following is in your serverless.yml.

From https://serverless.com/blog/serverless-python-packaging/

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: non-linux

Numpy lambda error using serverless

1 Answers