python - Pandas in AWS lambda gives numpy error

Question

I've been trying to run my code in AWS Lambda which imports pandas. So here is what I've done. I have a python file which contains a simple code as follows(This file has the lambda handler)

import json
print('Loading function')
import pandas as pd
def lambda_handler(event, context):
    return "Welcome to Pandas usage in AWS Lambda"

1. I have zipped this python file along with numpy, pandas and pytz libraries as a deployment package (Did all these in Amazon EC2 linux machine)
2. Then uploaded the package into S3
3. Created a lambda function(runtime=python3.6) and uploaded the deployment package from S3

But when I test the lambda function in AWS Lambda, I get the below error:

Unable to import module 'lambda_function': Missing required dependencies ['numpy']

I already have numpy in the zipped package but still I get this error. I tried to follow the hints given at Pandas & AWS Lambda but no luck.

Did anyone ran into the same issue. Would appreciate any hint or suggestions to solve this problem.

Thanks

Answer 1

EDIT: I figured out finally how to run pandas & numpy in a AWS Lambda python 3.6 runtime environment.

I have uploaded my deployment package to the following repo:

git clone https://github.com/pbegle/aws-lambda-py3.6-pandas-numpy.git

Simply add your lambda_function.py to the zip file by running:

zip -ur lambda.zip lambda_function.py

Upload to S3 and source to lambda.

ORIGINAL:

The only way I have gotten Pandas to work in a lambda function is by compiling the pandas (and numpy) libraries in an AWS Linux EC2 instance following the steps from this blog post and then using the python 2.7 runtime for my lambda function.

Answer 2

To include numpy in your lambda zip follow the instructions on this page in the AWS docs...

How do I add Python packages with compiled binaries to my deployment package and make the package compatible with AWS Lambda?

To paraphrase the instructions using numpy as an example:

1. Open the module pages at pypi.org. https://pypi.org/project/numpy/

2. Choose Download files.

3. Download:

For Python 2.7, module-name-version-cp27-cp27mu-manylinux1_x86_64.whl

e.g. numpy-1.15.2-cp27-cp27m-manylinux1_x86_64.whl

For Python 3.6, module-name-version-cp36-cp36m-manylinux1_x86_64.whl

e.g. numpy-1.15.2-cp36-cp36m-manylinux1_x86_64.whl

4. Uncompress the wheel file on the /path/to/project-dir folder. You can use the unzip command on the command line to do this. There are other ways obviously.

unzip numpy-1.15.2-cp36-cp36m-manylinux1_x86_64.whl

When the wheel file is uncompressed, your deployment package will be compatible with Lambda.

Hope that all makes sense ;)

The end result might look something like this. Note: you should not include the whl file in the deployment package.

Answer 3

After doing a lot of research I was able to make it work with Lambda layers.

Create or open a clean directory and follow the steps below:

Prerequisites: Make sure you have Docker up and running

1. Create a requirements.txt file with the following:

pandas==0.23.4
pytz==2018.7

2. Create a get_layer_packages.sh file with the following:

#!/bin/bash

export PKG_DIR="python"

rm -rf ${PKG_DIR} && mkdir -p ${PKG_DIR}

docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.6 \
    pip install -r requirements.txt --no-deps -t ${PKG_DIR}

3. Run the following commands in the same directory:

chmod +x get_layer_packages.sh

./get_layer_packages.sh

zip -r pandas.zip .

4. Upload the layer to a S3 bucket.

5. Upload the layer to AWS by running the command below:

aws lambda publish-layer-version --layer-name pandas-layer --description "Description of your layer"
--content S3Bucket=<bucket name>,S3Key=<layer-name>.zip
--compatible-runtimes python3.6 python3.7

6. Go to Lambda console and upload your code as a zip file or use the inline editor.

7. Click on Layers > Add a layer> Search for the layer (pandas-layer) from the Compatible layers and select the version.

8. Also add the AWSLambda-Python36-SciPy1x layer which is available by default for importing numpy.

Selecting the layer from the console

9. Test the code. It should work now!!!!

Thanks to this medium article https://medium.com/@qtangs/creating-new-aws-lambda-layer-for-python-pandas-library-348b126e9f3e

Answer 4

AWS Lambda use Amazon Linux operating system. Idea is download Pandas and NumPy compatible with Amazon Linux. What you download using pip is specific to Windows or Mac. You need to download the compatible version for Linux, so that your Lambda function can understand it. These files are called wheel files.

Create new local directory with lambda_function.py file. Install Pandas to local directory with pip:

$ pip install -t . pandas

Navigate to https://pypi.org/project/pandas/#files. Search for and download newest *manylinux1_x86_64.whl package. In my case, I'm using Python 3.6 on my Lambda function, so I downloaded the following:

• Pandas - pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl

• NumPy - numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl

Download whl files to directory with lambda_function.py. Remove pandas, numpy, and *.dist-info directories. Unzip whl files.

$ rm -r pandas numpy *.dist-info
$ unzip numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl
$ unzip pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl

Remove whl files, *.dist-info, and __pycache__. Prepare zip.zip archive:

$ rm -r *.whl *.dist-info __pycache__
$ zip -r zip.zip .

Upload the zip.zip file in your Lambda function.

Source: https://medium.com/@korniichuk/lambda-with-pandas-fd81aa2ff25e

Answer 5

To get additional libraries in Lambda we need to compile them on Amazon Linux (this is important if the underlying library is based on C or C++ like for Numpy) and package them in a ZIP file together with the python script you want to run in Lambda.

To get the Amazon Linux compiled version of the libraries. You can either find a version that someone already compiled, like the one by @pbegle, or compile it yourself. To compile it ourself there are two options: - compile the libraries on an EC2 instance https://streetdatascience.com/2016/11/24/using-numpy-and-pandas-on-aws-lambda/ - compile the libraries on a docker version of Lambda environment https://serverlesscode.com/post/scikitlearn-with-amazon-linux-container/

Following the last option with Docker, it is possible to make it work using the instructions in the blog post above and by adding:

pip install --use-wheel pandas

in the script to compile the libraries:

https://github.com/ryansb/sklearn-build-lambda/blob/master/build.sh#L21

Answer 6

Slightly duplicate of Cannot find MySQL in NodeJS using AWS Lambda

You need to package your libraries with Lambda. As lambda runs on a public cloud, you cannot configure it.

Now in your case, as you are using pandas, you need to package Pandas with your zip. Get a path to pandas(for example: /Users/dummyUser/anaconda/lib/python3.6/site-packages) and copy the library to the place where you have your lambda function code. Inside your code, refer to pandas from your local copy. While uploading, zip the whole set(code + libraries), and upload as you will. It should work.

Answer 7

The main problem is that libraries compiled within a specific OS will only work for that OS. So, if a library is compiled within macOS then it will not run in Linux environment. Why is this a problem?

One of the dependencies of Pandas is Numpy (which is compiled due to speed, see this answer on stackoverflow). AWS Lambda uses Linux hence if the Numpy compilation is done within macOS or windows, then the compilation will only be good for those specific architectures and will not work within AWS. I think those using Linux will not experience this problem.

To solve this problem and create a working AWS Lambda layer for Pandas, follow these simple steps (see the general steps on amazon forum):

mkdir awsPandasLayer #create a directory
cd awsPandasLayer #cd into the directory
pip3 install -t . Pandas #install pandas and all its dependencies 
rm -r pandas numpy *.dist-info __pycache__ #clean up the environment to remove the incompatible numpy and pandas

Then download the latest precompiled packages for Pandas and Numpy into the awsPandasLayer directory earlier created. For my case, since I am using Python 3.7, I downloaded these versions of Pandas and Numpy. Notice cp37 denotes the python version as clearly stated on the site. Then complete the installation as follows:

unzip pandas-1.0.5-cp37-cp37m-manylinux1_x86_64.whl #unzip the pandas precompiled package
unzip numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl #unzip the numpy precompiled package
pip3 install -t . openpyxl #I found this is required within AWS Lambda for excel files
rm -r *.whl *.dist-info __pycache__ #clean up unneeded files
zip -r awsPandasLayer.zip . # zip all the files

The zip file can then be uploaded as an AWS Lambda layer and it should work. Note that I am using macOS.

Answer 8

I tried some of the solution here but most of them didn't work. I liked the idea @Ranadeep Guha suggested of creating a container and downloading the repos over there so that's what I did.

I worked in the dir my lambda function was located on and created the following files :

Dockerfile :

FROM python:3.8-slim
WORKDIR /app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt -t /app

requirements.txt : (those were mine)

pandas
numpy
xmltodict

Now in my gitbash I run the following command that will generate a docker image for me with all the dependencies installed :

docker build -t image_name .

Sending build context to Docker daemon  5.632kB
Step 1/4 : FROM python:3.8-slim
 ---> 56930ef6f6a2
Step 2/4 : WORKDIR /app
 ---> Using cache
 ---> ea0bf539bcad
Step 3/4 : COPY requirements.txt ./
 ---> cb4c005f53cc
Step 4/4 : RUN pip install --no-cache-dir -r requirements.txt -t /app
 ---> Running in a0d179a372b4
Collecting pandas
  Downloading pandas-1.0.3-cp38-cp38-manylinux1_x86_64.whl (10.0 MB)
Collecting numpy
  Downloading numpy-1.18.3-cp38-cp38-manylinux1_x86_64.whl (20.6 MB)
Collecting xmltodict
  Downloading xmltodict-0.12.0-py2.py3-none-any.whl (9.2 kB)
Collecting pytz>=2017.2
  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting python-dateutil>=2.6.1
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting six>=1.5
  Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: numpy, pytz, six, python-dateutil, pandas, xmltodict
Successfully installed numpy-1.18.3 pandas-1.0.3 python-dateutil-2.8.1 pytz-2020.1 six-1.14.0 xmltodict-0.12.0

Now just create a Docker container of that image and confirm that everything installed :

winpty docker run --name container_name -it --entrypoint bash image_name

type ls and you will see all the installs.

Now let's copy all the installs to your local pc. You can replace the dot with any location on your pc:

 docker cp container_id:/app/* . 

Answer 9

I've been struggling with a similar error while trying to use the python3.6 engine. When I switched to 2.7 it worked fine for me. I used Amazon AMI to create my zip file, but it has only python3.5, not 3.6. I guess the version mismatch was the reason. But it's just a guess, I haven't tried the process on a python3.6 installation yet.

Answer 10

with the serverless framework, you can easily package and deploy your dependencies correctly.

you only need to;

1. install serverless

   npm install -g serverless
   

2. create a serverless.yml in the root of your project with the following:

   service: numpy-test
   
   # define the environment of your lambda
   provider:
     name: aws
     runtime: python3.6
   
   # specify the function you want to deploy
   functions:
     numpy:
       # path to your lambda_handler function
       handler: path/to/function.lambda_handler
   
   # add a plugin that allows serverless to package python libraries
   # specified in the requirements.txt or Pipfile
   plugins:
     - serverless-python-requirements
   
   # this section makes sure your libraries get build correctly 
   # for an aws lambda environment
   custom:
     pythonRequirements:
       dockerizePip: non-linux
   

   adjust the path/to/function.lambda_handler

3. make sure docker is running and execute

   serverless deploy

once the deployment is finished, go to the AWS console look for the function numpy-test-dev-numpy and test your function.

this article explains the necessary steps in detail.

Answer 11

This is similar to Randeep's answer but you don't need to use Lambda Layers if you don't want to do that.

As others have stated, this is not working because pandas/numpy require binaries to be built and the operating system of your build machine (Linux, Mac, Windows) does not match the operating system of Lambda (Amazon Linux).

To solve this, you can use docker to download/build your dependencies and package them up on Amazon Linux. Amazon provides a Docker image for this purpose. See below for how I built my python package for Python 3.6 runtime (they have other dockers for all other runtimes):

Put all of your dependencies into a requirements.txt file, for example:

openpyxl
boto3
pandas

Create a script (i.e. named build.sh) that will build your package, here is what mine looked like:

#!/bin/bash

# remove old build artifacts
rm -rf build
rm lambda_package.zip

# make build dir and copy my lambda handler file into it
mkdir build
cp lambda_daily_util_gen.py  build/

# Use requirements file to download/build dependencies into the build folder
cd build
pip install -r ../requirements.txt --target .

# Create an lambda package with my files and all dependencies
zip -r9 ../lambda_package.zip .

Ensure you have the Amazon Linux lambda build image pulled:

$ docker pull lambci/lambda

Run your build script inside of the docker container:

Windows:

$ docker run --rm -v "$PWD":/var/task lambci/lambda:build-python3.6 /var/task/build.sh

Mac/Linux:

docker run --rm -v ${PWD}:/var/task lambci/lambda:build-python3.6 chmod +x build.sh;./build.sh

You should now see a file named lambda_package.zip that was built on Amazon Linux you can upload to AWS.

Hope that helps.

Answer 12

Maybe module you're using is for python 2.7.

Try install pandas module for python 3.x

i.e.

pip3 install pandas -t .

It worked for me.

Or for instant result, Change your runtime to python 2.7 (not recommended)

Answer 13

To make a deployment package that's compatible with Lambda, download a precompiled package called a wheel (.whl). Uncompress the wheel file on /path/to/project-dir instead of using pip install.

Resolution

1. Open your module-name pypi.org page. For example: https://pypi.org/project/numpy/

2. Choose Download files.

3. Download:

For Python 2.7, module-name-version-cp27-cp27mu-manylinux1_x86_64.whl For Python 3.6, module-name-version-cp36-cp36m-manylinux1_x86_64.whl 4. Uncompress the wheel file on the /path/to/project-dir folder.

When the wheel file is uncompressed, your deployment package will be compatible with Lambda.

Answer 14

Your code always give this error
because lambda does not contain any external library it having a library which by default come with Python.

if you are using any external library like pandas, numpy or any other. you need to install that library on Aws Lambda
before using it

see you code

import json
print('Loading function')
import pandas as pd
def lambda_handler(event, context):
    return "Welcome to Pandas usage in AWS Lambda"

here no installation of pandas library so your code is not working.
my suggestion is use your code as follows. write all you code inside the lambda function

import json
def lambda_handler(event, context):
    #install python libray here 
    print('Loading function')
    import pandas as pd
    return "Welcome to Pandas usage in AWS Lambda" 

So final code look as follows

 def lambda_handler(event, context):
     import pip

    def install(package):
        if hasattr(pip, 'main'):
            pip.main(['install', package])
        else:
            pip._internal.main(['install', package])

    if __name__ == '__main__':
        install('pandas')

    #install python libray here 
    print('Loading function')
    import pandas as pd
    return "Welcome to Pandas usage in AWS Lambda"