Unable to import module 'lambda_function': No module named 'pandas'

Question

START RequestId: 3d5691d9-ad79-4eed-a26c-5bc3f1a23a99 Version: $LATEST Unable to import module 'lambda_function': No module named 'pandas'
END RequestId: 3d5691d9-ad79-4eed-a26c-5bc3f1a23a99

I'm using Windows 7 64-bit as the host OS.

What I want to do

I simply want to use pandas in AWS-Lambda environment. Just like I use it in windows environment, I am looking for a simple solution for Lambda.

What I have tried so far

• Installed Xubuntu on a virtual box.
• Create a virtual environment called myvenv in Xubuntu on virtual-box.
• Then I installed pandas3.6 in myvenv.
• Thereafter, I copied the folder contents in myvenv at location '/usr/local/lib/python3.6/site-packages/' to my host OS.
• In the host OS (windows 7), I created a folder called packs, pasted the contents of myvenv.
• created a lambda_function.py script in packs in host OS (windows 7)
• I then zipped the folder packs using 7zip software and upload it as zip in Lambda
• In Lambda, the lambda function handler name is, lambda_handler(). The code snippet looks like,

import pandas as pd

def lambda_handler(event, context):

    dates = pd.date_range('2019001', periods=6)

    df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
    print(df)

• The handler is named as lambda_function.lambda_handler. I have given the lambda-role AWSLambdaFullAccess permission.
• The time out is set to 4 min and 3 sec.

• The test event looks like

  { "key1": "This will be printed if all OK" }

I have tried the following solutions:

• Tried precompiled linux-compatible binaries for pandas & numpy from here -- no luck.
• In Lambda, changed the Handler info to python_filename.function_name. For my case, it was lambda_function.lambda_handler -- failed with no module named 'pandas' error.
• placed the lambda function in the root folder, zipped the folder using 7zip software and upload the folder to the S3 bucket. For my case, I placed the function at location python\lib\python3.6\site_packages\lambda_function.pyfailed with no module named 'pandas' error.
• Already tried these related solutions posted on SO, 1, 2, 3, 4, 5, 6

Note: I do not want to use Docker, because I do not know how to use it and I'm not willing to learn it as I'm exasperated now. I'm coming from a windows environment (it sucks, I now know.)

Any ideas on how to get this to work.

Answer 1

I was able to import the pandas library successfully using a Lambda layer and an Amazon linux Cloud 9 instance. There are the commands I executed in the Cloud 9 instance and the Lambda function's output. I had to change the code slightly since it was failing with an import error and string value error.

Alternatively, these commands can also be executed in an EC2 instance. If it's not possible to use the SAM CLI(which uses docker) or just plain docker on windows we'll need to use an Amazon Linux instance to build everything since that's what AWS Lambda uses currently. I don't believe using an ubuntu instance will work here.

Commands:

python --version
Python 3.6.8

# https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html
# python 3.6 uses Amazon Linux currently 

mkdir project
cd project
virtualenv v-env
source ./v-env/bin/activate
pip install pandas
deactivate

# creating layer
# https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-path
mkdir python
cd python
cp -r ../v-env/lib64/python3.6/dist-packages/* .
cd ..
zip -r panda_layer.zip python
aws lambda publish-layer-version --layer-name pandas --zip-file fileb://panda_layer.zip --compatible-runtimes python3.6 

The publish-later-version command will create a new AWS Lambda layer in the region given in the command or the config file for the CLI.

A Lambda layer will apply the library to the Lambda function's code without needing to apply it directly to the deployment package. This also allows the use of the online code editor in Lambda since the deployment package is under the limit of 3MB

I applied the Lambda layer by clicking on the Layer button in the web console and choosing the layer version that I most recently published. I have a second version there because the first time I attempted this is put the contents of the lib directory which isn't for a 64 bit OS and my code failed in AWS Lambda.

Alternatively, you can also apply the layer using the CLI command update-function-configuration

Lambda function code I used:

import pandas as pd
import numpy as np

def lambda_handler(event, context):
    dates = pd.date_range(start='1/1/2018', end='1/08/2018')
    df = pd.DataFrame(np.random.randn(8, 4), index=dates, columns=list('ABCD'))
    print(str(df))

---

Lambda output:

START RequestId: 27f09b6c-a4cd-49dd-bb3e-ae6fc7cd5850 Version: $LATEST
                   A         B         C         D
2018-01-01 -1.040318  0.450841 -0.381687 -0.105480
2018-01-02 -1.381793 -0.481572  0.828419 -0.885205
2018-01-03  1.437799 -0.649816 -0.577112  0.400670
2018-01-04 -0.730997 -0.778775 -1.514203  1.165661
2018-01-05  1.963595 -1.137054  0.920218  0.960210
2018-01-06 -0.429179 -0.745549  1.482562  0.298623
2018-01-07 -1.082388 -0.529476 -1.051663  1.616683
2018-01-08  0.042779 -2.338471 -0.142992  0.680399
END RequestId: 27f09b6c-a4cd-49dd-bb3e-ae6fc7cd5850
REPORT RequestId: 27f09b6c-a4cd-49dd-bb3e-ae6fc7cd5850  Duration: 536.76 ms Billed Duration: 600 ms Memory Size: 128 MB Max Memory Used: 122 MB Init Duration: 1721.51 ms   
XRAY TraceId: 1-5d741e40-1311daa29fc16c74735988fc   SegmentId: 61a595dd3492c331 Sampled: false