Installing numpy on Mac to work on AWS Lambda

Question

Is there a way to install numpy on a Mac so that it will work when uploaded to AWS Lambda? I have tried a variety of different ways, including using different pip versions, using easy_install, and following this post, but none of them seem to work. I also tried cloning the git repo and building from there, but I also wasn't able to get that to work (though I'm not sure if I copied the right files up after doing that)

The error I'm getting is:

Unable to import module 'lambda_function': Importing the multiarray numpy extension module failed. Most likely you are trying to import a failed build of numpy. If you're working with a numpy git repo, try git clean -xdf (removes all files not under version control). Otherwise reinstall numpy.

Inspired by this post, I was able to pip install numpy in a Linux environment and get it to work on Lambda.

So my question is: Is it possible to install numpy on a Mac so that it works on AWS Lambda?

Environment: MacBook Pro, MacOS 10.12.2, default python version 2.7.10

I've been testing it with a minor variation on the hello-world-python example on Lambda:

from __future__ import print_function
import numpy

def lambda_handler(event, context):
    #print("Received event: " + json.dumps(event, indent=2))
    print("value1 = " + event['key1'])

(Update) Extending the question: Why do some packages work and others don't?

Answer 1

Update: the preferred approach now is to just use the AWS-provided Lambda Layer for NumPy/SciPy, which is super easy to do.

In the console, select your function and then, under the "Design" section click on "Layers". Then click "Add a Layer", and select "AWSLambda-Python37-SciPy1x" under AWS Provided (or whatever the equivalent is for the version of Python you're using).

Then you can seamlessly import numpy, scipy, etc. into your code with no issues.

10/26/2020 - Added example screenshot:

Answer 2

Update: Please ignore the below and see my response above (here: Installing numpy on Mac to work on AWS Lambda) about using Lambda Layers. That is a MUCH easier approach.

===========================================================================

I had this same exact issue and resolved it easily by following the advice here:

https://serverless.com/blog/serverless-python-packaging/

Note that I also had to add the lines:

package:
  exclude:
    - venv/**

... to the end of my serverless.yml to get my zip file under the size limit.

I will re-copy the instructions from the blog post below, just in case the link ever goes stale.

---

==============================================

Create a Lambda Function that uses numpy

1. install node/npm
2. npm install -g serverless

3. configure aws for your computer (install the cli, run "aws configure" and add your credentials generated from IAM). 3a. install Docker for Mac

4.

Run the following:

serverless create \
  --template aws-python3 \
  --name numpy-test \
  --path numpy-test

cd numpy-test
virtualenv venv --python=python3
source venv/bin/activate

5. edit handler.py and replace its contents with:

handler.py

import numpy as np
def main(event, context):
    a = np.arange(15).reshape(3, 5)
    print("Your numpy array:")
    print(a)


if __name__ == "__main__":
    main('', '')

6.

Run the following:

pip3 install numpy
pip3 freeze > requirements.txt
cat requirements.txt

7. replace serverless.yml with:

serverless.yml

service: numpy-test

provider:
  name: aws
  runtime: python3.6

functions:
  numpy:
    handler: handler.main

8. run the following

run:

npm init

[accept defaults, then type "yes"]

npm install --save serverless-python-requirements

9. now replace serverless.yml with:

serverless.yml

service: numpy-test

provider:
  name: aws
  runtime: python3.6

functions:
  numpy:
    handler: handler.main

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: non-linux

package:
  exclude:
    - venv/**

10. Run the following

Run

serverless deploy --aws-profile [aws-account-you-want-to-upload-fxn-to]
serverless invoke -f numpy --log

11. If all goes well, you should see something like this:

Expected output:

START RequestId: b32af7a8-52fb-4145-9e85-5985a0f64fe4 Version: $LATEST
Your numpy array:
[[ 0  1  2  3  4]
 [ 5  6  7  8  9]
 [10 11 12 13 14]]
END RequestId: b32af7a8-52fb-4145-9e85-5985a0f64fe4
REPORT RequestId: b32af7a8-52fb-4145-9e85-5985a0f64fe4  Duration: 0.52 ms    
Billed Duration: 100 ms     Memory Size: 1024 MB    Max Memory Used: 37 MB

Answer 3

Building on @MarkB's comment, it would not be possible to build numpy on a Mac to use on AWS Lambda. So why do some packages work and others don't?

Python extension modules, as explained on Mark Nunnikhoven's blog here, are

written in C or C++ that can either extend python or call C or C++ libraries.

Since these modules are compiled specific to the system you're on, and AWS Lambda is a Linux environment, you'll need to install any extension modules on a Linux environment.

Answer 4

For using other packages which aren't available in AWS layers follow the following steps.

1. Go to your terminal
2. Make a directory named "python".
3. Go into the directory.
4. Enter "pip install " command.
5. Zip this file and add the zipped file to the AWS

Follow this link for more details : https://www.youtube.com/watch?v=3BH79Uciw5w