Python package management for AWS lambda function

Question

I'm struggling to understand how best to manage python packages to get zipped up for an AWS lambda function.

In my project folder I have a number of .py files. As part of my build process I zip these up and use the AWS APIs to create and publish my lambda function supplying the zip file as part of that call.

Therefore, it is my belief that I need to have all the packages my lambda is dependant on within my project folder.

With that in mind, I call pip as follows:

pip install -t . tzlocal

This seems to fill my project folder with lots of stuff and I'm unsure if all of it needs to get zipped up into my lambda function deployment e.g.

.\pytz
.\pytz-2018.4.dist-info
.\tzlocal
...
...

First question - does all of this stuff need to be zipped up into my lambda?

If not, how do I get a package that gives me just the bits I need to go into my zip file?

Coming from a .Net / Node background - with the former I NuGet my package in and it goes into a nice packages folder containing just the .dll file I need which I then reference into my project.

If I do need all of these files is there a way to "put" them somewhere more tidy - like in a packages folder?

Finally, is there a way to just download the binary that I actually need? I've read the problem here is that the Lambda function will need a different binary to the one I use on my desktop development environment (Windows) so not sure how to solve that problem either.

Alexey Smirnov Alexey Smirnov · Accepted Answer · 2018-06-15T08:46:26

Binary libraries used for example by numpy should be compiled on AWS Linux to work on lambda. I find this tutorial useful (https://serverlesscode.com/post/deploy-scikitlearn-on-lamba/). There is even newer version of it which uses docker container so you do not need EC2 instance for compilation and you can do everything locally.

As for the packages: the AWS docs says to install everything to the root, but you may install them all in a ./packages directory and append it to path in the beginning if the lambda handler code

import os
import sys

cwd = os.getcwd()
package_path = os.path.join(cwd, 'packages')
sys.path.append(package_path)

Python package management for AWS lambda function

1 Answers