Serverless deploy with serverless-python-requirements does not work

Question

I am trying to deploy a python lambda function using serverless. I need the psycopg2 library so I used the serverless-python-requirements plugin and added it to my serverless.yml file.

plugins:
  - serverless-python-requirements
custom:
    pythonRequirements:
        dockerizePip: non-linux

I can deploy the function successfully:

Serverless: Installing requirements from 
/Users/Desktop/postgresql/.serverless/requirements/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.6
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Injecting required Python packages to package...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (43.07 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...

..............
Serverless: Stack update finished...

But when I check my Cloudwatch logs, it says that there is "No module named 'psycopg2' ".

The structure of my zip file is:

    |--node_modules
    |--.gitignore
    |--handler.py
    |--package-lock.json
    |--package.json
    |--serverless.yml

No .serverless/ dir was created when I ran serverless deploy...

Any help would be greatly appreciated!

Answer 1

You only need to define your Python dependencies in the requirements.txt file. You can do this manually or use pip freeze with:

 pip freeze > requirements.txt

Answer 2

Did you installed python dependencies correctly?

serverless is trying to install requirements from Serverless: Installing requirements from /Users/Desktop/postgresql/.serverless/requirements/requirements.txt

I would suggest following

• create virtualenv virtualenv ./env
• activate it source env/bin/activate
• install your depdencies pip install psycopg2
• freeze your local dependencies pip freeze > requirements.txt

Assuming your code is importing your dependency, deploy using serverless.

Answer 3

I deployed Python Flask application on AWS and while setting it up, got the same error. I solved it by doing the following,

I hope you are using Virtual Environment where you installed all the Python libraries you need. I didn't see virtual environment files in your list of folders. You can check the readme file of my git repo to see how to build from scratch. I documented all the steps I followed - https://github.com/shyam454/Flask_AWS

Note: The repo works but I still developing it, so the endpoints mentioned in Readme are not yet in the code. But it is still good regarding Python + Flask setup

If you already have virtual environment, do the following steps,

Before deployment using serverless, do pip freeze > requirements.txt

This command will include all the python libraries you installed in your virtual environment to be present in requirements.txt file

In the serverless.yml file, add these lines

plugins: - serverless-wsgi - serverless-python-requirements

Now do, serverless deploy

You can observe that the serverless will install all the requirements from the requirements.txt. you should see the below lines while serverless is deploying,

Serverless: Generated requirements from xxxxxx\AWS\Flask_AWS\requirements.txt in xxxxxx\AWS\Flask_AWS.serverless\requirements.txt...

Serverless: Installing requirements from xxxxxxx\AWS\Flask_AWS.serverless\requirements\requirements.txt ...

This is when it installs the python libraries in the cloud and you will not get the error no module available.

I hope this helps you.

Answer 4

A bit late but for those who are facing the similar issue.

You are deploying your requirements as zip. You have to unzip the requirements before you import them in your python script. Add this line of code on top of your python script where you want libraries to be imported:

try:
 import unzip_requirements
except ImportError:
 pass

You can throw an error on exception if you desire or just pass.

Answer 5

I suggest that you use AWS lambda layer. Upload manually the working packages of psycopg2 in aws lambda layer.

and reference it on serverless.yml name: [name of your serverless project] layers: - [arn-of aws layers package]

Note: layers is on the same level of name in serverless.yml