Pip install Python package within AWS Lambda?

Question

I'm trying to pip install a package in an AWS Lambda function.

The method recommended by Amazon is to create a zipped deployment package that includes the dependencies and python function all together (as described in AWS Lambda Deployment Package in Python). However, this results in not being able to edit the Lambda function using inline code editing within the AWS Lambda GUI.

So instead, I would like to pip install the package within the AWS Lambda function itself. In AWS Lambda, the filesystem is read-only apart from the /tmp/ directory, so I am trying to pip install to the /tmp/ directory. The function is only called once-daily, so I don't mind about the few extra seconds required to re-pip install the package every time the function is run.

My attempt

def lambda_handler(event, context):
    # pip install dependencies
    print('begin lambda handler')
    import subprocess
    import sys
    subprocess.call('pip install cryptography -t /tmp/ --no-cache-dir'.split())
    from cryptography.fernet import Fernet
    pwd_encrypted = b'gAAAAABeTcT0OXH96ib7TD5-sTII6jMfUXPhMpwWRCF0315rWp4C0yav1XAPIn7prfkkA4tltYiWFAJ22bwuaj0z1CKaGl8vTgNd695SDl25HnLwu1xTzaQ='
    key = b'fP-7YR1hUeVW4KmFmly4JdgotD6qjR52g11RQms6Llo='
    cipher_suite = Fernet(key)
    result = cipher_suite.decrypt(pwd_encrypted).decode('utf-8')
    print(result)
    print('end lambda handler')

However, this results in the error

[ERROR] ModuleNotFoundError: No module named 'cryptography'

I have also tried replacing the subprocess call with the following, as recommended in this stackoverflow answer

    cmd = sys.executable+' -m pip install cryptography -t dependencies --no-cache-dir'
    subprocess.check_call(cmd.split())

However, this results in the error

OSError: [Errno 30] Read-only file system: '/var/task/dependencies'

I would not recommend running pip inside Lambda. You could package your dependencies into an AWS Lambda Layer, which would then let you edit the remaining code in the Lambda management console. — John Rotenstein
@JohnRotenstein - Why is this not recommended? I'm not intending to use this in production. It would make the debugging/learning process easier if everything can be self-contained within a single python file. Are there any blockers to making this work? — maurera
Yes, the blockers are exactly what you are experiencing. An AWS Lambda function can only write to the /tmp/ directory. — John Rotenstein
@JohnRotenstein - I realize that, and that's why I'm attempting to only write to the /tmp/ directory by specifying /tmp/ as the install target. Is there a way to then import the package, once the package is pip installed to /tmp/? The two methods I've tried of pip-installing to a custom directory seem to work outside of AWS Lambda — maurera

maurera maurera · Accepted Answer · 2020-02-20T23:01:04

I solved this with a one-line adjustment to the original attempt. You just need to add /tmp/ to sys.path so that Python knows to search /tmp/ for the module. All you need to do is add the line sys.path.insert(1, '/tmp/').

Solution

import os
import sys
import subprocess

# pip install custom package to /tmp/ and add to path
subprocess.call('pip install cryptography -t /tmp/ --no-cache-dir'.split(), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
sys.path.insert(1, '/tmp/')
from cryptography.fernet import Fernet

def lambda_handler(event, context):
    # pip install dependencies
    pwd_encrypted = b'gAAAAABeTcT0OXH96ib7TD5-sTII6jMfUXPhMpwWRCF0315rWp4C0yav1XAPIn7prfkkA4tltYiWFAJ22bwuaj0z1CKaGl8vTgNd695SDl25HnLwu1xTzaQ='
    key = b'fP-7YR1hUeVW4KmFmly4JdgotD6qjR52g11RQms6Llo='
    cipher_suite = Fernet(key)
    result = cipher_suite.decrypt(pwd_encrypted).decode('utf-8')
    print(result)

Output

Hello stackoverflow!

Note - as @JohnRotenstein mentioned in the comments, the preferred method to add Python packages is to package dependencies in an AWS Lambda Layer. My solution just shows that it is possible to pip install packages directly in an AWS Lambda function.

Pip install Python package within AWS Lambda?

2 Answers