Unable to import module 'lambda_function': No module named 'pyspark'

0
votes

I'm trying to run my PySpark job code using AWS Lambda function. Here i'm getting below error: Unable to import module 'lambda_function': No module named 'pyspark'

My PySpark job has below imports. So, how can I import below library into my Lambda function. How to install these in Lambda function ?

from pyspark.sql import SparkSession
from pyspark.sql import functions as F, Column as col
from pyspark.sql.functions import when, coalesce
from pyspark.sql.functions import lit

Thanks.

2
python-3.xpysparkaws-lambda

2 Answers

1
votes

Guys you must aware three things, when you got this error

Unable to import module 'lambda_function'

  1. File permission
  2. handler format would be python_filename.lambda_handler
  3. When you compress files you must give option as zip -r name.zip. (hidden files are add only when we end dot(.) at end of zip file)
0
votes

You need to package all AWS Lambda dependencies together with your code into deployment zip file.

https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html