Dependencies Missing while deploying an AWS Lambda using Serverless Framework

Question

I am trying to use serverless framework to deploy a prediction Lambda handler to AWS, my code's dependencies are pytorch and transformers(HF).

I am facing this problem, not sure how to overcome the same,

I have configured the requirements.txt file properly and it has the below 2 lines,

transformers==2.10
https://download.pytorch.org/whl/cpu/torch-1.5.0%2Bcpu-cp38-cp38-linux_x86_64.whl

When I try to use the command "serverless deploy", at times it creates a package with all the dependencies in the requirements.zip within the package, which comes to around 144 MB, this is the expected result.

The log for your reference

C:\Users\Rajeshwaranp\serverless-HFT>sls deploy
Serverless: Deprecation warning: Starting with next major version, API Gateway naming will be changed from "{stage}-{service}" to "{service}-{stage}".
            Set "provider.apiGateway.shouldStartNameWithService" to "true" to adapt to the new behavior now.
            More Info: https://www.serverless.com/framework/docs/deprecations/#AWS_API_GATEWAY_NAME_STARTING_WITH_SERVICE
Serverless: Adding Python requirements helper...
Serverless: Generated requirements from C:\Users\Rajeshwaranp\serverless-HFT\requirements.txt in C:\Users\Rajeshwaranp\serverless-HFT\.serverless\requirements.txt...
Serverless: Installing requirements from C:\Users\Rajeshwaranp\serverless-HFT\cache\72130b8b1c12a9cc09719c938b787b829ceb78a73eeafa251ccf974d7dafec58_slspyc\requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.8
Serverless: Using download cache directory C:\Users\Rajeshwaranp\serverless-HFT\cache\downloadCacheslspyc
Serverless: Running docker run --rm -v C\:/Users/Rajeshwaranp/serverless-HFT/cache/72130b8b1c12a9cc09719c938b787b829ceb78a73eeafa251ccf974d7dafec58_slspyc\:/var/task\:z -v C\:/Users/Rajeshwaranp/serverless-HFT/cache/downloadCacheslspyc\:/var/useDownloadCache\:z -u 0 lambci/lambda\:build-python3.8 python -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache...
Serverless: Zipping required Python packages...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Removing Python requirements helper...
Serverless: Injecting required Python packages to package...
Serverless: WARNING: Function predict_answer has timeout of 60 seconds, however, it's attached to API Gateway so it's automatically limited to 30 seconds.
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
........
Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service serverless-hft.zip file to S3 (144.02 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
.................................
Serverless: Stack update finished...

But at times, when I use the same command to deploy, it creates a package with 128KB(approx) without any dependencies in the requirements.zip file within the package.

The log for your reference,

C:\Users\Rajeshwaranp\serverless-HFT>sls deploy
Serverless: Deprecation warning: Starting with next major version, API Gateway naming will be changed from "{stage}-{service}" to "{service}-{stage}".
            Set "provider.apiGateway.shouldStartNameWithService" to "true" to adapt to the new behavior now.
            More Info: https://www.serverless.com/framework/docs/deprecations/#AWS_API_GATEWAY_NAME_STARTING_WITH_SERVICE
Serverless: Adding Python requirements helper...
Serverless: Generated requirements from C:\Users\Rajeshwaranp\serverless-HFT\requirements.txt in C:\Users\Rajeshwaranp\serverless-HFT\.serverless\requirements.txt...
Serverless: Using static cache of requirements found at C:\Users\Rajeshwaranp\serverless-HFT\cache\72130b8b1c12a9cc09719c938b787b829ceb78a73eeafa251ccf974d7dafec58_slspyc ...
Serverless: Zipping required Python packages...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Removing Python requirements helper...
Serverless: Injecting required Python packages to package...
Serverless: WARNING: Function predict_answer has timeout of 60 seconds, however, it's attached to API Gateway so it's automatically limited to 30 seconds.
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service serverless-hft.zip file to S3 (124.15 KB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
..............
Serverless: Stack update finished...

So why is this happening? and how can I tweak the deployment command so that the dependencies are always included?

To pack dependencies we will use lambda layers and provide the function those layers. I will add a answer for your reference. — Sridhar Raju
Thanks for your response Sridhar, I am using the Lambda layer but still, I would like to understand what causes this strange behavior, i.e sometimes it packs sometimes it leaves the dependencies. — Rajesh Waran

Sridhar Raju Sridhar Raju · Accepted Answer · 2020-11-16T10:00:10

Use lambda layer to pack all your requirements, Make sure you have all requirements in requirements.txt file.

This works only when serverless-python-requirements plugin is listed in plugins section. Replace your custom key with this and give the functions that need those requirements a reference to use that layer.

plugins:
  - serverless-python-requirements
custom:
  pythonRequirements:
    layer: true
functions:
  auth:
    handler: data.auth
    layers:
      - { Ref: PythonRequirementsLambdaLayer}

Dependencies Missing while deploying an AWS Lambda using Serverless Framework

1 Answers