10
votes

I am trying to use encrypted environment variables in an AWS Lambda function running in Node.js 4.3, but the code hangs when trying to decrypt the variables. I don't get any error messages, it just times out. Here is what I have tried:

I created the encryption key in the same region as the Lambda, and ensured that the role the Lambda runs as has access to the key. (I've even tried giving the role full control of the key.)

When creating the Lambda, I enable encryption helpers, select my encryption key, and encrypt the environment variable:

enter image description here

Next I click the "Code" button which gives me javascript code that's supposed to handle the decryption at runtime. Here is the code--the only change I have made is to add console.log statements and I added a try/catch:

"use strict";

const AWS = require('aws-sdk');

const encrypted = process.env['DBPASS'];
let decrypted;


function processEvent(event, context, callback) {
    console.log("Decrypted: " + decrypted);
    callback();
}

exports.handler = (event, context, callback) => {
    if (decrypted) {
        console.log('data is already decrypted');
        processEvent(event, context, callback);
    } else {
        console.log('data is NOT already decrypted: ' + encrypted);
        // Decrypt code should run once and variables stored outside of the function
        // handler so that these are decrypted once per container
        const kms = new AWS.KMS();
        console.log('got kms object');
        try {
        var myblob = new Buffer(encrypted, 'base64');
        console.log('got blob');
        kms.decrypt({ CiphertextBlob: myblob }, (err, data) => {
            console.log('inside decrypt callback');
            if (err) {
                console.log('Decrypt error:', err);
                return callback(err);
            }
            console.log('try to get plaintext');
            decrypted = data.Plaintext.toString('ascii');
            console.log('decrypted: ' + decrypted);
            processEvent(event, context, callback);
        });
        }
        catch(e) {
            console.log("exception: " + e);
            callback('error!');
        }
    }
};

Here is what I get when I run the function:

data is NOT already decrypted: AQECAH.....
got kms object
got blob
END RequestId: 9b7af.....
Task timed out after 30.00 seconds

When I run the function, it times out. I see that it prints all log statements up to "got blob" then it just stops. No error message other than timed out. I've tried increasing timeout and memory for the Lambda but it just makes it wait longer before timing out.

How is decryption supposed to work when I never tell the app what decryption key to use? The documentation for decrypt does not mention any way to tell it what decryption key to use. And I am not getting any error messages that would tell me it doesn't know what key to use or anything.

I've tried going through this tutorial but it just tells me to do the same thing I've already done. I've also read all of the environment variables documentation but it says that what I'm doing should just work.

1
Is your Lambda function running inside a VPC?garnaat
@garnaat yes it is in a VPC. It is part of a security group and subnet that is able to connect to an RDS MySQL instanceKip
I think the issue is that you need internet access to make the call to KMS since it is hitting the KMS endpoints. There is no VPC endpoint for KMS so you would need to set up a NAT server in your VPC.garnaat
@garnaat yes that was the issue. if you can convert your comment to an answer i can accept it.Kip
Side note: Depending on how often your Lambda will be invoked, to reduce cost of your KMS usage, you may want to consider storing a symmetric key in KMS and decrypt that outside of your handler. Then use it inside the handler to decrypt the real objects.Pieter Ennes

1 Answers

12
votes

Decrypting the environment variables requires an API call to the KMS service. To do that, your Lambda function must have access to the internet since there are no VPC endpoints for KMS. So, if your Lambda is running in a VPC, make sure you have a NAT configured for the VPC to allow your Lambda function to call KMS.