AWS Lambda access Secrets Manager from within VPC

Question

I have a lambda that needs to communicate 'locally' with an EC2 instance in a private VPC. The API key is being stored in Secrets Manager.

Using the default code provided by Secrets Manager and the necessary IAM roles I am able to read the API key from Secrets Manager in my Lambda:

# Use this code snippet in your app.
# If you need more information about configurations or implementing the sample code, visit the AWS docs:   
# https://aws.amazon.com/developers/getting-started/python/

import boto3
import base64
from botocore.exceptions import ClientError

def get_secret():

    secret_name = "MYSECRET"
    region_name = "ap-southeast-2"

    # Create a Secrets Manager client
    session = boto3.session.Session()
    client = session.client(
        service_name='secretsmanager',
        region_name=region_name
    )

    # In this sample we only handle the specific exceptions for the 'GetSecretValue' API.
    # See https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
    # We rethrow the exception by default.

    try:
        get_secret_value_response = client.get_secret_value(
            SecretId=secret_name
        )
    except ClientError as e:
 ... # Default error handling..
    else:
        # Decrypts secret using the associated KMS CMK.
        # Depending on whether the secret is a string or binary, one of these fields will be populated.
        if 'SecretString' in get_secret_value_response:
            secret = get_secret_value_response['SecretString']
            return secret
        else:
            decoded_binary_secret = base64.b64decode(get_secret_value_response['SecretBinary'])
            return decoded_binary_secret

def lambda_handler(event, context):
    secrt = get_secret()

    return {
        'statusCode': 200,
        "headers": {
            'Access-Control-Allow-Origin': '*',
            'Content-Type': 'application/json'
        },
        'body': secrt 
    }

This Lambda is able to successfully retrieve and print the API key from Secrets Manager.

To communicate with the EC2 instance I have a Lambda with a helper layer and some simple test code:

import apihelper
import json

def lambda_handler(event, context):
  conn = apihelper.getConnection('API KEY')
  return {
    'statusCode': 200,
    "headers": {
      "Access-Control-Allow-Origin": "*"
    },
    'body': json.dumps(conn.listProducts())
  }

This lambda is in the VPC, subnet and has the necessary security group rules to communicate with the EC2 instance. Hard coding the API KEY it successfully returns the expected data from the EC2 instance.

When I try to combine them so that the API key is not hard-coded the Lambda no longer works. There is no error message it just times out.

I have tried:

Increasing the timeout to over a minute
Placing allow all inbound and outbound rules on the security group
Configuring a VPC endpoint for Secrets Manager

I think I have narrowed it down to the VPC. The first Lambda that just prints out the secret works perfectly until I place it in the VPC. But I don't know where to look or how to configure it to allow the Lambda to talk to both the EC2 inside the VPC as well as Secrets Manager.

Marcin Marcin · Accepted Answer · 2020-06-09T02:39:20

There is no error message it just times out.

Unfortunately, a lambda function in a VPC does not have internet access nor public IP. From docs:

Connecting a function to a public subnet does not give it internet access or a public IP address.

Therefore, when you use boto3:

    client = session.client(
        service_name='secretsmanager',
        region_name=region_name
    )

to connect to the Secrets Manager it times out, because boto3 can't connect to the Secrets Manager manager from the VPC.

There are two ways to rectify this:

Place your function in a private subnet and use NAT gateway/instance with correctly configured route tables to provide the internet access, and consequently, to the Secrets Manager.
Setup VPC interface endpoint for Secrets Manager in a private subnet. This way your lambda function will be able to connect to the Secrets Manager using the endpoint, without needing the internet.

AWS Lambda access Secrets Manager from within VPC

1 Answers