aws Lambda created ENI not deleting while deletion of stack

Question

CloudFormation creates Lambda function. When the function is executed an ENI is provisioned automatically by lambda. The ENI seems to be left in existence after function execution for to speed up subsequent function execution. CloudFormation deletes the lambda function. TheEN remains behind. When attempting to delete the VPC CloudFormation stack, stack deletion fails as the ENI is using a security group and subnet.

in my lambda role the delete permission are there.

"Effect": "Allow", "Action": [ "ec2:CreateNetworkInterface", "ec2:DeleteNetworkInterface", "ec2:DescribeNetworkInterfaces" ], "Resource": "*"

I am using custom resource to run the lambda from CloudFormation template, so lambda will be called both stack creation and deletion. The ENI will be used in creation of stack and deletion of stack. Now how to handle the eni deletion?

Answer 1

There is a known issue when using Lambda Functions in a VPC, as documented in Configuring a Lambda Function to Access Resources in an Amazon VPC:

There is a delay between the time your Lambda function executes and ENI deletion. If you do delete the role immediately after function execution, you are responsible for deleting the ENIs.

The documentation doesn't specify exactly how long this "delay" will be, but a forum post by Richard@AWS suggests it can last up to 6 hours(!). (In my observations using AWS CloudTrail, the delay between Lambda execution and ENI deletion was around one hour.)

Until AWS addresses this issue further, you can workaround the issue by detaching and deleting the leftover ENIs in between deleting the Lambda function and deleting the associated Security Group(s) and Subnet(s). This is how Terraform currently handles this issue in its framework.

You can do this manually by separating the VPC/Subnet/SG layer and the Lambda-function layer into two different CloudFormation Stacks, or you can automate it by implementing a Custom Resource to delete the ENIs using the AWS SDK.

Here's a complete working example that creates a VPC-Lambda Custom Resource, cleaning up its ENIs when deleted using the VPCDestroyENI Custom Resource:

Description: Creates a VPC-Lambda Custom Resource, cleaning up ENIs when deleted.
Parameters:
  VPCId:
    Description: VPC Id
    Type: AWS::EC2::VPC::Id
  SubnetId:
    Description: Private Subnet Id
    Type: AWS::EC2::Subnet::Id
Resources:
  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Lambda VPC security group
      VpcId: !Ref VPCId
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal: {Service: [lambda.amazonaws.com]}
          Action: ['sts:AssumeRole']
      Path: "/"
      ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
      Policies:
      - PolicyName: DetachNetworkInterface
        PolicyDocument:
          Version: 2012-10-17
          Statement:
          - Effect: Allow
            Action: ['ec2:DetachNetworkInterface']
            Resource: '*'
  AppendTest:
    Type: Custom::Split
    DependsOn: VPCDestroyENI
    Properties:
      ServiceToken: !GetAtt AppendItemToListFunction.Arn
      List: [1, 2, 3]
      AppendedItem: 4
  AppendItemToListFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Code:
        ZipFile: !Sub |
          var response = require('cfn-response');
          exports.handler = function(event, context) {
             var responseData = {Value: event.ResourceProperties.List};
             responseData.Value.push(event.ResourceProperties.AppendedItem);
             response.send(event, context, response.SUCCESS, responseData);
          };
      Timeout: 30
      Runtime: nodejs4.3
      VpcConfig:
        SecurityGroupIds: [!Ref SecurityGroup]
        SubnetIds: [!Ref SubnetId]
  VPCDestroyENIFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Code:
        ZipFile: !Sub |
          var response = require('cfn-response');
          var AWS = require('aws-sdk');
          exports.handler = function(event, context) {
            console.log("REQUEST RECEIVED:\n", JSON.stringify(event));
            if (event.RequestType != 'Delete') {
              response.send(event, context, response.SUCCESS, {});
              return;
            }
            var ec2 = new AWS.EC2();
            var params = {
              Filters: [
                {
                  Name: 'group-id',
                  Values: event.ResourceProperties.SecurityGroups
                },
                {
                  Name: 'description',
                  Values: ['AWS Lambda VPC ENI: *']
                }
              ]
            };
            console.log("Deleting attachments!");
            // Detach all network-interface attachments
            ec2.describeNetworkInterfaces(params).promise().then(function(data) {
              console.log("Got Interfaces:\n", JSON.stringify(data));
              return Promise.all(data.NetworkInterfaces.map(function(networkInterface) {
                var networkInterfaceId = networkInterface.NetworkInterfaceId;
                var attachmentId = networkInterface.Attachment.AttachmentId;
                return ec2.detachNetworkInterface({AttachmentId: attachmentId}).promise().then(function(data) {
                  return ec2.waitFor('networkInterfaceAvailable', {NetworkInterfaceIds: [networkInterfaceId]}).promise();
                }).then(function(data) {
                  console.log("Detached Interface, deleting:\n", networkInterfaceId);
                  return ec2.deleteNetworkInterface({NetworkInterfaceId: networkInterfaceId}).promise();
                });
              }));
            }).then(function(data) {
              console.log("Success!");
              response.send(event, context, response.SUCCESS, {});
            }).catch(function(err) {
              console.log("Failure:\n", JSON.stringify(err));
              response.send(event, context, response.FAILED, {});
            });
          };
      Timeout: 300
      Runtime: nodejs4.3
  VPCDestroyENI:
    Type: Custom::VPCDestroyENI
    Properties:
      ServiceToken: !GetAtt VPCDestroyENIFunction.Arn
      SecurityGroups: [!Ref SecurityGroup]
Outputs:
  Output:
    Description: output
    Value: !Join [",", !GetAtt AppendTest.Value]

_{Note: To create the VPC and Private Subnet required in the above example, you can use the AWS Quick Start Amazon VPC Architecture template.}