Specifying delete stack order in cloudformation

Question

I am creating AWS stack with cloudformation. I am creating EBS volumes as part of the Cloud Formation and attaching them to EC2 in bootstrap.

The creation part works fine. However, while deleting the stack, Cloudformation is trying to delete the EBS volumes first. As the volumes are already attached, it is failing to delete and deleting remaining resources.

Because of this, EBS volumes are staying there unless delete manually or use delete stack again.

Is there a way i can specify the order when Deletion(only) for the resources in cloudformation template.?

Answer 1

The only mechanism you have to control the order of CloudFormation operations is using DependsOn. But that won't solve this particular problem.

The problem you describe here occurs because CloudFormation doesn't know that the volume is attached: you attached it through a separate mechanism (as you described, using EC2 bootstrap, which I assume would be something like an aws command on the EC2 Instance User Data script, for example).

What you could do, instead, is have CloudFormation attach the volume for you. That way, CloudFormation knows that the volume has been attached, and it knows that it has to detach the volume, too.

To do that, you need to use a resource of type AWS::EC2::VolumeAttachment. A YAML snippet for that would be something like:

Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      ...
  MyVolume:
    Type: AWS::EC2::Volume
    Properties:
      ...
  MyVolumeAttachment:
    Type: AWS::EC2::VolumeAttachment
    Properties:
      Device: /dev/sdf
      InstanceId: !Ref MyInstance
      VolumeId: !Ref MyVolume

There are quite a few *Attachment resource types on CFN, for this exact purpose: you let CFN attach the resource to you, you specify a Ref to the resources that participate in the attachment, so CFN knows the "order" (ie, it first create the 2 resources, then attach them, or when deleting, it first detaches, then deletes both resources) and CFN can then handle the entire process for your.

Answer 2

I've solved this problem by terminating the instance using a custom lambda function. The function depends on VolumeAttachment resource and is being called before CloudFormation tries to detach the volume. It does nothing on stack creation, but terminates the instance on stack deletion:

Resources:
  ...
  TerminateInstance:
    Type: Custom::InstanceTermination
    DependsOn: VolumeAttachment1
    Properties:
      ServiceToken: !GetAtt TerminateInstanceFunction.Arn
      InstanceId: !Ref EC2Instance
  TerminateInstanceFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.handler
      Role: !GetAtt TerminateInstanceLambdaExecutionRole.Arn
      Runtime: nodejs4.3
      Timeout: 30
      Code:
        ZipFile: !Sub |
          var aws = require("aws-sdk");
          var response = require('cfn-response');

          exports.handler = function(event, context) {
              console.log("request received:\n" + JSON.stringify(event));

              var physicalId = event.PhysicalResourceId;

              function success(data) {
                  data = data || {}
                  console.log('SUCCESS:\n', data);
                  return response.send(event, context, response.SUCCESS, data, physicalId);
              }

              function failed(err) {
                  console.log('FAILED:\n', err);
                  return response.send(event, context, response.FAILED, err, physicalId);
              }

              // ignore non-delete requests
              if (event.RequestType !== 'Delete') {
                  console.log('Non-delete request is ignored');
                  return success();
              }

              var instanceId = event.ResourceProperties.InstanceId;
              if (!instanceId) {
                  return failed('InstanceId required');
              }

              var ec2 = new aws.EC2({region: event.ResourceProperties.Region});

              ec2.terminateInstances({InstanceIds: [instanceId]})
              .promise()
              .then((data) => {
                  console.log('"terminateInstances" Response:\n', JSON.stringify(data));
                  success();
              })
              .catch((err) => failed(err));
          };
  TerminateInstanceLambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
              - lambda.amazonaws.com
          Action:
            - 'sts:AssumeRole'
      Path: /
      ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      - arn:aws:iam::aws:policy/service-role/AWSLambdaRole
      Policies:
      - PolicyName: EC2Policy
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
              - 'ec2:TerminateInstances'
              Resource: ['*']