AWS - Moving data from one S3 bucket to another with CloudFormation

Question

I'm trying to create a stack with CloudFormation. The stack needs to take some data files from a central S3 bucket and copy them to it's own "local" bucket.

I've written a lambda function to do this, and it works when I run it in the Lambda console with a test event (the test event uses the real central repository and successfully copies the file to a specified repo).

My current CloudFormation script does the following things:

Creates the "local" S3 bucket
Creates a role that the Lambda function can use to access the buckets
Defines the Lambda function to move the specified file to the "local" bucket
Defines some Custom resources to invoke the Lambda function.

It's at step 4 where it starts to go wrong - the Cloudformation execution seems to freeze here (CREATE_IN_PROGESS). Also, when I try to delete the stack, it seems to just get stuck on DELETE_IN_PROGRESS instead.

Here's how I'm invoking the Lambda function in the CloudFormation script:

"DataSync": {
    "Type": "Custom::S3DataSync",
    "Properties": {
        "ServiceToken": { "Fn::GetAtt" : [ "S3DataSync", "Arn" ] },
        "InputFile": "data/provided-as-ip-v6.json",
        "OutputFile": "data/data.json"
    }
},
"KeySync1": {
    "Type": "Custom::S3DataSync",
    "Properties": {
        "ServiceToken": { "Fn::GetAtt" : [ "S3DataSync", "Arn" ] },
        "InputFile": "keys/1/public_key.pem"
    }
},
"KeySync2": {
    "Type": "Custom::S3DataSync",
    "Properties": {
        "ServiceToken": { "Fn::GetAtt" : [ "S3DataSync", "Arn" ] },
        "InputFile": "keys/2/public_key.pem"
    }
}

And the Lambda function itself:

exports.handler = function(event, context) {
    var buckets = {};
    buckets.in = {
        "Bucket":"central-data-repository",
        "Key":"sandbox" + "/" + event.ResourceProperties.InputFile
    };   
    buckets.out = {       
        "Bucket":"sandbox-data",       
        "Key":event.ResourceProperties.OutputFile || event.ResourceProperties.InputFile   
    };   

    var AWS = require('aws-sdk');   
    var S3 = new AWS.S3();   

    S3.getObject(buckets.in, function(err, data) {       
        if (err) {           
            console.log("Couldn't get file " + buckets.in.Key);           
            context.fail("Error getting file: " + err)       
        }       
        else {           
            buckets.out.Body = data.Body;           
            S3.putObject(buckets.out, function(err, data) {               
                if (err) {                   
                    console.log("Couln't write to S3 bucket " + buckets.out.Bucket);                   
                    context.fail("Error writing file: " + err);               
                }               
                else {                   
                    console.log("Successfully copied " + buckets.in.Key + " to " + buckets.out.Bucket + " at " + buckets.out.Key);                   
                    context.succeed();               
                }           
            });       
        }   
    });
}

This seems like a complex, and fragile, solution for moving data from one bucket to another. Have you considered simply using an awscli script to create the target bucket and move the data? Your solution, as it stands, will fail if any object is larger than the Lambda /tmp space (500MB) or exceeds the Lambda timeout. It also will likely fail if there are transient network issues. And it will be slower than necessary. — jarmod

James James · Accepted Answer · 2015-11-02T17:36:58

Your Custom Resource function needs to send signals back to CloudFormation to indicate completion, status, and any returned values. You will see CREATE_IN_PROGRESS as the status in CloudFormation until you notify it that your function is complete.

The generic way of signaling CloudFormation is to post a response to a pre-signed S3 URL. But there is a cfn-response module to make this easier in Lambda functions. Interestingly, the two examples provided for Lambda-backed Custom Resources use different methods:

Walkthrough: Refer to Resources in Another Stack - uses the cfn-response module
Walkthrough: Looking Up Amazon Machine Image IDs - uses pre-signed URLs.

AWS - Moving data from one S3 bucket to another with CloudFormation

2 Answers