I'm modifying the AWS Lambda with Amazon S3 tutorial found here so that it will simply--
- read a JSON file (the content index) already resident in that bucket,
- update it with the newly created Key (a new "folder" in that bucket that triggered this Lambda),
- and then save (put) the updated JSON file back.
CLARIFICATION: The bucket should only trigger the Lambda function when a folder object is created in it. The content index (index.json) is already resident in the bucket. So, bucket will have folders (e.g., {folder-1, folder-2, folder-n}) and index.json. Every time a new folder is added, it's added to the JSON array.
To be specific, in my case I have a root (destination) bucket that will have a series of folders created by Elemental MediaConvert. Each folder represents a new video; and within each folder are the different formats that may be served to different devices. Back at the root level I have index.json which is an array of these video folders; it is the content index. Now, I could modify the Lambda that is part of the MediaConvert flow.. but I'll consider that at another time. Here, I just want to trigger a new S3 Lambda every time MediaConvert writes a new video folder... which is just some random GUID thing.
I'm learning Node JS and this is the first time I've used some of the structure and calls you see here. (I'm at least aware that this form is cleaner and clearer than using callbacks.)
Since it's going to be tricky (for me!) to test this as a Lambda function, would someone kindly point me at any obvious mistakes??
Also, would someone provide me some direction on how to test this Lambda function manually using Amazon S3 event data (prior to configuring the actual bucket to publish the required event)? I would think I need the event.json to specify the name of that newly created folder so it can be added to my index.json which is also sitting in the same bucket.
Here's my code:
// dependencies
var async = require('async');
var AWS = require('aws-sdk');
var util = require('util');
// constants
//const DEST_FOLDER = 'my-triggering-bucket';
const CONTENT_INDEX_FILENAME = 'index.json';
// get reference to S3 client
var s3 = new AWS.S3();
exports.handler = function(event, context, callback) {
// Read options from the event.
// Need the new folder (key) that's been added to the bucket...
console.log("Reading options from event:\n", util.inspect(event, {depth: 5}));
// I assume this is the triggering bucket...
var triggerBucket = event.Records[0].s3.bucket.name;
// And I assume this is the folder that was added and caused the trigger
var newKey = decodeURIComponent(event.Records[0].s3.object.key);
const indexKey = CONTENT_INDEX_FILENAME;
// Get the content index and add the newly created dir to it
async.waterfall([
function download(next) {
// Download the content index from S3 into a buffer.
s3.getObject({
Bucket: triggerBucket,
Key: indexKey
},
next);
},
function update(response, next) {
// Update the content index with the new dir that was added
console.log('UPDATE...');
var obj = JSON.parse(response);
obj.push(newKey);
var jsonStr = JSON.stringify(obj);
next(null, jsonStr.ContentType, jsonStr);
},
function upload(contentType, data, next) {
// Stream the updated content index back
s3.putObject({
Bucket: triggerBucket,
Key: indexKey,
Body: data,
ContentType: contentType
},
next);
}
], function (err) {
if (err) {
console.error('error: ' + err);
} else {
console.log('Success);
}
callback(null, "message");
}
);
};
UPDATE
I've abandoned this approach in favor of updating the content index via another means that does not risk runaway execution of my Lambda. I have discovered first-hand that it is not a good idea to try to trigger a specific createObject event in a bucket when one's design does not provide a solid event notification filter. (I could not filter on a suffix of a simple /
.) Also, I was expecting a single folder key object create event to trigger my Lambda, but in reality other folders and keys created inside that new root level folder ended up triggering my Lambda too. So, this sent me into the video conversion workflow to modify the Lambda that notifies the successful completion of the workflow such that it updates my content index.
PUT
actions that can cost real $ if not detected promptly. Ensure, at minimum, that the new object will not trigger inclusion of itself. – Michael - sqlbot