Uploading an audio file using Node.js Streams in AWS Lambda

Question

So I'm trying to retrieve an mp3 audio file (approximately 9 MB) from s3, pipe the data to a write stream, and then upload to another destination s3 bucket using a readStream from the /tmp/ file. This is a lambda function that receives an s3 upload event and attempts to write the data from the created object to another bucket.

const fs = require('fs');
const AWS = require('aws-sdk');

const s3 = new AWS.S3();

exports.handler = (event, context, callback) => {
    var sourceBucket = event.Records[0].s3.bucket.name;
    var sourceKey = event.Records[0].s3.object.key;

    var getParams = {
        Bucket: sourceBucket,
        Key: sourceKey
    };

    const inputFilename = '/tmp/' + sourceKey;

    //writing and reading streams
    const writeStream = fs.createWriteStream(inputFilename);
    s3.getObject(getParams).createReadStream().pipe(writeStream);

    var putParams = {
        Body: fs.createReadStream(inputFilename),
        Bucket: "example-destination-bucket",
        Key: 'transfer-' + sourceKey        
    };

    s3.upload(putParams, function(err, data) {
        if (err) console.log(err, err.stack); // an error occurred
        else     console.log('logging data' + data);  // successful response
    });
};

This results in the key successfully being put to the s3 bucket, but the file uploaded is 0 bytes in size. Any idea why this may result in an empty upload?

Banjo Obayomi Banjo Obayomi · Accepted Answer · 2018-09-19T18:43:18

The file needs to be downloaded, which takes some time, so you need to use the file.on('finish') call like this..

  const writeStream = fs.createWriteStream(inputFilename);
  s3.getObject(getParams).createReadStream().pipe(writeStream);
  writeStream.on('finish', function() { 

  //upload to S3 code
  }

Uploading an audio file using Node.js Streams in AWS Lambda

2 Answers