How do I properly encode audio bytes when writing to Google Cloud Storage

Question

I am trying to read an audio file from Twilio and save it in Google Cloud Storage (GCS). I've gotten to the point where I can read and write bytes, but what I end up with in GCS is no longer a valid .mp3 file and almost twice the size of a manually downloaded .mp3 file (from Twilio). This is all in Node.js using Firebase server. Here's the code:

const recordingResults = await Recording.getTwilioRecording(req.body.RecordingSid)
if (recordingResults){
  await Recording.saveRecording(req.body.RecordingSid,RECORDING_TYPE_MESSAGE,recordingResults['data'])
}

exports.getTwilioRecording = async function(recordingSid){
  const url = Config.TWILIO_API_BASE_URL+Config.TWILIO_ACCOUNT_SID+'/Recordings/'+recordingSid+'.mp3'
  const promise =  axios({
    'method':'get',
    'url':url,
    'headers': {
      'Content-Type': 'audio/mpeg'
      }
    })
  const data = await promise.then(mp3 =>{
    return mp3
  }).catch(err=>{
    console.log('Error connecting to Twilio to get recording',err.message)
    return false
  })
  return data
}

exports.saveRecording = async function(recordingSid, recordingType, data){

  const projectId = Config.GOOGLE_PROJECT_ID
  const keyFilename = Config.GOOGLE_STORAGE_SA_JSON
  const storage = new Storage({projectId, keyFilename});
  const myBucket = storage.bucket(Config.GOOGLE_STORAGE_ROOT_DIR);
  const gscname = '/recordings/'+recordingType+'/'+recordingSid+'.mp3'
  const file = myBucket.file(gscname);
  file.save(data)
}

Just a heads up: file.save() without a callback returns a promise so you either have to await it or handle the promise with file.save().then(...).catch(...). I also notice that in the const data = line, you are handling the promise twice with await and .then() which could explain why the file in GCS ends up being twice as large as the original one. In any case, can you try these steps and let me know if the issue persist? — JKleinne

JohnL JohnL · Accepted Answer · 2019-08-13T19:53:55

After numerous attempts at "getting" audio content, and then "pushing" it, I finally tried the method that appears most often in the examples. That is piping from the get URL directly to Google Cloud Storage's createWriteStream method. I wish I could say why other methods don't work. Perhaps someone else can educate us.

function someFunction(recordingSid, recordingType){ 
  const url =  Config.TWILIO_API_BASE_URL+Config.TWILIO_ACCOUNT_SID+'/Recordings/'+recordingSid+'.mp3'
  const projectId = Config.GOOGLE_PROJECT_ID
  const keyFilename = Config.GOOGLE_STORAGE_SA_JSON
  const storage = new Storage({projectId, keyFilename});
  const myBucket = storage.bucket(Config.GOOGLE_STORAGE_ROOT_DIR);
  const gscname = '/recordings/'+recordingType+'/'+recordingSid+'.mp3'
  const file = myBucket.file(gscname);
  axios({
    method: 'get',
    url: url,
     responseType:'stream'
    })
  .then(res => {
    res.data.pipe(file.createWriteStream({
      'contentType':'audio/mp3',
      'resumable':false,
    }))
    .on('error',(err)=>{console.log(err.message)})
    .on('success',()=>{console.log('success')})
    return null
  })
  .catch(err => console.log(err));
}

How do I properly encode audio bytes when writing to Google Cloud Storage

1 Answers