Google Apps Script: Email quotas and failed messages

Question

I have a mail merge type Apps Script projetct that sends out emails to multiple recipients, listed in a spreadsheet. For about 5-10% of the emails sent out in a batch, I get the following error:

Exception: Service invoked too many times in a short time: premium gmail expensive rateMax. Try Utilities.sleep(1000) between calls

The sendEmail() function is called for each email to be sent from a for loop within a modal dialog window, and at the end of the apps script function, I call SpreadsheetApp.flush(); as well as Utilities.sleep(1000); to try to mitigate the issue. I have also tried increasing sleep time, but that hasn't solved the issue either.

I have not managed to identify any other factors that would help reproduce the issue - there's no correlation between the actual recipient or any variables within the messages sent out.

Any tips?

EDIT: I am nowhere near exploiting the quotas, and rerunning the same script for any failed messages solves the issue.

I suspect that execution happens at arbitrary times despite the sleep() and flush() prompts – but it's a puzzle to me why.

EDIT 2: Code

Core code used in panel:

function doSendEmail(){
  sentEmailCounter = 0;
  var senderName = document.getElementById('senderName').value;
  var senderEmail = document.getElementById('senderEmail').value;
  var subject = document.getElementById('subject').value;
  var message = document.getElementById('message').value;
  var draftId = document.getElementById('draft').value;
  var bcc = document.getElementById('bcc').value;
  var cc = document.getElementById('cc').value;

  //Check if CC need to be sent
  var ccMe = '';
  var ccBox = document.getElementById('ccMe');
  if (ccBox.checked){
    ccMe = <?=me?>;
  }         

  //iterate through recipients
  for (var i = 0; i < recipientsData.length; i++){
    var recipient = recipientsData[i]["email"];
    if (recipientsData[i]["selected"] == true){
      var num = recipientsData[i]["num"];
      var firstName = recipientsData[i]["firstName"]
      var familyName = recipientsData[i]["familyName"]
      var preferredName = recipientsData[i]["preferredName"]
      var title = recipientsData[i]["title"]
      var userName = recipientsData[i]["userName"]
      var tempPassword = recipientsData[i]["tempPassword"]
      var newEmail = recipientsData[i]["newEmail"]
      google.script.run.withSuccessHandler(sendEmailSuccess).withFailureHandler(sendEmailFailure).sendEmail(recipient,senderName,senderEmail,subject,message,cc,ccMe,bcc,draftId,composeOrDraft,num,firstName,familyName,preferredName,title,userName,tempPassword,newEmail);
      }
    }
  }
  
  function sendEmailSuccess(returnObj){
    sentEmailCounter++
    if (returnObj.status == true){
      successes++
    }
    else {
      failures++
    }
  }

  function sendEmailFailure(failureMessage){
    // Not relevant here     
  }

GAS code:

function sendEmail(recipient,senderName,senderEmail,subject,message,cc,ccMe,bcc,draftId,composeOrDraft,num,firstName,familyName,preferredName,title,userName,tempPassword,newEmail){
  Utilities.sleep(2000);


  //Try sending message
    try{ 
      //If using composition
      if (composeOrDraft == 'compose'){

      GmailApp.sendEmail(recipient, subject, '', {
        name:senderName,
        from:senderEmail,
        htmlBody:htmlstring,
        cc:allCc,
        bcc:bcc,
      })
    }

    //If using a draft
    else {
      var allDrafts = GmailApp.getDrafts()
      allDrafts.forEach(function(draft){
        if (draft.getMessage().getId() == draftId){

          //Get subject
          var subject = draft.getMessage().getSubject()
     
          //Get htmlbody and replace placeholders
          var htmlBody = draft.getMessage().getBody()
        
          GmailApp.sendEmail(recipient, subject, '', {
            name:senderName,
            from:senderEmail,
            htmlBody:htmlBody,
            cc:allCc,
            bcc:bcc
          })              
        };
      });
    }
    success = true;
    //Record status on Sheet
    if (num >= 0){
      feedbackCell.setValue('EMAIL SENT')
      feedbackCell.setNote('EMAIL SENT to '+recipient+' on '+ new Date())
    }
  }
  catch(e){
    if (num >= 0){
      feedbackCell.setValue('FAILED')
      feedbackCell.setNote(e)
    }
  }
  var returnObj = {status:success};
  SpreadsheetApp.flush();
  return returnObj;
}

Answer 1

SOLUTION

Instead of using Utilities.sleep(1000), use a custom async await function.

FINDINGS - TLDR

I have tried quite a few things, including building an exponential backoff solution, as suggested by Matteo Randwolf in the comments, which eventually offered a solution, but only through many failed attempts. So it does the job, but jsut bypasses the original issue.

In the meantime, however, I also realised the following:

• Fail/Error messages are returned by GmailApp.sendEmail() typically when the script sends a draft from the user's inbox, which has large attachments. (The error still reads as per the original post, Exception: Service invoked too many times in a short time: premium gmail expensive rateMax., but it doesn't seem to be time triggered. In comparison, the same script working with the same number of recipients but with message bodies that include very small texts completes without any issues.
• Messages are sometimes actually sent, despite the Fail/Error feedback. There seems to be some miscommunication within GAS, which results in the wrong status message sent back on execution.
• Utilities.sleep(1000) (or 2000 or 4000 for that matter) just doesn't work in this case. Logs and visual feedback prove that it doesn't stop the execution of the script, even though I included both that and a SpreadsheetApp.flush() function within each iteration, right after each other.
• The only thing that actually works is swapping Utilities.sleep() to an async await function as below.

FUNCTION TO USE INSTEAD OF Utilities.sleep(1000)

• Make your function an async function()
• Instead of Utilities.sleep(), call a custom sleep function sleep() as below

Sleep function that works:

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

If anyone understands why, do let me know. :)