How to tell if an email sent via Gmail REST API has bounced?

Question

I'm sending emails via Gmail API and would like to know when the messages bounce. How can I do this?

As I understand it, bounced emails usually contain some sort of header indicating a bounce such as:

X-Failed-Recipients: [email protected]

However, there doesn't seem to always be a header indicating what original messageID it was that bounced.

I was thinking of the following plan, but there are so many holes that I think I must be approaching this wrong.

1. Send email (to a failed email) via Gmail API ---> It goes through successfully
2. Receive the email bounced email inbox
3. Scan email for emails containing bounced headers
4. Try to figure out which original email it was that bounced.

Problems

• Gmail api returns Gmail Message ID, not the actual message ID
• Have to continuously monitor/poll inbox to see if there's bounced emails
• Is it even possible to search by existence of a header?
• Each email provider out there seems to have different bounce headers
• The headers might not indicate original Message ID

A couple other ideas I had were to:

• Search for emails with the string "Undeliverable" in the subject?
• not use gmail rest api for sending as bounce tracking is not feasible. perhaps use SMTP api instead?

Answer 1

Messages that gets bounced when sent through the Gmail API gets a response from the mailer deamon ([email protected]). You could continually check the user's messages to see if a new message from the daemon has been received.

Make sure to store the timestamp in seconds since your last check, so you don't get any nasty duplicates next time around.

query = from:[email protected] after:<TIME_SINCE_EPOCH_IN_SECONDS>

GET https://www.googleapis.com/gmail/v1/users/me/messages?q=from%3Amailer-daemon%40googlemail.com+after%3A1437055051&access_token={YOUR_API_KEY}

Response:

{
 "messages": [
  {
   "id": "14e97f7ed03b7e88",
   "threadId": "14e97f7ea9b794a4"
  },
 ]
}

I got a bounce! Let's fetch the entire mail and decode it and get the Message-ID you were alluding too.

GET https://www.googleapis.com/gmail/v1/users/me/messages/14e97f7ed03b7e88?fields=payload%2Fbody%2Fdata&access_token={YOUR_API_KEY}

Response:

{
 "payload": {
  "body": {
   "data": "RGVsA0K..."
  }
 }
}

Converting the mail to regular base64 from its URL safe version (replace all "-" with "+" and "_" with "/"), and base64-decoding it we get:

atob("RGVsA0K...".replace(/\-/g, '+').replace(/\_/g, '/'));

Decoded mail:

"Delivery to the following recipient failed permanently:

     [email protected]

Technical details of permanent failure: 
DNS Error: Address resolution of sadsads.asdsad. failed: Domain name not found

----- Original message -----

.
.
.

Received: from 292824132082.apps.googleusercontent.com named unknown by
 gmailapi.google.com with HTTPREST; Thu, 16 Jul 2015 13:44:43 -0400
from: [email protected]
Date: Thu, 16 Jul 2015 13:44:43 -0400
Message-ID: <[email protected]>
Subject: Subject Text
To: [email protected]
Content-Type: text/plain; charset=UTF-8

The actual message text goes here

Here we have the Message-ID! Let's get the bounced email!

query = rfc822msgid:<[email protected]>;

GET https://www.googleapis.com/gmail/v1/users/me/messages?q=rfc822msgid%3A%3CCADsZLRzOs1wT4B5pgR7oHHdbjkQhuaCQQs8CEckhLwVw73QFEQ%40mail.gmail.com%3E&key={YOUR_API_KEY}

Response:

{
 "messages": [
  {
   "id": "14e97f7ea9b794a4", // <-- Here is the message that bounced!
   "threadId": "14e97f7ea9b794a4"
  }
 ],
}

Answer 2

When you send a message via

service.users().messages().send(userId, message).execute();

It will return a Message. You can use its threadId to check if you got any reply to that message.

Here's an easy way of checking if it bounced (give a 1 second delay after sending):

public static boolean isBounced(Gmail service, String threadId) throws IOException {
    List<Message> list = service.users().messages().list("me")
                        .setQ("[email protected]")
                       .execute().getMessages();

    return list.stream().anyMatch(msg -> msg.getThreadId().equals(threadId));
}

Answer 3

This is how you can do it using backend technologies and trying to follow current documentation guides. Since it appears after: and before: tags only support a date not date + time. So whilst above EPOCH time may have worked on earlier code more work is required in current API. Some of this below yet to be added to a test project I have on github. But to give an idea:

We are asking it to look back for any bounced messages older than a day, then parsing headers to find received date - converting string to real date and comparing -20 minutes ago from results anything older not added to the listing

List verifyBounceList (Gmail service) {
        List foundResults=[]

        Date date = new Date()
        use (groovy.time.TimeCategory) {
            date= date -20.minute
        }
        Date yesterday = new Date()-1
        def bounceRecords = listMessagesMatchingQuery(service,'me','from:[email protected] after:'+yesterday.format('YYYY/MM/dd'))
        bounceRecords?.each {
            Message message = getMessage(service,'me',it.id)
            String receivedDateString = message.getPayload().headers?.find{it.name=='Received'}.value.split(';')[1].trim()
            SimpleDateFormat df = new SimpleDateFormat('EEE, dd MMM yyyy HH:mm:ss z (Z)')
            Date receivedDate=df.parse(receivedDateString)
            if (receivedDate>date) {
                foundResults<<[bouncedRecord:it,mapRecord:message]
            }
        }
        return foundResults
    }

Message getMessage(Gmail service, String userId, String messageId) throws IOException {
    Message message = service.users().messages().get(userId, messageId).execute()
    ///System.out.println("Message snippet: " + message.getSnippet())
    return message
}
    /**
 * Simply does a query in given mailbox 
 * used to query for mail failures
 * @param service
 * @param userId
 * @param query
 * @return
 * @throws IOException
 */
List<Message> listMessagesMatchingQuery(Gmail service, String userId, String query) throws IOException {
    ListMessagesResponse response = service.users().messages().list(userId).setQ(query).execute()

    List<Message> messages = new ArrayList<Message>()
    while (response.getMessages() != null) {
        messages.addAll(response.getMessages())
        if (response.getNextPageToken() != null) {
            String pageToken = response.getNextPageToken()
            response = service.users().messages().list(userId).setQ(query).setPageToken(pageToken).execute()
        } else {
            break;
        }
    }
    for (Message message : messages) {
        //System.out.println(message.toPrettyString());
    }

    return messages;
}

Above is returned as a list iterating through returned results:

<g:if test="${instance.size()>0}"> 
        <h2>Bounces found : ${instance.size()}</h2><br/>
        <div class="errors">
        <g:each in="${instance}" var="failed">
            ${failed.bouncedRecord} --> ${failed.mapRecord?.id} ${failed.mapRecord?.getSnippet()} ${ }<br/>
            <g:each in="${failed.mapRecord.getPayload().headers}" var="a">
            ${a }<br/>---
            </g:each>
        </g:each>