Automatically processing the Amazon SES notifications for bounce and complaint notifications

Question

We are using AWS SES for sending mails. Amazon SES sends bounce and complaint notifications through emails or AWS SNS. We would like to automatically process the bounce and complaint notifications (coming from email or AWS SNS) to extract the email ids, so that these emails can be removed from the original list.

One way to automate is to send these notifications to a topic in AWS SNS, then subscribe to the topic using AWS SQS and finally read the messages in the AWS SQS. SNS supports subscription over the following protocols - HTTP/HTTPS/EMail/EMail(JSON)/SMS/SQS. This is feasible, but I find it too cumbersome for a simple task of automatically processing the bounce and complaint notifications.

Is there any elegant way of tacking this problem?

---

I have found a blog entry from Amazon with the code in C#. Is there a better solution?

Answer 1

I find that directly subscribing to SNS using an HTTP endpoint is the most straightforward approach. You literally have to write only a few lines of code. Here's my django example:

def process(request):
    json = request.raw_post_data              
    js = simplejson.loads(json)
    info = simplejson.loads(js["Message"])
    type = info["notificationType"]           # "Complaint" or "Bounce"
    email = info["mail"]["destination"][0]


    # do whatever you want with the email

Answer 2

I think the way you describe IS probably the most elegant way. You already have very appropriate services in SNS and SQS that have associated SDK's in most major languages to allow you to do what you need easily. The hardest part is writing the code to update/remove the records in your mailing lists.

Answer 3

Through trial an error I have come up with this one - it is for Django but does a decent job for me.

First the models, then the request handler...

class ComplaintType:
    ABUSE = 'abuse'
    AUTH_FAILURE = 'auth-failure'
    FRAUD = 'fraud'
    NOT_SPAM = 'not-spam'
    OTHER = 'other'
    VIRUS = 'virus'


COMPLAINT_FEEDBACK_TYPE_CHOICES = [
    [ComplaintType.ABUSE, _('Unsolicited email or some other kind of email abuse')],
    [ComplaintType.AUTH_FAILURE, _('Unsolicited email or some other kind of email abuse')],
    [ComplaintType.FRAUD, _('Some kind of fraud or phishing activity')],
    [ComplaintType.NOT_SPAM, _('Entity providing the report does not consider the message to be spam')],
    [ComplaintType.OTHER, _('Feedback does not fit into any other registered type')],
    [ComplaintType.VIRUS, _('A virus was found in the originating message')]
]


class SES_Complaint(models.Model):
    subject = models.CharField(max_length=255)
    message = models.TextField()
    email_address = models.EmailField(db_index=True)
    user_agent = models.CharField(max_length=255)
    complaint_feedback_type = models.CharField(max_length=255, choices=COMPLAINT_FEEDBACK_TYPE_CHOICES)
    arrival_date = models.DateTimeField()
    timestamp = models.DateTimeField()
    feedback_id = models.CharField(max_length=255)

    created = models.DateTimeField(auto_now_add=True)
    modified = models.DateTimeField(auto_now=True)

    class Meta:
        verbose_name = 'SES Complaint'
        verbose_name_plural = 'SES Complaints'

    def get_reason(self):
        return self.get_complaint_feedback_type_display()


class BounceType:
    UNDETERMINED = 'Undetermined'
    PERMANENT = 'Permanent'
    TRANSIENT = 'Transient'


class BounceSubType:
    UNDETERMINED = 'Undetermined'
    GENERAL = 'General'
    NO_EMAIL = 'NoEmail'
    SUPPRESSED = 'Suppressed'
    MAILBOX_FULL = 'MailboxFull'
    MESSAGE_TOO_LARGE = 'MessageToolarge'
    CONTENT_REJECTED = 'ContentRejected'
    ATTACHMENT_REJECTED = 'AttachmentRejected'


BOUNCE_TYPE_CHOICES = [
    [BounceType.UNDETERMINED, _('Unable to determine a specific bounce reason')],
    [BounceType.PERMANENT, _('Unable to successfully send')],
    [BounceType.TRANSIENT, _('All retry attempts have been exhausted')],
]

BOUNCE_SUB_TYPE_CHOICES = [
    [BounceSubType.UNDETERMINED, _('Unable to determine a specific bounce reason')],
    [BounceSubType.GENERAL, _('General bounce. You may be able to successfully retry sending to that recipient in the future.')],
    [BounceSubType.NO_EMAIL, _('Permanent hard bounce. The target email address does not exist.')],
    [BounceSubType.SUPPRESSED, _('Address has a recent history of bouncing as invalid.')],
    [BounceSubType.MAILBOX_FULL, _('Mailbox full')],
    [BounceSubType.MESSAGE_TOO_LARGE, _('Message too large')],
    [BounceSubType.CONTENT_REJECTED, _('Content rejected')],
    [BounceSubType.ATTACHMENT_REJECTED, _('Attachment rejected')]
]


class SES_Bounce(models.Model):
    subject = models.CharField(max_length=255)
    message = models.TextField()
    bounce_type = models.CharField(max_length=255, choices=BOUNCE_TYPE_CHOICES)
    bounce_sub_type = models.CharField(max_length=255, choices=BOUNCE_SUB_TYPE_CHOICES)
    timestamp = models.DateTimeField()
    feedback_id = models.CharField(max_length=255)
    status = models.CharField(max_length=255)
    action = models.CharField(max_length=255)
    diagnostic_code = models.CharField(max_length=255)
    email_address = models.EmailField(db_index=True)

    created = models.DateTimeField(auto_now_add=True)
    modified = models.DateTimeField(auto_now=True, db_index=True)

    class Meta:
        verbose_name = 'SES Bounce'
        verbose_name_plural = 'SES Bounces'

    def get_reason(self):
        return '%s - %s' % (self.get_bounce_type_display(), self.get_bounce_sub_type_display())

And here is the request handler:

@csrf_exempt
def aws_sns(request):
    logger.debug('Incoming SNS')

    if request.method == 'POST':

        logger.debug('Incoming SNS is POST')

        sns_message_type = request.META.get('HTTP_X_AMZ_SNS_MESSAGE_TYPE', None)

        if sns_message_type is not None:

            logger.debug('Incoming SNS - %s', sns_message_type)

            json_body = request.body
            json_body = json_body.replace('\n', '')
            js = loads(json_body)

            if sns_message_type == "SubscriptionConfirmation":

                subscribe_url = js["SubscribeURL"]
                logger.debug('Incoming subscription - %s', subscribe_url)
                urllib.urlopen(subscribe_url)

            elif sns_message_type == "Notification":

                message = js.get("Message", None)
                message = message.replace('\n', '')
                message = loads(message)

                notification_type = message.get("notificationType", None)

                if notification_type == 'AmazonSnsSubscriptionSucceeded':
                    logger.debug('Subscription succeeded')

                elif notification_type == 'Bounce':

                    logger.debug('Incoming bounce')

                    bounce = message['bounce']
                    bounce_type = bounce['bounceType']
                    bounce_sub_type = bounce['bounceSubType']
                    timestamp = bounce['timestamp']
                    feedback_id = bounce['feedbackId']

                    bounce_recipients = bounce['bouncedRecipients']

                    for recipient in bounce_recipients:
                        status = recipient.get('status')
                        action = recipient.get('action')
                        #diagnostic_code = recipient['diagnosticCode']
                        email_address = recipient['emailAddress']

                        SES_Bounce.objects.filter(email_address=email_address).delete()

                        SES_Bounce.objects.create(
                            message=message,
                            bounce_type=bounce_type,
                            bounce_sub_type=bounce_sub_type,
                            timestamp=timestamp,
                            feedback_id=feedback_id,
                            status=status,
                            action=action,
                            #diagnostic_code=diagnostic_code,
                            email_address=email_address
                        )

                elif notification_type == 'Complaint':

                    logger.debug('Incoming complaint')

                    complaint = message['complaint']

                    user_agent = complaint.get('userAgent')
                    complaint_feedback_type = complaint.get('complaintFeedbackType')
                    arrival_date = complaint.get('arrivalDate')

                    timestamp = complaint['timestamp']
                    feedback_id = complaint['feedbackId']
                    recipients = complaint['complainedRecipients']

                    for recipient in recipients:
                        email_address = recipient['emailAddress']

                        SES_Complaint.objects.filter(email_address=email_address).delete()

                        SES_Complaint.objects.create(
                            #subject=subject,
                            message=message,
                            email_address=email_address,
                            user_agent=user_agent,
                            complaint_feedback_type=complaint_feedback_type,
                            arrival_date=arrival_date,
                            timestamp=timestamp,
                            feedback_id=feedback_id
                        )

                else:
                    logger.exception('Incoming Notification SNS is not supported: %s', notification_type)

            return HttpResponse()
        else:
            logger.exception('Incoming SNS did not have the right header')

            for key, value in request.META.items():
                logger.debug('Key: %s - %s', key, value)

    else:
        logger.exception('Incoming SNS was not a POST')

    return HttpResponseBadRequest()

Answer 4

Recently, I was able to get this working using an HTTP Endpoint via SNS. I use python/django to consume the notification. You have to process the subscription message first before you consume the notifications; you can read about subscriptions in the SNS documentation.

I think if you have a smaller application that doesn't send to many emails; http endpoint should work fine. This code requires that you have a notification model created.

#process an amazon sns http endpoint notification for amazon ses bounces and complaints
@csrf_exempt
def process_ses_notification(request):

    if request.POST:

        json_body = request.body
        #remove this control character(throws an error) thats present inside the test subscription confirmation
        js = loads(json_body.replace('\n', ''))

        if js["Type"] == "SubscriptionConfirmation":

             subscribe_url = js["SubscribeURL"]
             urllib.urlopen(subscribe_url)
             return HttpResponse(status=200)

    elif js["Type"] == "Notification":

        #process message from amazon sns
        arg_info = loads(js["Message"]) # may need to use loads(js["Message"]) after testing with amazon
        arg_notification_type = arg_info["notificationType"]

        if arg_notification_type == 'Bounce':
            #required bounce object fields
            arg_emails=arg_info["bounce"]["bouncedRecipients"]
            arg_notification_subtype=arg_info["bounce"]["bounceType"]
            arg_feedback_id=arg_info["bounce"]["feedbackId"]
            arg_date_recorded=arg_info["bounce"]["timestamp"]
        elif arg_notification_type == 'Complaint':
            #required complaint object fields
            arg_emails=arg_info["complaint"]["complainedRecipients"]
            arg_feedback_id=arg_info["complaint"]["feedbackId"]
            arg_date_recorded=arg_info["complaint"]["timestamp"]
            #check if feedback type is inside optional field name
            if "complaintFeedbackType" in arg_info["complaint"]:
                arg_notification_subtype=arg_info["complaint"]["complaintFeedbackType"]
            else:
                arg_notification_subtype=""
        else:
            HttpResponse(status=400)

        #save notifications for multiple emails
        for arg_email in arg_emails:
            notification = SES_Notification(info=json_body, notification_type=arg_notification_type, 
                                            email=arg_email["emailAddress"], notification_subtype=arg_notification_subtype, 
                                            date_recorded=arg_date_recorded, feedback_id=arg_feedback_id)
            notification.save()
        return HttpResponse(status=200)

return HttpResponse(status=400)  

Answer 5

all the answers above are great, but just a small and important addition:

you first need to verify that the request is from amazon sns: (as described at Verifying the Signatures of Amazon SNS Messages)

for the python code that validate the signature - a good example is here