Trace not delivered / bounced email with PHP, phpmailer and Message-ID

1
votes

I have a PHP web app with a section to send invoice by email to all partners. Each partner could have more than 1 email.

For example: COMPANY-1 with emails: email_1@company1.com and email_2@company1.com

I send all with phpmailer.

I save into my database the send-response from server mail ( in this case gmail).

If sent, i'll save true, else false - foreach email.

But some emails sometimes are incorrect (due to error typing when insert) or are dismissed by the company (in this case the company do not inform us and we still send to this deleted email) and I like to retrieve these "problems" using web app.

(If I login into google webmail I can see the email returned for each sent to an not existing email, with some advice/error code inside... but i don't know how to retrieve using PHP)

I tried using

$inbox = imap_open($hostname,$username,$password);

$emails = imap_search($inbox,'ALL');

This script works, but using ALL i get timeout.

I don't know the best method to retrieve these email of "NOT DELIVERED EMAILS FOR SOME PROBLEMS".

My first ideas was to set a custom Message-ID in the header when I send:

$messageID = "<" . md5($_SERVER['HTTP_HOST'].(idate("U")-1000000000).uniqid()).'-'.idate("U") . '@' . $_SERVER['HTTP_HOST'] . '>';
$mail->MessageID = $messageID;

and then in my script, using a command like:

$messageID = retrieve_from_database(...);
$emails = imap_search($inbox,'HEADER ' . $messageID);

but it doesn't exist a query HEADER

I'd like in my web app to show, for each email sent, eventually problem of bounced or undelivered but I don't know how and what I need to "ask" to imap_search to retrieve responses associated to the original sent email.

I read criterias list from docs, but I don't find anything usefull I think: https://www.php.net/manual/en/function.imap-search.php

Example of a scenario I need:

  1. I send an email with Message-ID: <test@testXYZ> from my_email@mywebapp.com to email_not_existing@test.com. The server response true cause sent is done.

  2. I insert a new entry on my database table emails_sent with send result true: INSERT INTO emails_sent (ID, email_dest, sent, MessageID, when) VALUES (455, 'email_not_existing@test.com', TRUE, $messageID, '2019-05-30 8:00:00')

  3. The gmail daemon reply to my_email@mywebapp.com with a message of undelivered email

  4. In my web app, when I go into page of sent emails, foreach ID from emails_sent table, i need to search on imap eventually response email with errors. In this example case when I ask for ID 455, I need to search on imap_search messages replyes associated with my email sent with MessageID assigned by me (or other method i don't know....)

1
phpphpmailerimap
First of all, I’d suggest you use a dedicated account to handle those bounces - so that you don’t have to fish them out of a heap of other mails handled by the same account. Specify a Return-Path for that purpose. – 04FS
Secondly, if you are on a server that you have enough control over (and not some shared hosting situation), look into piping incoming emails directly to a script. That way, you don’t have to bother with periodically reading the contents of imap folders, but your script can process each mail directly when it arrives. – 04FS
HI @04FS, unfortunately i'm on shared hosting and my solution is to interrogate periodically the imap with a script. For example every 10 minutes. OK If I create a new mailbox like: bounced@mywebapp.com and set a 'Return-Path' I can search inside this mailbox: probably i don't have timeout problem (until mailbox has not a lot email yet)...but my question remain: how to associate a returned email with the original email sent? Another question, but i'm not sure: 'Return-Path' I think was deprecated by phpmailer script. – Giuseppe Lodi Rizzini
MUAs (like PHPMailer) should never set a return-path header; it's not their job. You should specify an envelope sender which is used at the SMTP level as the MAIL FROM address; the receiving mail server then adds that to the message as a return-path header. This is also where bounces will be sent. – Synchro

1 Answers

2
votes

I would advise you to get away from shared hosting and get a proper server (try scaleway.com). As you're discovering, it means you don't get sufficient control.

I'd recommend against using IMAP as your database - as you're finding it's not good enough for the kind of searching you want to do. If you import all the bounced messages into your own DB, you're free to parse and search in whatever way you like, and it's also easy to get all messages from IMAP.

The most efficient way to handle bounces is to get your SMTP server to pipe bounce messages directly into a script. This will give you the complete bounced message, along with environment variables that tell you more about the delivery. That of course requires that you have config-level access to your mail server, or a hosting provider with a clue.

Be warned that bounce handling is an exceedingly unpleasant task, and is very difficult to do well. You may want to look at a 3rd-party bounce handler like Boogie Tools bounce studio to save you some sanity! Aside from some very broad top-level standards, there is very little consistency at a lower level, and bounces are often useless. For example, MS Exchange can produce bounces that do not contain the bounced message's original target address at all! The way around this is to use VERP, where every address (or even every message) gets its own unique return path, allowing you to know where a message bounced from even if the message content contains no clues.