I am developing an email sending service, probably for sending bulk emails using sendgrid web API, but I am not able to figure out best practice for scalable system. I wish to keep a record of all those emails which failed to deliver and retry sending to those failed emails after all emails have been sent. I am using NodeJs, so just wanted to know if there is any way to speed up my process(something like sending multiple emails at the same time)
1
votes
usually send grid return a error code or status code while sending the email . One way to sort it out is to log each email to be send in the db and mark 0 if failed 1 if email send successfully and later try out again using some scheduler or cron job. But you will have to handle the to ignore invalid email or non existing emails
- Joel Joseph
But in a situation where I have to send around 999 emails(according to documentation its 1K emails at one time), if there is an error it won't process the email delivery, rather it will give me an error which is not a scenario I am hoping for rather I need a list of only those emails which fails and while processing it should not break the program in case any email fail to deliver, so using Promise.all is also not encouraged. It should work in parallel(i know nodejs is single threaded, just not getting the right word) without breaking and yes I have to use some db for keeping a check.
- Jayesh Bidani
2 Answers
1
votes
There are multiple ways to handle this, I will suggest two which seems obvious to me.
(Recommended - Easy) Use Async module's control flow option called queue Async Documentation. You can feed in all the request in form of an array of object request and then change concurrency setting to let's say 100, it'll run concurrent 100 workers at one time and to log errors make a separate mechanism and once all the values have been run through handle it separately.
Spawn multiple workers using node.js native approach.
0
votes
Sendgrid offers an npm package for node.js integration, so you don't have to reinvent the wheel. It accepts messages at a high rate, so you shouldn't have problems delivering yours to sendgrid. You just dump your messages into sendgrid.
Email, being a store-and-forward system, is inherently asynchronous. That means it operates far from real time. Some messages are delivered in a few seconds, and others take hours (when they get soft--"retry later"--rejections from destination servers, for example).
Sendgrid handles this issue with a "bounces" API. (And with "bounces" features in their web back end application). Many bounces are "hard" bounces, meaning you must avoid trying to send messages to that address again. You can use the bounces API to retrieve a list of bounced messages. You should remove those addresses from your email list, and not try to send them again. (Sendgrid bans users who repeatedly send mailings with a high undeliverable rate.)
They also have an "invalid emails" API. This works like "bounces" and returns lists of addresses that are ill-formed or, if sendgrid can tell, not present on the destination server. Again, you should remove these addresses from your email list. If they're invalid now, they will be invalid tomorrow.
Sendgrid offers all sorts of tutorials on this subject.
