Leonid Shevtsov, software architect at Railsware and developer of a forthcoming mail transfer agent talks with host Robert Blumen about email protocols and transactional email. The discussion opens with an overview of email; the architecture of email; DNS records and email; components of an email address; how email is forwarded; bounces; other things that can go wrong; the SMTP (simple mail transfer protocol); comparisons to HTTP; authentication and email; vulnerability to spoofing; SPF and DKIM verification frameworks; limitations of verification frameworks. The second part of the discussion covers transactional email – email sent in response to a user event such as password reset or order update; failure modes; bounces; spam reports; handling of adverse events; build versus buy; difficulties in testing email use cases; testing tools. Challenges of bulk email delivery.
This episode sponsored by SignalWire.
Show Notes
Related Links
- Railsware
- Leonid Shevtsov’s Blog
- Leonid Shevtsov on Twitter
- Wikipedia entry on SMTP
- Wikipedia entry on DKIM
- Wikipedia entry on SPF
- Wikipedia entry on mail transfer agent
- postfix MTA
- heraka SMTP Server
- zone MTA
- Mailtrap
Transcript
Transcript brought to you by IEEE Software
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected].
SE Radio 00:00:00 This is software engineering radio, the podcast for professional developers on the [email protected] se radio is brought to you by the computer society, as well as your billing software magazine online at computer.org/software
SE Radio 00:00:20 Signal. Why a real-time video technology allows you to create interactive video experiences that were previously impossible. Signal wire gives developers access to broadcast quality ultra low latency video for everything from video collaboration tools for film and TV studios and fortune 500 enterprises to engaging virtually fence, they can even assist with one of a kind fully interactive virtual concerts. See why the future of video communication is being built on signal wire. They’re easy to deploy APIs and SDKs are available in most popular program languages. Signal wire is a complete unified platform for integrating video as well as voice and messaging capabilities into any application. Try it [email protected] and use code se radio for $25 and developer credit. Go to signal wire.com that’s signal wire.com and use code se radio to receive $25 in developer credit today
Robert Blumen 00:01:07 For software engineering radio. This is Robert Blumen. I have with me today. Lay need shuts off land need, or as he goes by, Leo is a software architect at rails where a company that offers email testing products. He has over 13 years of experience building software for the web the past year and a half. He has been developing a mail transfer agent. We will be talking about email, generally transactional email and bulk email Leo, welcome to software engineering radio.
Leonid Shevtsov 00:01:41 It’s nice to be here. Would you
Robert Blumen 00:01:43 Like to add anything to your background that I didn’t cover?
Leonid Shevtsov 00:01:47 Uh, well, that MTA has been in two years in development. Now it’s taken awhile. Uh, yeah. SMTP is a fascinating technology. I’m excited to talk about.
Robert Blumen 00:01:58 We start out thinking about transactional and bulk emails, a topic I wanted to spend some time and talking about email generally, because we all use it, but it’s probably not as widely understood as some newer technologies like HTTP. So let’s start out talking about email and then we’ll talk about some more modern applications. Are you familiar with the history of email?
Leonid Shevtsov 00:02:26 Not really. I’m not much of a historian, but I can tell you that the SMTP is pretty much as old or older than the web. And it’s a pretty amazing that it’s survived this far. It’s a very open, I would say it’s the most open messaging format on the internet. And in these days when we take much care into de-centralization and, uh, like moving away from, uh, just allowing people to participate in a platform, uh, well, uh, SMTP has allowed, uh, individual people even to participate without any like center from the beginning. It originated in universities. So every university had a server with the people on it, and, uh, sometimes they exchange the mail on the server and that was relatively easy to do. But then, uh, what if you wanted to send mail to someone outside of your own, uh, university or a domain as we now call touch things? So that’s when was born and, uh, it is basically a protocol to send mail from one domain to another. That’s what it is,
Robert Blumen 00:03:43 SMTP being an acronym. What does that stand for?
Leonid Shevtsov 00:03:47 Oh, it’s just a simple mail transfer protocol or simple message transfer protocol. It doesn’t seem much so SMTP, as I said is extremely decentralized and all you need to send mail is a domain name, or perhaps you can even get away with an IP address, but to really want a domain name. And once you have that, you can send mail, you can do that from your own machine. You can do that from your application server, or maybe you should do that, uh, because of reasons that school talk later, it’s fascinating that while SMTP was designed as a protocol for humans now it’s the vast majority of emails to send from, uh, automated systems it’s bulk mail. And the worst of bulk mail is of course, uh, spam messaging and, uh, protecting people from, uh, unwanted mail is the AC huge. And it’s, it’s the most important, uh, thing going on in email for years and years. And there’s no side, no, no, no. And insight in that matter,
Robert Blumen 00:04:56 Situate SMTP let’s go one level higher from where we are, what is the overall structure of email delivery?
Leonid Shevtsov 00:05:09 So as MTP as a text protocol, justice like HTTP and to deliver mail, you’d just connect to a port. So that’s, that would be a port, uh, uh, well for clients is usually 5 87 and, uh, for service it’s 25. And then you have a text dialogue with the server. So compose and send a mail from your machine. Your mail app will communicate with a mail transfer agent, uh, which has a server that runs SMTP protocol, and it will, uh, queue your message for delivery and, uh, in the next possible a moment which hopefully is like within a second, it will, uh, determine, uh, through the DNS system, what is other, the, the recipients, uh, mail transfer agent, uh, connect to it and deliver the mail. And the recipient transportation will, uh, put the mail in a, some form of a mailbox and a usually let the recipient know that there’s a new message and then the recipient can go and use, uh, one of the standard, uh, mailbox protocols of which there are two, uh, pop three and I’m up and read the message.
Robert Blumen 00:06:27 Does the mail go directly from the sending SMTP server to the recipient or can it transmit over intermediaries or,
Leonid Shevtsov 00:06:39 Yeah, that’s an interesting question. So the way the protocol is structured, it seems like there’s multiple service in the chain, and you will think that there’s like to deliver a mail. It has to go through hops like you would see in an IP community, IP packet, communicating actually, uh, there’s usually just two servers, the sender server and the recipient server. And what might happen is that either of those have, uh, several servers in their internal chain, for example, uh, there’s such a service on the web as, uh, a pre-filter that will, uh, reject spam messages. So you can, that, that is another kind of SMTP server, some to be a relay that you use in place of your own SMTP and that it will filter the messages and it will forward the legit messages to your, uh, MTA. So then you would have a, to service on the receiver end of the SMTP message flow and on the center, end for example, maybe you have some internet server for your enterprise company or university, which will then go through a relay before leaving the intranet to go to someone else’s MTA.
Robert Blumen 00:07:54 Let’s start out with an email address. What are the components of the email address?
Leonid Shevtsov 00:07:59 Yeah, so as I said, uh, a whole, uh, mail is attached to a domain. So the domain is what you see before the, at sign, in an email address. And it defines, uh, where does your mailbox live? So it defines where to send mail. So it gets to you and the, uh, part to the left of the at sign is your it’s not your username. It’s actually not well-defined because, uh, it, the behavior of that part depends on the specific, uh, mail server that you’re running. Well, it’s usually the name of a mailbox on the server, but it doesn’t have to be just some identifier
Robert Blumen 00:08:41 Looking more in depth at the domain name, the sending side, we’ll use DNS to locate the recipient. What kind of DNS query does it make? What does it give back?
Leonid Shevtsov 00:08:52 Yeah, so there’s a center DNS, which is the MX record, uh, which, uh, points to the mail server for a specific domain. So a mail domain that record specifies where is the mail server for your domain and where would in MTA connect to send mail? So it is received by your domain because in many, many cases, the domain that is, for example, serving the website is not the same domain, which would receive mail. That would be on convenient. So you can define a separate domain name for the mail server or a multi.
Robert Blumen 00:09:32 I want to clarify, do you mean I might have a domain or you might have a domain say it’s a male dot male trap.com and we’re going to send mail there, or you might use mailtrap.com, but the MX circuit would have a different IP address than the website.
Leonid Shevtsov 00:09:51 The differences in the domain name and the IP addresses don’t matter as much. Uh, let’s have a more, uh, familiar example. So if you’re using Google apps for your domain, so your software engineering.com and then, but you’re actually using Google apps. So you don’t host your own mail server. It’s the Google. So the MX record for the domain will then be something like is BMX dot something, something that google.com. And when I want to send mail to software engineering.com and I compose a message, then my MTA will look up the MX record and we’ll see that it has to send the mail to Google, which is another fascinating thing about this MDB and mail that while it is designed and imagined as a decentralized protocol, it’s the vast majority of mail is hosted on the same, uh, big corporate servers. Like,
Robert Blumen 00:10:45 Is there an authentication component for the user to send an email?
Leonid Shevtsov 00:10:51 Yes, of course, uh, as a other service, uh, you have to authenticate. So the authentication happens on the first leg of the flow. When your user application connects to the first submission agent, the first NT in the chain, it will authenticate and usually requires a username and password. And then the rest of the way the mail is actually not authenticated in that way. So the rest of the communication is literally open for anyone else. That is a big challenge in delivering mail
Robert Blumen 00:11:27 Or some of the problems caused by that
Leonid Shevtsov 00:11:30 Into words. That means that anyone can deliver a mail that looks like you sent it. There is no mechanism step in that there are mechanism mechanisms that would allow the receiver to verify that the message is sent from you, but anyone can try to just connect to recipients, MTA, and deliver a message from anyone in the world
Robert Blumen 00:11:51 Practice. Is that something that happens a
Leonid Shevtsov 00:11:54 Lot? And it does happen a lot with spam mail. There are pretty good mechanisms to stop a little, to detect the mail before it gets to the user, because people are easy to confuse and computers are sometimes less. So, however, the idea of spam is that if you send a billion messages, then maybe a thousand comes through and you have some results. So they just find strength in numbers and abuse the mail system with a much more mailed in.
Robert Blumen 00:12:27 I understand than any computer, anywhere with an SMTP server could look up another SMTP server and send a mail and say, this smell is from Robert [email protected]. And there isn’t anything in the protocol in Aaron Lee that would verify or reject that as being not true,
Leonid Shevtsov 00:12:49 That is correct that anyone can connect and send such an email.
Robert Blumen 00:12:54 Some of the mechanisms that have been built to validate send.
Leonid Shevtsov 00:12:59 Yeah, so there are two major mechanisms. First one is SPF it’s the older one, and SPF is another type of a DNS record that specifies the allowed senders for domain. So for example, you can state that only your MTA can send mail or some specific AP can send mail for your domain or well, that, that those are the two options, really that way, when the recipient receives an email, they can check the origin and if it doesn’t match the SPF record, then they know that it’s not from you. The problem with that is that relies on, uh, so setups fluctuate. So SPF records often become invalid. So rejecting the mail just on the basis of SPF would, uh, result in many false positives. And that’s why it’s not a, a hard rule to accept or reject email. And, uh, the other mechanism is a DK, um, which is essentially a signature for the mail.
Leonid Shevtsov 00:14:08 And the way it works is again with the DNS record. So you publish the public key, uh, for the signature in your DNS record, and then you use the private key to sign the mail. And then the recipient again, can verify that the signature meshes the public key in the demand record. And the way this is problematic is also because, uh, setups fluctuate and, uh, th that’s one problem. And the other problem is sending messages is not very simple. So it’s simple if you’re running an MTA, but it’s not simple. If you’re just sending mail from like your own machine, which is a thing you can do. So requiring such a mechanism to be present, always would cut off many people from the male ecosystem and for an practical example of where you send mail without any big MTA is if you’re just running a script on your server and you want to get the results and you don’t want to complicate the setup, you can just use send mail, which is, uh, units, uh, utility present on most machines. And you just use it to connect directly to the recipient MTA and deliver your mail. And this mostly works
Robert Blumen 00:15:33 The SPF and D K I M records. Are those a type of DNS record?
Leonid Shevtsov 00:15:39 Yes. Yes. Those are DNS records.
Robert Blumen 00:15:41 You would host them with your DNS provider.
Leonid Shevtsov 00:15:44 Exactly. And the owner of the domains that’s the up. So that’s the, uh, the trust part there. So you have to own the domain and have access to the DNS set up to modify them.
Robert Blumen 00:15:56 So what you described as if you were to strictly use those frameworks, you might end up rejecting too many emails, given that how widely used are those two practices
Leonid Shevtsov 00:16:08 Used by Amy sender that wants to appear trusted. So I would say they are used in all of the mail going on, unless it’s a like hobby server, for example, there is little reason not to use them. And that is that also poses a problem.
Robert Blumen 00:16:31 If I’m a large retailer, I have 2 million customers, people sign up for email because they want to know what are the deals this week. So we’re talking about a totally legitimate type of email, which people have opted in. And I want to be sure that this email doesn’t run into any problems at all with any kind of filtering that might take place or spam detection, then I would set up an SPF and D K I am, because if I am a legitimate, why not? Is that more or less correct? Yes.
Leonid Shevtsov 00:17:05 That’s one case where you definitely want an SPF and DKM, but the problem is that the spammers will also set up SPF and DK. And that’s another reason why the mechanisms are not very robust. So if I set up at the main, that looks like your domain with a slight difference, I can completely correctly set up SPF and DKA DKA for that domain and send mail to the Prius. Well, it is sent from that domain and it is validly sand, and it’s a completely legitimately signed email, but it’s still coming from a domain that is designed to mislead the user into thinking that it’s, for example, their bank, which is called
Robert Blumen 00:17:49 Spoofing. When you’re talking about is the spammer can curve that the email did originate from the domain that it said, it’s simply not a good domain, which is a different problem.
Leonid Shevtsov 00:18:02 Exactly. And the mains are, uh, also widely accessible. So there’s no mechanisms stopping spammers from the registering more and more domains.
Robert Blumen 00:18:12 Let’s drill down a little bit into SMTP. So you have the two sides of the protocol. What are the messages that are exchanged in an SMTP session?
Leonid Shevtsov 00:18:25 A mail message looks very similar to an HTTP response. It uses the same Heather and body structure. So if you’ve seen a nation DP request, it’s just the same. Uh, there are several required Heathers, like from, or to, and a meany optional healers of which there’s the DKM signature Heather, and some others that serve other communicators for prisoners or they’re used for tracking mailing lists. And then there’s a body which is a, either an HTML or text document or an attachment, which has base 64 encoded. So it’s all in all the same format as HTP requests, just with the different flavors.
Robert Blumen 00:19:16 The whole process from beginning to end would imagine a large number of things could go wrong. And we don’t have time to talk about all of them, but what are some of the major things that can fail and cause an email not to get through.
Leonid Shevtsov 00:19:30 All right. So first of all, I’d like to mention that, uh, SMTP is in the synchronous protocol. So it’s designed to operate when your recipient is not home, and that’s why it was created. Because if you want to send a message, you want someone to receive it while the person may be not that they computer and will receive. So that’s why you have mail servers. And, uh, first of all, you have to have a mail server running on the recipient end. Uh, we’ll end the end on the center end to get the message through, but that is an obvious one. Uh, you have to have the, uh, uh, demand records, the MX demand record set up correctly so that your MTA can find where to deliver the mail. Uh, you have to, uh, then be able to connect to their domain. And, uh, then you have to provide a valid mailbox name on that domain. And then there are internal rules such as a mailbox size because of which the receiving MTA might reject your message. So if the mailbox is full, it will reject the message. If there’s no mailbox that will reject the message. And if your email does not pass reputation checks, it will also reject the message. And all of those are called bounces in mail lingo. So it bounds is where you connect to the server, the recipient server. And it does not accept your message.
Robert Blumen 00:21:09 We have a disappoint, a pretty good broad overview of email. And now I want to move into a particular more modern application, which is called transactional email. And it’s something you have expertise in. I want to start out defining it in term transaction, it’s heavily overloaded in computing. What does it mean in the context of email? Yeah,
Leonid Shevtsov 00:21:32 As in email transactions are not the same that a programmer would be used to because email uses the term in, in more of a business context and, and transaction just means some thing happening that leads to an email being sent. So it’s like an event in programmer terms, but I also don’t want to, uh, attach too much meaning to the event word. So transactional mail simply means mail directed to a specific, uh, uh, recipient, uh, for a specific individual reason. And that is in contrast to a bulk mail where you have a message that you want to send too many people. Sometimes it’s millions of people and, uh, some very different mechanism mechanisms apply when you need to send just one message versus a million.
Robert Blumen 00:22:31 It’s not going to be helpful if I think database transaction, this is more like I bought something and the transaction is more of a transaction. And the business sense, is that correct? Yes. Yes. Well, what are some examples of transactional emails? Some use cases?
Leonid Shevtsov 00:22:47 Well, the most transactional emails are those that have to a cure at the instance that they were requested. For example, if you’re receiving login credentials score, if few is at a password and then, uh, in the order of degree decreasing priority. So maybe you want an order update and that could also happen instantly. So sending mail instantly is harder than sending it on a schedule or with a delay. And for example, at a, do you want to receive a instantly and we really are irritated when it doesn’t arrive fast and for an order update, it doesn’t happen because of your actions. So while it is still directed to you personally, it’s not a problem if it’s delayed a bit. So maybe you want to bulk send order updates to collect them like in a, uh, some buffer. And then there are some quasi marketing emails, like the welcome, the onboarding message. It might be a transactional message, or usually a AA Balt message. And then you go into complete the marketing territory where you just send messages and messages on a schedule.
Robert Blumen 00:23:57 Everything goes as expected. What is a pretty good timeframe between sending and receiving any
Leonid Shevtsov 00:24:06 Sending itself is a very fast, so it will take within a second, if everything goes well. And the time over that delivery time is a queuing in case the MTA, your empty is, uh, over capacity, or, uh, you might also see slow delivery because of, uh, intermittent, uh, bounces.
Robert Blumen 00:24:31 What are the main difficulties in implementing transactional email?
Leonid Shevtsov 00:24:36 So concessional mail is usually sent by some application, right? So someone runs say web application. And, uh, uh, the first idea you might have is that, as I said, you can just send the mail from the server, you’re running the application on. And the tricky part is that that’s completely valid and possible. And if you’re just running Linux, you have send mail, you can just send, send the mail to the recipient, and then you start running into problems. The biggest problem of all I would say is that unless you process the bounces and the spam reports that occur. So if you’re not being a good mail citizen and, uh, you just sending mail with no understanding where it ends up, you quickly get blacklisted. So the, a part of your reputation is, uh, how many palms messages, how many messages that qualify as spam you sent.
Leonid Shevtsov 00:25:48 And that doesn’t matter that there are spam messages because mechanisms are not perfect. The spam algorithms are not perfect, so anyone can send a message that looks like spam. And the proper thing to do in this case is to detect that and adjust your message, or do not send it if the recipient rejects it. And that is hard to do when you are running and delivering mail from your own server on your own efforts. So that’s the big challenge. There’s also the one that you have to set up all the SPF and DKM and Ron signatures. So that kind of thing, but that is a one-off problem. And processing rejects is an ongoing problem that you have to do constantly
Robert Blumen 00:26:38 Drill down into the mounts. You said balance is a general term for many different kinds of failures. Does the point in the system where the bounce occurs, generate a return message back to the sender?
Leonid Shevtsov 00:26:53 So a bounce is specifically a failure that happens at the moment of delivery. So you always get a message back at that instance, and it usually describes what’s going on. And by that message, you classify bounces as soft bounces and hard bounces. And this is not a well-defined distinction. It’s just the south bounces are something that might resolve in some time. For example, if the mail server is overcapacity, or if the mailbox is full, that might get resolved. And if there’s no such recipient, uh, then there’s no chance you’re gonna deliver the mail later. So the proper way to do in such case is not send the mail again. And that’s called a hard bounce.
SE Radio 00:27:41 As you radio listeners, we want to hear from you, please visit se-radio.net/survey, to share a little information about your professional interests and listening habits. It takes less than two minutes to help us continue to make se radio even better. Your responses to the survey are completely confidential. That’s S e-radio.net/survey. Thanks for your support of the show. We look forward to hearing from you soon.
Robert Blumen 00:28:06 You mentioned that if I write a script on a server, it can correctly execute the protocols, but it doesn’t the ability to handle some of these error cases. So is that suggests an architecture where the application would submit email requests to a subsystem that’s better able to deal with the failure modes?
Leonid Shevtsov 00:28:28 Uh, yes, that’s the reason, uh, mail transfer agents as service exists is because you want some third party handling, uh, errors for you. Uh, you want them to do monitoring and, uh, you want to know that the errors don’t end up in the void. So what I didn’t mention is that’d be beyond me besides bounces. There’s also a case where someone marks the message as spam, which is a different kind of error, and that does generate a separate message to your mail server, which is like another SMTP message. And you have to parse and, uh, process that. And also record, probably record the recipient as in non deliverable and, uh, processing mail messages is not an easy test to do. I’ve tried it in the past. I’ve tried to send mail from my server and process balances and process return messages. And, uh, it’s not something that, uh, can be done reasonably. If you decide your main focus,
Robert Blumen 00:29:48 What are the difficulties in getting
Leonid Shevtsov 00:29:51 The well, uh, there’s just that it’s a separate, uh, service to run. And then, um, the format of messages, uh, is very well based on the recipient empty. There’s no hard standard for these messages. Um, they’re all, as I mentioned, text-based and the, there are error codes, but they also vary. And there’s the error codes have deep nesting. So you might parse it, generic error code, like mailbox full, but not parse the specifics. And, uh, you don’t, you can just pull up the protocol spec and implemented. That’s not possible in this case, you have to, with time over time, collect more message formats and, uh, build your sort of database algorithm of handling the messages. And that’s what MTS do.
Robert Blumen 00:30:54 If a mailbox is full, I would expect to get a bounce pretty quickly, but something might sit in somebody’s inbox for a few days, and they mark it as spam. The indication that email was considered spam could come back much later, is that correct?
Leonid Shevtsov 00:31:12 It can come at anytime.
Robert Blumen 00:31:14 You’re sending emails and then immediately, or much later you get different kinds of error codes coming back. How do you associate the error messages coming back with the messages that went out?
Leonid Shevtsov 00:31:28 There’s a standard for that, at least, uh, when you send them mail, you specify, uh, something called a variable envelope return path, which is complicated way of saying it’s an, the address that you want, the, uh, the non-delivery, uh, report to be delivered to. And that, that address is an example of an address that does not correspond to a mailbox on the server, because the, the username, the local name in that address is, and then coded string that contains usually either the message ID or the recipient address, or some other, whatever your MTE needs to identify the message that failed
Robert Blumen 00:32:22 Or bounces sent back by SMTP.
Leonid Shevtsov 00:32:25 Well, no bounces are just a in protocol. It’s like an HTTP response code. So you send a mail and you get back a set of scopes and some texts back,
Robert Blumen 00:32:36 Even if the balance comes back a day later,
Leonid Shevtsov 00:32:40 But bounces always come at the moment of delivering,
Robert Blumen 00:32:44 Right? Okay. Something marked as spam is that scent, is that an SMTP message to,
Leonid Shevtsov 00:32:52 So it took a like those, you have to not only run in union need, not only have to send mail, but you also need to have a receiving empty because that’s a message that’s going to be sent to you, and then you have to receive the message and then you have to parse it, uh, find the recipient mail, uh, find the reason for the report and record it in some database, which you will check against, uh, when sending future mail.
Robert Blumen 00:33:23 And what does that look like? You have a database you’ve collected bounces, some of which might clear in a short time, some which are considered permanent failures, and you have these async delayed spam reports. What do you do with that information?
Leonid Shevtsov 00:33:38 Well, it’s not anything complicated really. Uh, south monsters usually go to the same queue without a permanent record anywhere. So you just retry the message in a couple of minutes, then in an hour, then maybe in a day. And then within two days of the initial delivery, you just fail. Or if you just get a hard bounce, or if you get a spam report, then those records go into a table which lists the, the recipient’s mail addresses and, uh, a set us code that says, why would you not send mail to them? And some of these expire over time, like mailbox full could be expired in a week, or otherwise you just, don’t the, when you send you mail, the recipient addresses checked against the table and the mail is projected before sending. And, uh, that is also how unsubscribes work. So he, you also, by the way, you have to have a mechanism for unsubscribes. Uh, that is a crucial part of delivering mail in our times, uh, because it is used by, uh, recipient MTS to mark your messages, not spam. So if you don’t have an OBS subscribed link, that is one of the factors that mark your message as spam.
Robert Blumen 00:35:03 What happens when someone unsubscribes
Leonid Shevtsov 00:35:06 They get added to the, the table of unsubscribes? Uh it’s again, that part, at least is fairly simple. So you click on subscribe, uh, usually see some long and coded key in the URL. That key contains included information about well, who you are. And, uh, it puts the information into the unsubscribed and a good dance subscribing mechanism doesn’t require you to log in anywhere. It just works on the mail delivery layer, and it doesn’t need any more information that this contained with the unsubscribe link.
Robert Blumen 00:35:50 Yeah. About range of different components of infrastructure, mill transfer agents, processing of, uh, unsubscribe processing of things marked as spam. How much of this does there exist open source and how much of it is commercial?
Leonid Shevtsov 00:36:06 So while there are many open source MTS to name some well-known ones, so postfix is the one that probably most people are familiar with and it’s shipped with Linux. There’s also edX. I am, if you want something fresh and interesting to play with there’s her Rocca and zone MTA. Those are two on that I experimented with. But unfortunately when it comes to actually processing the error messages and, uh, doing delivery at scale, there’s no open source solutions for that because building the knowledge base required is, uh, a requires a lot of human resource. And that’s how I understand it. So first of all, people usually want to get paid for that. And second of all, it’s too much of a niche product to attract many volunteers. So you don’t see good open source products in that category.
Robert Blumen 00:37:20 What does the vendor space look like? How many vendors are there and is it typically segmented by, um, a country or language, or are there major worldwide players in that space?
Leonid Shevtsov 00:37:36 So if we’re talking about proprietary MTAs as software, there are several of them. Uh, and, uh, some are, well, I seen, I’ve seen yes, oriented MTAs and European oriented MTS, and I’m not sure if that’s a big distinction in terms of what they can process because everybody sends mail worldwide these days, or rather just the regional flavor, I guess. But what I can say is that, uh, those, that software is usually employed by companies like hosting providers or big enterprises that need to send a mail for their clients, or if you’re building your own empty. So you might use one of the solutions and build your business logic on top of that. Uh, if you want to send mail for your application, you usually want, uh, some software as a service. And that is because the licensing costs of proprietary MTS do not suit individual centers. And on top of that, you also have to run your own infrastructure, which might get, uh, complicated and expensive. If you’re sending at scale,
Robert Blumen 00:39:07 They are talking about is testing. How do you go about testing transactional emails with so many pieces? And it’s so highly distributed
Leonid Shevtsov 00:39:18 To test mail. You usually deliver it to someone. So you want to exercise the protocol. Otherwise you, you, you just stop the delivery. So historically what people have done is just 10 mail to their own account, the, on their own mailbox and see what comes up and, uh, incidentally, uh, working at MailTrap. So we build the service to serve as the, uh, submission MTA, uh, that will, uh, collect your mail and then, uh, collected into a mailbox. So you can see what your application is sending instead of using your own mailbox for that, because there are many issues with, uh, sending mail out of, into the world, even if it’s into your own mailbox, uh, the first of all, being that you might send to the wrong mailbox, to the wrong people. So if you have, uh, some staging environment and you have, uh, real user emails, which happens, I’ve seen that happen and you send mail like your destiny mailing list, and you send mail to everybody on the server and use them real people.
Leonid Shevtsov 00:40:46 And that is always a problem. Sometimes it makes the news and, uh, to avoid that we build, uh, MailTrap, which is an MTA that instead of delivering mail to the recipient, collects it and displays it. And it gives you all sorts of analytics. So that’s the kind of solution you want to have when, uh, testing that, uh, your mail would be delivered correctly. So what do you want to see is a paper application is composing the email correctly. If you have the right headers, you want, if, uh, your message looks right, and if it doesn’t qualify as spam based on the contents. So there’s a software called SpamAssassin that analyze us the content of the message itself. If that looks like spam. And it’s one of the factors they use to analyze, to, to sort of spam messages from non spam. So you want to see, uh, that your message scores well on spam assess. And, uh, you also want to see if, uh, the application you’re sending from is not on the blacklist, that server, if your domain is not blacklisted, so there’s, are there, those are things that you want to know before you start sending email to real customers.
Robert Blumen 00:42:18 You mentioned blacklisting couple of times. And the first time you said, I wanted to ask you a question, and then I described who maintains a blacklist and how do servers either get on them or get off?
Leonid Shevtsov 00:42:31 Yeah, so there are many companies, me third-party companies that operate like lists. And, uh, the idea of a black list is, well, it’s a list of, uh, APS or domain names that are not, uh, trusted, uh, reliable email senders. And that doesn’t mean they send spam necessarily. Uh, it’s important to understand that, well, there is legitimate spam. There’s also mail that is either unsolicited or it appears like spam. And you usually, you don’t want to send those messages because you get blacklisted. So there are two kinds of there’s actually, there are two parts to reputation management and mail. So there are two kinds of black lists. One is a destitute domains, domain names, and those are the domain name that you’re sending from. So for example, if you, as I mentioned, if you’re a spammer that tries to fish passwords are the fusers by appearing as a domain of some bank, uh, then your, a phishing domain might get blacklisted because it sends spam.
Leonid Shevtsov 00:43:47 And the other kind of blacklists is IP blacklists. And, uh, they serve a different purpose. So spammers, usually, uh, abuse, uh, machines on the internet. And that’s also a huge, interesting, uh, uh, area. So the main purpose actually to hack into servers these days is to send mail from them. Uh, you want some fresh APS, uh, that’s what hackers are against thee. Usually don’t want to steal passwords unless you’re someone big, but they do want to get your AP because every fresh AP is an opportunity to send some males before getting blacklisted. Once a male recipients see a large volume of spam coming from some IP. So they cooperate with these black list companies and they collect the information about which IPS are sending unsolicited mail. And, uh, sometimes the blacklists operate in a matter of minutes or hours. So they’re very highly dynamic because if someone hacks and machine the, just start using it right away, and it’s important to stop the activity as soon as it happens and to get off such a blacklist.
Leonid Shevtsov 00:45:14 Uh, so there’s, again, I think there’s maybe 20 or 30 companies that run blacklists and, uh, they usually use a tool to see if your domain or IP is listed. And then you can appeal to the individual company and, uh, ask to be removed from the black list and the way you actually get on the black list as a legitimate sender is either you inherit an AP that has a bad reputation. So that happens with cloud providers. Although recently, I think like Amazon cracked down on spam sending, so you can send mail from Amazon machines, uh, before they approve you. So that helps a bit. So the APS are more clean, but in the past decade, there has been problem, big, big problem that if we were with a cloud provider, you get an IP and it’s already blacklisted, or another case is if you run mail. And as I mentioned, you don’t check if it’s qualified as spam and you eventually get blacklisted unknowingly, and then hopefully you start investigating the issues here, blacklisted, and you learn from the mistakes
Robert Blumen 00:46:35 We have in the last few minutes. I wanted to talk a bit more about bulk email. You’ve mentioned scale a couple of times in the context of you could have a very large number of transactional emails, like a very large number of order updates or, uh, businesses. Solars, it’s, you’re doing a large volume of password resets. What are particular challenges? If I have a mailing list with 2 million people, and I want to get out 2 million emails in a reasonably short span of time, like maybe overnight to let everybody know what are all the sales or specials going on in my business
Leonid Shevtsov 00:47:15 The next day, that’s an important target to meet. So me, uh, emails are time sensitive, order updates, certain maybe not the most time sensitive. And because imagine you have to send a calendar event updates or like flight cancellation updates, and, uh, you really want people to receive them right on time or job dates, of course. And, uh, when you need to send a many messages at once, uh, you usually use an empty that’s tuned, uh, for not transactional mail, but bulk email and, uh, the, uh, have, uh, a bigger array of, uh, uh, machines to send from. But also, uh, they can, so a lot of time that it takes to send the mail is preparing the mail to be sent. So templating for example is an obvious one. So you have to prepare the, you have to render the mail and the bulk message system with the render, many males more efficiently than one by one.
Leonid Shevtsov 00:48:22 So that’s when you go cheap bulk sender. But I have to say that the bulk sending and transactional sending are two categories that while they’re intersect, you usually know if you want one or the other. And, uh, I will suggest, uh, people to start with transactional centers while they’re small, because they’re more controllable they’re. So they’re event driven. You can send just one mail and with a bulk center, usually have to submit like a mailing list. So start with a transactional sender. And when you get to that million of customers, you might transition to box sender.
Robert Blumen 00:49:01 We’ve reached the end of our time. Is there anything else you wanted to cover that we haven’t talked about before? We were about
Leonid Shevtsov 00:49:08 Not really, again, to repeat my, uh, suggestions to people who start out sending mail. I suggest, uh, use, uh, email as a service MTS as a service because, uh, you avoid many, uh, rookie problems, uh, test your mail before sending and, uh, yeah, don’t send spam. Where
Robert Blumen 00:49:31 Can listeners find you on the internet?
Leonid Shevtsov 00:49:35 They can find, uh, me personally at, uh, l-s.me, uh, I have a blog on technology. Uh, they can find me interesting things on mail, on the MailChimp blog at MailChimp, that IO slash blog. That’s it.
Robert Blumen 00:49:54 Lanny, thank you so much for speaking to software engineering radio. Thank you. This has been Robert bloomin. Thank you for listening.
SE Radio 00:50:04 I see radio listeners. We want to hear from you please visit se-radio.net/survey to share a little information about your professional interests and listening habits. It takes less than two minutes to help us continue to make se radio even better responses to the survey are completely confidential. That’s S e-radio.net/survey. Thanks for your support of the show. We look forward to hearing from you soon. Thanks for listening to se radio an educational program brought to you by either police software magazine or more about the podcast, including other episodes, visit our [email protected]. To provide feedback. You can comment on each episode on the website or reach us on LinkedIn, Facebook, Twitter, or through our slack [email protected]. You can also email [email protected], this and all other episodes of se radio is licensed under creative commons license 2.5. Thanks for listening.
[End of Audio]
SE Radio theme: “Broken Reality” by Kevin MacLeod (incompetech.com — Licensed under Creative Commons: By Attribution 3.0)
