Mailing a letter

Hey all, it’s been a while. Hope you’ve been well and I’m sorry for the radio silence—can’t offer much in the way of excuses other than it’s been difficult to find the time and energy to write one of these posts. But, well, that doesn’t mean I’ve been otherwise idle.

I’ll jump right to it and talk about a problem I’ve been working on: email. Why? Well, it relates to having user accounts on THP. (I’ve touched upon that briefly in the past and I’ll just quickly restate that it’s not meant to replace anonymous posting just as a form for users to have more features and store relevant information/options. More on that when other stuff is done!)

Basically, email is the logical way to have users log onto a site. Yes, user names, but those would be kind of pointless since they’re not public-facing (ie no one would see your account name if you post) and you’d want an email address anyways for account recovery and as a basic anti-spam protection. This is a simple conceptual implementation but there’s a lot of moving pieces that you might not realize and that are made more inconvenient due to the self-hosted nature of the site and few resources we have available.

I’ll do a quick rundown about email: it’s an incredibly simple spec that’s been around since the 60s and has evolved a little but many of the fundamentals haven’t been changed since the early 80s. It’s ubiquitous and there’s all sorts of implementations and myriad of mail servers and clients. There’s also a lot of archaic convention and assumptions made that aren’t so obvious if you’re used to more modern (read: post-mid-90s computing) and so setting it up on a machine is simple in many ways but it’s very easy to misconfigure something. There’s all sorts of mail hosting services that do a lot of steady business for this reason and they take care of the inevitable gotchas that pop up.

What is more relevant is that THP has had its own mail server for a many year. It can both send and receive email. It also deals well with all sorts of modern additions like encryption and properly uses a database to store data (as opposed to plain text messages on the server, which is common enough).

That’s all fine and good but there’s a big problem with hosting your own email: a lot of mail providers will refuse to deliver your email by default. Obviously, this is because of spam. So—this is where it gets complicated—there’s a few different additional authentication that many services require in order to be able to receive email from an address. SPF and DKIM are the bare minimum required to get your email successfully to the recipient and those involve more complex configurations and generating keys and setting domain records appropriately so that emails are “signed” and restricted to coming from specific addresses. Many of the large email providers like Google and Microsoft will outright reject emails from servers that aren’t configured exactly right. Additionally, these large players also tend to ask for additional verification for domains, meaning that webmasters have to create accounts with them to have emails from their domain go through without a problem. And, even after doing all that, there may be issues on the network side of things as specific address ranges may be blocked which necessitates the hosting company reaching out to the service provider to sort things out. This is can be frustrating to diagnose and fix as you might imagine. Just a few days ago I had to ask the hosting company to get into contact with Microsoft to remove the (new?) block on THP’s IP address.

With that aspect established, let’s move on. There’s also the issue of validating email addresses for anyone who would create an account. Not only is it necessary to check if it’s a valid address, doesn’t contain illegal characters and the like but also whether it’s not from a disposable/temporary/spam email provider. PHP has a simple in-built tool but it’s inadequate for anything but the most basic of checks. Like how big providers check a senders details before accepting delivery, there needs to be a check on a few parameters to see if the address is legitimate.

As it’s unreasonable for any one person to maintain a list of valid/invalid domains (they number in the millions, if not billions) this necessitates using community/third-party databases to check against. At the cost of code complexity, there are libraries that I can import and I’ve had a look at a few of them that might do the job. As they’re only really going to be called at account creating it’s not really that big of a deal to include one and, at any rate, wouldn’t be worth my time to reinvent the wheel myself. A potential limitation there is how fresh the databases might and how often the library may need to be updated. There’s a lot of spam sites out there and it’s difficult to keep up with them and so a locally-installed solution has the glaring downside of potentially being out-of-date.

There are services, including free ones, that allow you to use their API to do these types of check as well. They are more up-to-date but come at the cost of having to make a request to another server as well as potentially data being outside of my control. I’ve looked at the privacy policy of some and many are reasonably-private, but I’m always wary of relying on others given the general state of the tech world.

I haven’t made up my mind fully yet and am still playing around with different solutions but I think I’m leaning towards a hybrid setup depending on how feasible it would be to maintain a whitelist of addresses; I’d maybe use an online service to check for more unusual domains while otherwise defaulting to local libraries and databases.

Still with me? I’m almost done. The last large stumbling block is about sending emails. From the webpage, that is. Think things like confirmation emails, password resets, and anything else that might warrant sending an email. PHP has an built-in email function that automagically hooks in with the server’s setup to deliver mail. It’s simple and I already use it to send emails to the moderation staff or to do a few other things when appropriate. There’s nothing wrong with it except that it doesn’t scale up too well and requires a lot of verbosity and specific formatting to encode emails properly. So I’ve been thinking of using another library, PHPMailer, instead. There’s no real pressing need to do so at the moment but it might be a nice thing to have in the future as more user features come in as sending attachments, images, and other things in a painless manner might be useful for things like summarizing stories users are subscribed to or mass emails that follow a template.

So I hope that with this post you’ve been able to understand some of the issues I’ve been grappling with as of late. I want to say that I’ll have it sorted fairly soon but I wanted to test more and also bundle in any sort of sign ups with other things I’ve been doing. And I didn’t even get into the sort of database changes that I’ll have to implement for a lot of this! That’ll have to wait for a future date.

I hope to have more for you soon, if only to clear up my backlog of other writeups that I intended to do last year.

Until then, be well and take it easy!