Build your project or community with self-hosted "Public inbox"


Mailing lists have largely gone out of style in favor of centralized communication platforms like Slack and Discord. A big reason for this is that these platforms are easy to use and more featureful than existing platforms. However, this means giving up full control over your community and relying on a large, private corporation in order to build a community. This may not be palatable for everyone.


Email remains, despite the best efforts of the big players, the most ubiquitous decentralized communication method. It is a relic from the old internet, built by academics focused on communicating and sharing research, rather than platforms aiming for a $X billion valuation. It is, of course, archaic, somewhat awkward, and constantly under threat from large providers trying to privatize this enormously successful piece of web infrastructure. But it remains essential to many projects and communities.


Email is not, in a word, "cool" -- most people under the age of 30 rarely use it for personal communication, and its primary use these days is as a dumping ground for "newsletters" and forgot-password links. However, I believe email should be treated with a certain reverence. It represents values that run counter to the web of today: it is decentralized, stable, long-form, basically universal, based on open standards, and difficult to mine for profit.


Unfortunately, Email software is sorely out of date. Many tools are ancient, outdated, and difficult to use. I decided that mailing lists could use a bit of a refresh, which is why I built crabmail.


Crabmail does one thing: takes a maildir folder and converts it into a static HTML archive. In combination with other tools, you can create a simple archives-first mailing list or "public inbox"


Archives-first "public inbox"

One problem with mailing lists is they don't really fit into the way people use email or computers these days. Few people are email "power users", and getting mailing lists set up, making sure they don't go to spam, etc, can be a somewhat difficult task. On the other side, making sure emails reliably get sent to list members can be a difficult task, which requires additional infrastructure or reliance on a hosted service.


My solution to this is to have an "archives-first" "public inbox" the list only handles receiving and publishing emails, not sending them. If you want to make sure someone is included on an email, you can CC them directly. If you want to follow an email discussion thread or list, you can use an Atom/RSS feed reader. To reply to an email, you use mailto: links on the web archive. This dramatically simplifies the infrastructure, making self-hosting much easier.


The technical details


This guide will show you how to set up crabmail to host your own public inboxes. What you will end up with is a set of IMAP folders that are mirrored on the web, similar to lists.flounder.online. Here's what you'll need:



You will end up with something that looks like this:


https://lists.flounder.online/


Gettting an email account with IMAP access

First, you will need an email provider that supports IMAP, and preferably sieve[x] filters.


Some options include:

fastmail

migadu


This filter will match any email sent to lists+[listname]@[yourdomain] and filter it into the folter lists/[listname]. In Migadu, which I use, this folder is created automatically by this filter (which means people can create their own lists!)


You will use your IMAP email in order to moderate your list: deleting spam or off-topic posts, blocking users, and so on. You could also set up a private allowlisted list using sieve filters.


Building the archive

To build the archive, we will first need to sync our IMAP folders to a local maildir folder.


Download isync (mbsync) and configure it to pull from your IMAP email to a local maildir folder. I recommend a folder with lists as subfolders.


https://wiki.archlinux.org/title/isync


Crabmail will use this folder to generate the archive. An example config is in misc/config. Add your IMAP credentials and create the ~/mail/lists directory. Then copy the mbsync config file in misc/config to ~/.mbsyncrc


Then run:


mbsync --all

This should copy your lists/ IMAP folder and subfolders to ~/mail/lists. You may want to send a few test emails to these folders to make sure you have some data first.


Make sure you have cargo installed, then download crabmail and install it:


git clone git://git.alexwennerberg.com/crabmail 
cd crabmail && cargo install --path .  

Copy crabmail.conf to mylists.conf and reference your email and the domain you'll host your site on.


Build the archive:


crabmail ~/mail/lists -mylists.conf 

Check out site/index.html to see what your site looks like!


Deploying your list archive

(I will flesh this section out a bit more when I feel more solid about the easiest way to do things)


You'll need two things to deploy your site. First, some place to run this job at a regular interval. I wrote a script to run the IMAP NOTIFY command and wait for messages:

https://git.alexwennerberg.com/crabmail/file/misc/imapwatch.sh.html


Otherwise, you can run cron at a reasonable interval (no less than 5 mins probably). You'll need to run this on an always on server, like a raspberry pi, or a cloud server / job scheduler. Note that wherever you host this will have FULL access to your email account, so you may want to either use a separate email account for this purpose or run this program on a server that isn't exposed to the internet.


Static file hosts: there are many of them. Take your pick. Check out the lists/rsync.sh command for how to deploy via rsync


TBD TBD

Footnotes

https://en.wikipedia.org/wiki/Maildir

https://en.wikipedia.org/wiki/Sieve_(mail_filtering_language)



/tech/