Email is still relevant

This is expanded from a slide I wrote a couple of years ago in a dagen@ifi talk for Oslo University about what I do at FastMail.  I answered the assertion “email is dead, social networking is the future” with the following points:

Compatibility

Just about every system out there can interact with email.

I have code I wrote over 10 years ago to process email. It still works. Try that with the API of your favourite social networking site.

Persistence and Immutability

I still have emails from over 10 years ago. I can still read them, check them against my memory.  Nobody else can change or delete them later – they are MY copy.

You can send attachments via email, and the links never go 404.

Privacy

Email is decentralised, and built on standards … mostly.

This means you can select a provider, or even host email yourself.  Once the email is on your server, nobody else can read it.  You’re not at the mercy of a single central authority, so you make the risk tradeoffs for yourself.

(there are some benefits to using your own domain here – so if your chosen provider turns evil, you can easily move.  We support hosting your own domain at FastMail)

Business – Orders/Receipts/etc

In any situation where you would normally use paper, the similar properties of email (immutibility and persistence) make it a worthy replacement.

(aside: the lack of reliable delivery authentication is a major reason why email hasn’t totally replaced fax – it’s the one major feature that fax has and email doesn’t)

Summary

Social networking is fine for the “fun stuff” – parties, cat pictures. For serious things though – email is still king.

Advertisements
Posted in Uncategorized | Leave a comment

The three types of data

I’ve used this taxonomy of data to help shape my thinking for many years now.

One of the things that frustrates me the most about Linux is the dotfile madness and caches spread throughout $HOME. Modern Windows is getting much better at offering this separation, but still falls short.

A lack of principled separation of data types leads to suffering.

Without further ado; the three types of data:

  1. Own Output – creative effort you have produced yourself. In theory you can reproduce it, though anyone who has lost hours of work to a crash knows just how disheartining it can be to repeate yourself.
  2. Primary Copy of others’ creative output. Unreproducable. Lose this, it’s gone forever.
  3. Secondary Copy of anything. Cache. Can always be re-fetched (possibly at some cost)

There’s a bit of blurring between categories in practice, particularly because you can sometimes find secondary copies in a disaster and get back primary data you thought was lost. It’s never pretty though. And in converse, sometimes you are the secondary copy which saves the day when a primary goes away.

Given these classifications, it’s easy to reason about some things:

Care of data

Own Output – stick it in version control. Easy. Since the effort that goes into creating is so high compared to data storage cost, there’s no reason to discard historical work. The repository then becomes a “Primary Copy”, and we fall through to;

Primary Copy – back it up. Replicate it. Everything you can to ensure it’s never lost. This stuff is gold.

In FastMail’s case as an email host, it’s other peoples’ precious memories. We store emails on RAID1 on every backend server, and each copy is replicated to two other servers, giving a total of 3 copies on RAID1 or 6 disks in total with a full copy of every message on them.

One of those copies is in a datacentre a third of the distance around the world from the other two.

On top of this, we run nightly backups to a different operating system with different configuration and a different file system.

Secondary Copy – disposable. Who cares. Actually, we do keep backups of Debian package repositories for every package we use just in case we want to reinstall and the mirror is down. And we keep a local cache for fast reinstalls in each datacentre too. But if something happens to them, meh. Re-download.

Cached data

It’s amazing how much stuff is just cached. For example – Operating System installs. The annoying thing about reinstalling a home computer is that you add a bunch of category 1 (own creative output) to the install. You choose a bunch of options during the install (modern installers are getting better at doing this up-front and then chugging away for half an hour rather than asking something new every 5 minutes, but still).

And it’s still not done. There’s all the other programs to install. Not so much on a Linux system where you just add a list of repositories to apt/yum and then install the package list… but still work. On Windows, it’s a bunch of different installers, each with their own click-through and possibly “enter licence code”.

And then you still have to configure each app to your liking.

Finally, done. You have a system which is 99% cache, 1 percent own creative output. Intermingled. Reinstalling will be just as much work next time. Ouch.

Separation of data types

We fixed that at FastMail by never changing config files directly.. All config goes in templates in git. No ifs, no buts. The process of reinstalling a machine is a clean install with FAI which installs the operating system and then builds the config from git onto the system. Repeatably. Meaning the OS install is 100% cache, and hence disposable.

If I was doing it again today, I would probably build from puppet. Right now we use Makefiles and perl’s Template-Toolkit to generate the configuration files. You can ‘make diff’ to see what’s different between a running machine and the new configuration, then ‘make install’ to upgrade the config and restart the related services.

Finally, user data. It goes on different disk partitions. The default “reinstall” leaves it untouched. The configuration files to access it are built from git, and go on the disposable OS disk. By keeping this separation clear, reinstall is a breezy, and considered “safe at any time”. With failover configurations for every service, it should never take more than 20 minutes to shut down a host, replace the hardware, reinstall the OS and have it back up and running. For hosts which don’t store user data locally (most of them), that’s it!

Log data also goes to a separate partition. It’s kind of “user data” too. Not replicated quite so aggressively as emails, because it’s frankly less precious.

Designing for types of data

So you’ve read this and you’re all excited to make life easier for the users of a piece of software you work on? It all comes down to one thing:

SEPARATION

The biggest sin I see is huge configuration files with hundreds of options, where the user is expected to make manual changes to a few lines. Then the next version comes out, and there are a few more options in the default config file, so the user has to manually merge their changes forwards, or be without those changes. No, NO, NO. Bad programmer. The default configuration is “cache” – it comes from somewhere else. The explicit changes the user has requested are “Own Output” for the user, but they are “Primary Copy” to the developer. Breaking them is a crime against the poor sucker who uses your app.

The next worst, and this is endemic in Linux, is storing cached data in $HOME/.appname/cache/ or similar. This is really annoying because it bloats backups. The sucky thing is, there’s nowhere else reliably available. At least keep it separate so the poor user can back up just your $HOME/.appname/ directory and know it’s their local changes, and ignore your $HOME/.cache/$appname/ or $HOME/.appname-cache/ directory

I find this taxonomy provides a lot of insight into the world of data. I would love to hear if there’s any major categorisation I’m missing, or other sensible lines to break data along

Posted in Uncategorized | Leave a comment

SSL – not easy to get right

I’ve been ignoring blogging here for a long time – it’s time to write something!

Customer Support

One of my roles is providing technical support for queries related to my areas of the system which can’t be resolved by our frontline support staff. This usually means either a bug at our end, or some odd issue which hasn’t been seen before.

Malfunctioning software

Here’s something interesting that came up with a customer query last week. They were using XFCE’s built in “biff” equivalent, but it was failing to list folders. In the default configuration it only counts UNREAD messages in INBOX, but you can ask for a server folder listing and add additional folders to watch.

The user’s problem was that upon opening the folder list, the program would sit saying “loading” forever. It worked fine with their gmail account using the same “imap” backend, so they wanted to know why it didn’t work with our server.

Red herrings

My initial debugging step, in a situation like this, is to enable protocol debugging on our Cyrus IMAP server. This is done by creating a folder with the user’s username in $confdir/log/. Upon seeing this directory, Cyrus daemons will open a file per process and log all protocol traffic after the login to the file.

The debugging showed that the mailwatcher plugin uses multiple list commands, like this (listing from my own account):

00006 LIST "" "%"
* LIST (\HasChildren) "." INBOX
* LIST (\Noselect \HasChildren) "." RESTORED
* LIST (\Noselect \HasChildren) "." user
00006 OK Completed (0.020 secs 225 calls)
00007 LIST "INBOX." "%"
* LIST (\HasNoChildren) "." INBOX.AANotify
* LIST (\HasNoChildren) "." INBOX.AATemp
* LIST (\HasNoChildren) "." INBOX.AAUnprocessed
* LIST (\HasNoChildren) "." INBOX.Account
* LIST (\HasChildren) "." INBOX.Archive
* LIST (\HasNoChildren) "." INBOX.Chats
* LIST (\HasNoChildren) "." INBOX.Drafts
[...]

The debug logs on the server showed it completing the initial top level listing of the INBOX, but then the client wasn’t listing the subfolders as expected.

Another one of our team also uses the same program, and he said it was working for him – but he didn’t have subfolders.

Another potential candidate was a folder named INBOX._dup_ in the user’s namespace. The underscores were a potential piece of weirdness.

I had a look at the source code for xfce4-mailwatch-plugin online and couldn’t see any obvious causes, so I recommended the user file a bug with XFCE and left it there.

A deeper look

But I do run XFCE – and something annoying me with their terminal app a few days later convinced me I should sign up with the XFCE bug tracker and clone a couple of their git repositories. While I was doing that, it made sense to check out the mailwatch program again too.

I tried it – and amazingly enough, my own mailbox had the same issue.

So I built from git and started debugging. There are two things which are very useful here. One is the compile-time option --enable-debug=yes which makes DBG() statements in the code print to stderr. The other is switching the XFCE panel itself to debug mode. This is documented here: http://wiki.xfce.org/howto/panel_plugin_debug

Debugging showed that it was parsing just some of the INBOX folder listing before freezing for long enough to obviously be a timeout, and saying “(nada)” returned. The code shows this means that a zero length response was returned by imap_recv.

GnuTLS and SSL

We don’t allow non-encrypted IMAP to our servers because unencrypted email => identity theft in this day and age, so every test was over SSL. On a whim, I created an ssh tunnel directly to the backend server with my mail, and tried setting up an account without SSL to the tunnel port. This time it worked fine!

Woah. There was already a voice in the back of my head saying “buffers” – particularly since it always failed at the same folder and there was nothing special about the name.

So I looked some more at GnuTLS documentation. And hit a falacy that I already knew. It’s one of the worst things about working with SSL. We use openssl in Cyrus, but the same conditions hold with GnuTLS.

  1. If you write encrypted data into the decryption function, it doesn’t mean anything will come out (possibly multiple times)
  2. If there’s nothing readable on the underlying socket, it doesn’t mean there’s nothing still in the decrypter buffer.
  3. The output may be larger than the input.

In this case, xfce4-maliwatch-plugin fails to account for item 2. It checks for data to read from the underlying filehandle, and if there is none, it never even checks if there’s any more to read from the decrypter function.

Buffer sizes

In this case, my folder listing was 1280 bytes, which fits into a single ethernet packet. But the output buffer was only 1024 bytes – so the gnutls decoder had extra data left after the first read. This is a common occurrence in C code where arbitrary fixed size buffers are the norm, but it’s going to happen anywhere at some point. You can’t have infinitely expanding buffers – and see point three above, sometimes the output is larger than the input.

Working around

The ideal solution is to use non-blocking gnutls and only select if you have already had a zero byte read from the decrypter. There’s more in this thread http://lists.gnu.org/archive/html/gnutls-devel/2011-05/msg00028.html.

Their basic point is – if you’ve already had a zero byte read from the pipeline, you’re not going to get any more out unless you put something more in first! Fair enough.

So the real solution would be to track zero byte reads and only THEN go back to selecting on the file handle.

But there’s a really skanky workaround possible – just bump the output buffer to 4096 bytes. Big enough that any likely single packet will fit comfortably. It’s not a real solution, but I proposed it on the bug anyway:

https://bugzilla.xfce.org/show_bug.cgi?id=9284

For all that it’s not a “real” solution, it will work around the bug in one very simple change.

Next steps

Since I’m a sucker for fixing things, I’m going to have another look at the code and see how hard the real fix is.

Still, it’s amazing how often a simple workaround is “good enough”. If this was affecting our production systems, I would have rolled out the buffer size bump immediately to avoid the symptoms while working on a proper cure.

Posted in Uncategorized | Leave a comment

So what would a zen master do?

Mostly, not be in this situation.  Unfortunately, I often find myself in the situation of trying to figure out how to do something with a piece of software, and the official documentation isn’t up to scratch.

Sometimes the only useful information I can find is from blogs of people who have hit the same issue, and have written about their experience and how they resolved their problems.

I am always very thankful when I find one of these, and I figure it’s about time I paid them back in kind by writing about my issues and how I solved them.  I’m not looking for lots of readers – or even regular readers – just hoping that what I write will one day be found and be useful to somebody.

Posted in Uncategorized | Leave a comment