Beating IMAPS and SMTPS (TLS) to a pulp

by Paul Fischer

I think I have finally beaten IMAPS (IMAP with SSL/TLS don’t ask me which) and SMTPS (SMTP with TLS) certificate configurations on my server.

In my case, I am using an OS X 10.3 which has built-in IMAP and SMTP (postfix). They both come pre-configured to use a server.pem file in /var/imap and /etc/postfix respectively.

We use secure IMAP and SMTP to protect our home email server. I’m mostly concerned about people not being able to sniff our passwords.

We use a self-signed certificate and for the past several years we’ve been using different certs for IMAP and SMTP. In the new setup, we use a single cert and put it in both directories. Easy peasey!

This new config satisfies Apple Mail (a.k.a. Mail.app) and Entourage so we’re very very happy with it.

The IMAP and SMTP server config is stock Apple OS X server 10.3. We chose CRAM-MD5 authentication for both. SMTP SSL is set to “use” and IMAP SSL is set to “require”. Once you do that, all you need to do is make the certificate and put them in place. Then restart the server.

1) Create the CA and the cert. You’ll need to answer some questions. The key is for “Common Name” to be the FQDN of your mail server. If your mail server is mail.foo.com, use mail.foo.com for “Common Name”. I used it for “Common Name”, “Organizational Name”, and “Organizational Unit” just to be safe.

sudo openssl req -new -x509 -nodes -out imapd.cer -keyout imapd.pem -days 3650

2) Concatenate the two files into the server.pem file that OSX wants.

cat imapd.pem imapd.cer > server.pem

3) Copy server.pem to the correct locations
cp server.pem /etc/postfix
cp server.pem /var/imap

4) Restart IMAP and SMTP servers

On the client side, use Mail.app to connect to the IMAP server using SSL on port 993. You’ll get an error message about the cert not being signed by a valid root cert. That’s OK. Hit the “show certificate” button. Holding down the option key, click on the picture of the certificate and drag it to the desktop. It should create a file with the FQDN of your server and “.pem” on your desktop. Open your keychain and select X.509 anchors from the left hand column. The choose File->Import and choose the .pem file.

Now, when you look at your certificates you should see one with the FQDN of your server. Double click on it, and scroll to the bottom. Hit the little black triangle to the left of “Trust Settings”. Scroll down and the first box you come to is labeled “When using this certificate”. Change it from “Use System Settings” to “Always Trust”. This should change all the boxes below it to “Always Trust”. Close the window and exit KeyChain.

Finally, configure outbound mail to use your server on port 25, with SSL. Choose “MD5 Challenge-Response” for the authentication method first. If this doesn’t work, back off to “password” but make sure that authentication mode is configured on the server. Our old G4 iBook wasn’t able to do MD5 authentication. But the new Macbook Pro was. It may be some difference in the Intel vs PPC binaries, or it may have something to do with the speed of the machine.

I hope this helps anyone else who is trying to make this work. Cheers to you for fighting the good fight.

-Paul



Comments are closed.