SMTP STARTTLS in sendmail 8.11/Secure Switch

[Terminology] [Compilation] [Configuration] [Operation] [Misc]

Introduction

Terminology

• CA: certificate authority,
• cert: a certificate (signed/issued by a CA),
• private key: the private key belonging to a cert.

Initial Setup

1. Install one or more CA certs in confCACERT
2. Install zero or more CA certs in confCACERT_PATH with (symbolic) links of their hashes pointing to them:

   C=Name_of_CA
   ln -s $C `openssl x509 -noout -hash < $C`.0
   

   (or sslc instead of openssl)

   These CA certificates are required to successfully authenticate another entity. The signature of the certificate presented by the other side is checked against these CAs. If one of them issued the certificate, the authentication is considered successful. Moreover, during the TLS handshake, the DNs of these CA certificates are sent to the client so it can properly select a certificate that is signed by one of those CAs.

3. Get a cert (make sure the CN is the fully qualified name of your host) and install it as confSERVER_CERT and the private key as confSERVER_KEY (make sure the file is only readable by root or the trusted user). For simplicity, use the same filenames for confCLIENT_CERT and confCLIENT_KEY, respectively.
   Note: the private key must not be encrypted. This is required for unattended startup of sendmail. Otherwise someone/something would have to enter the passphrase each time sendmail is started as server or client.
4. If you run sendmail 8.11 and your OS doesn't have /dev/urandom(4), then you need to setup a source to seed the pseudo random number generator. For Solaris 7 and 8, you may have a look at a kernel module for /dev/random (please check yourself whether it is ok to use!), or see whether Sun has a package called SUNWski for your OS. It is strongly advised to use at least EGD (Entropy Gathering Daemon) and compile sendmail with the flag EGD, and point confRAND_FILE to the socket used by EGD (use `egd:' as prefix!). If neither /dev/urandom(4) nor EGD are available, you have to make sure that useful random data is available all the time in confRAND_FILE (use `file:' as prefix). If the file hasn't been modified in the last 10 minutes before it is supposed to be used by sendmail the content is considered obsolete. In this case, the PRNG for TLS is only seeded with other random data if the DontBlameSendmail option InsufficientEntropy is set. This is almost always not sufficient for security.

Extract from a .mc file:

define(`CERT_DIR', `MAIL_SETTINGS_DIR`'certs')dnl
define(`confCACERT_PATH', `CERT_DIR')dnl
define(`confCACERT', `CERT_DIR/CAcert.pem')dnl
define(`confSERVER_CERT', `CERT_DIR/MYcert.pem')dnl
define(`confSERVER_KEY', `CERT_DIR/MYkey.pem')dnl
define(`confCLIENT_CERT', `CERT_DIR/MYcert.pem')dnl
define(`confCLIENT_KEY', `CERT_DIR/MYkey.pem')dnl

Initial Test

250-STARTTLS

% telnet localhost 25
Trying 127.0.0.1...
Connected to localhost
Escape character is '^]'.
220 local.sendmail.COM ESMTP Sendmail Switch 2.0/Switch 2.0; Thu, 9 Sep 1999 10:48:44 -0700 (PDT)
ehlo localhost
250-local.sendmail.COM Hello localhost [127.0.0.1], pleased to meet you
250-ENHANCEDSTATUSCODES
250-DSN
250-STARTTLS
250 HELP
quit

Operation

New Macros

{cert_issuer}

holds the DN of the CA (the cert issuer).

{cert_subject}

holds the DN of the cert (called the cert subject).

{tls_version}

the TLS/SSL version used for the connection: TLSv1, SSLv3, SSLv2.

{cipher}

the cipher suite used for the connection, e.g., EDH-DSS-DES-CBC3-SHA, EDH-RSA-DES-CBC-SHA, DES-CBC-MD5, DES-CBC3-SHA, RC2-CBC-MD5, RC4-MD5

{cipher_bits}

the key length (in bits) of the symmetric encryption algorithm used for the connection.

{verify}

holds the result of the verification of the presented cert. Possible values are:

OK

verification succeeded

NO

no cert presented

FAIL

cert presented but could not be verified, e.g., CA is missing.

NONE

STARTTLS has not been performed

TEMP

temporary error occurred, e.g., the other side replied with 454.

PROTOCOL

protocol error occurred

SOFTWARE

problems during the handshake at the TLS level. In this case the connection will be dropped.

{server_name}

the name of the server of the current outgoing SMTP connection.

{server_addr}

the address of the server of the current outgoing SMTP connection.

Relaying

To make things a bit more flexible (or complicated), the values for {cert_issuer} and {cert_subject} can be optionally modified by regular expressions defined in the m4 variables _CERT_REGEX_ISSUER_ and _CERT_REGEX_SUBJECT_, respectively. To avoid problems with those macros in rulesets and map lookups, they are modified as follows: each non-printable character and the characters '<', '>', '(', ')', '"', '+' are replaced by their HEX value with a leading '+'. For example:
/C=US/ST=California/O=endmail.org/OU=private/CN=Claus Assmann (Cert)/Email=ca+cert@sendmail.org
is encoded as:
/C=US/ST=California/O=endmail.org/OU=private/CN=Claus+20Assmann+20+28Cert+29/Email=ca+2Bcert@sendmail.org

Of course it is also possible to write a simple rulesets that allows relaying for everyone who can present a cert that can be verified, e.g.,

LOCAL_RULESETS
SLocal_check_rcpt
R$*		$: $&{verify}
ROK		$# OK

Allowing Connections

The rulesets tls_server and tls_client are used to decide whether an SMTP connection is accepted (or should continue).

tls_server is called when sendmail acts as client after a STARTTLS command (should) have been issued. The parameter is the value of {verify}.

tls_client is called when sendmail acts as server, after a STARTTLS command has been issued, and from check_mail. The parameter is the value of {verify} and STARTTLS or MAIL, respectively.

Both rulesets behave the same. If no access map is in use, the connection will be accepted unless {verify} is SOFTWARE, in which case the connection is always aborted. Otherwise, {client_name}/{server_name} is looked up in the access map using the tag TLS_Srv (or TLS_Clt), which is done with the ruleset LookUpDomain. If no entry is found, {client_addr}/{server_addr} is looked up in the access map (same tag, ruleset LookUpAddr). If this doesn't result in an entry either, just the tag is looked up in the access map (included the trailing :). Values for those entries in the access map should be:

VERIFY

verification must have succeeded

VERIFY:bits

verification must have succeeded and {cipher_bits} must be greater than or equal bits.

ENCR:bits

{cipher_bits} must be greater than or equal bits.

The RHS can optionally be prefixed by TEMP+ or PERM+ to select a temporary or permanent error. The default is a temporary error code (403 4.7.0) unless the macro TLS_PERM_ERR is set during generation of the .cf file.

If a certain level of encryption is required, then it might also be possible that this level is provided by TLS from a SASL algorithm, e.g., DIGEST-MD5. The number of bits used for the key of the symmetric cipher (here: 3DES) is stored in the macro {auth_ssf}.

Example: e-mail send to secure.example.com should only use an encrypted connection. e-mail received from hosts within the laptop.example.com domain should only be accepted if they have been authenticated.

TLS_Srv:secure.example.com	ENCR:112
TLS_Clt:laptop.example.com	PERM+VERIFY:112

Received: Header

(using ${tls_version} with cipher ${cipher} (${cipher_bits} bits) verified ${verify})

Testing

Advantages and Limits of TLS

• authentication: client and server of a SMTP connection can be identified.
• privacy/confidentiality: the transmission of an e-mail between a client and server utilizingSTARTTLS can not be read and retranslated into plaintext provided a sufficiently secure ciphersuite has been negotiated.
• integrity: the plaintext of an e-mail between a client and server utilizing STARTTLS can not be modified by an adversary provided a sufficiently secure ciphersuite has been negotiated.

• It does not provide end-to-end encryption, since a user can usually not control the whole transmission. This is in contrast to the use of TLS for http: here the user's client (a WWW browser) connects directly to the server that provides the data. e-mail can be transferred via multiple hops of which the sender can control at most the first.
• It does not provide message authentication, unless the e-mail has been sent directly from the client's (STARTTLS-capable) MUA to the recipients MTA which must record the client's certificate. Even then the message might be faked during local delivery.

Misc

Interoperability

Interoperability Problems

• CommuniGate Pro 3.2.4 (and 3.2.3): if this MTA is not correctly configured, it offers STARTTLS but then fails during handshake.
• CommuniGate Pro 3.3betaX: this MTA speaks SSLv3 but not TLSv1. (2000-05-19): This was a bug in older versions of CommuniGate Pro 3.3 which is solved in Beta7 (?).
• InterChange v3.61.01. This MTA seems to understand only SSLv3, but not TLSv1 or the default mode that OpenSSL uses. However, v3.61.06 seems to interact just fine with sendmail.
• sendmail can't talk with MTAs that use TLS implementations which use RSARef, if key length greater than 1024bits(?) are used. Please don't use RSARef since there are fully functional RSA implementations available.

The big problem is: you can't turn off STARTTLS in sendmail (as client) unless you recompile it. This feature is planned for the next release.

A patch has been posted to the OpenSSL mailing list which contains a program similar to s_client to connect to MTAs for testing STARTTLS.

Credits

References

If you prefer french instead of english, please read the instructions from Jean-Philippe Donnio.