10 Best and Worst Practices for Running an Email Server

Enforce Transport Layer Security (TLS) for both incoming and outgoing emails to protect data in transit. TLS is an optional feature that comes into play if the receiving server supports STARTTLS, a mechanism to upgrade an existing TCP/IP connection from a non-encrypted to encrypted. Unfortunately, a proxy server, an ISP, or a malicious user watching the communication can easily hide the STARTTLS feature during the initial communication. This results in the message being delivered without encryption. Mail Transfer Agent-Strict Transport Security (MTA-STS) solves this problem, making encryption mandatory when two SMTP servers communicate. Click here for details.

Set up Sender Policy Framework (SPF), DomainKeys Identified Mail (DKIM), and Domain-based Message Authentication, Reporting, and Conformance (DMARC) to authenticate and validate emails, reducing the risk of spoofing and phishing.

A PTR record is the opposite of an A record. In other words, you can resolve a hostname from an IP address using a PTR record. This type of record is required when sending outgoing emails. Failure to have a PTR record will result in getting your messages into the recipient's junk folder. Click here for details.

Port 25 is typically used for incoming emails. Internal users should use either 587, 465 or a non-standard port for sending their outgoing messages. Therefore, there is no need to allow SMTP authentication on port 25, which will prevent malicious users from guessing passwords through your SMTP server. Click here for details.

Many email servers contain a built-in mechanism to confirm best practices are being followed, and it is important to run these checks at least once a month. For example, Xeams contains diagnostic checks for inbound as well as outbound. Click here for details.

Spam filtering for outgoing emails is equally important as incoming messages. There are several benefits:

• You want to ensure a malicious user is not misusing a comprised account
• You want the spam filter to improve its filtering by learning from outbound messages
• An audit trail and logs for message tracking
• Ensure sensitive information is not leaked by mistake

Encryption technologies have evolved tremendously since its inception in the early 1990s. As vulnerabilities are discovered, newer specifications replace older ones, and therefore, disabling technologies that are no longer considered secure is important. For example, SSL 1.0, 2.0, 3.0, and TLS 1.0 is no longer considered secure. A preferred version is either TLS 1.2 or TLS 1.3. Consider switching to newer versions of encryption for your email infrastructure. Click here for details.

Temporary email addresses provide a secure way of communication if you know the relationship will not be needed after some time. A good example is when purchasing a one-time product from any vendor. Email servers often offer features allowing users to create temporary email addresses that do not last more than a couple of months. Using such addresses hides the user's address, reducing unwanted messages from vendors. Click here for details.

There are two schools of thought when handling emails for invalid recipients:

• Accept the message but then generate a non-delivery report (NDR) to the sender
• Reject the message during SMTP communication

It is important to keep in mind that emails go through multiple SMTP servers, and NDRs are generated by the last server that accepts an email. Therefore, if your server accepts an email for invalid users, it becomes your responsibility to generate an NDR. This increases the volume of outbound emails, and potentially, you could become a victim of a reverse NDR attack. Therefore, rejecting invalid users during SMTP communication is a better approach.

Set up server monitoring tools to detect and address issues promptly, such as high server load, disk space, and unusual traffic patterns. Many email servers display such information right after an administrator logs in. You could also configure the server to send email alerts if a critical error occurs.

Configure your email system to generate security alerts when unusual activities are detected. For example, you want to know if multiple attempts are being made to authenticate against your server. Many systems are often configured to display a warning message when credentials are incorrect, but they fail to take any proactive action when thousands of requests are submitted using bots.

Similar to any other public-facing server, an email server opens several doors for intruders to get in. Every TCP/IP server you enable should be considered a door an intruder can enter. For example, someone on the Internet can guess user ID/password combinations from an IMAP server.

Many organizations put a public-facing SMTP that listens on port 25. In such cases, every other service (POP3, IMAP, HTTP, HTTPS) should not be accessible from the Internet. Additionally, consider disabling SMTP authentication on port 25, as recommended above.

Although it is easier to integrate email users with the operating system, this strategy becomes a liability when someone's password gets compromised. Malicious users can not only send spam messages through a compromised user credential but also try targeting other services like RDP and web-based services. Decoupling the user accounts from the operating system could limit the damage to a single node.

Often, users forward their incoming emails to a different account, such as their gmail.com or hotmail.com address. Consider the following scenario, which creates a hidden problem:

• John (john.doe@yourcompany.com) decided to forward his emails to john.doe@gmail.com and wrote a rule on the server to accomplish this task.
• An incoming email from someone@hotmail.com is sent to john.doe@yourcompany.com
• The server-side rule forwards this message to Gmail but leaves the sender's email address as someone@hotmail.com
• Gmail's server gets an email where the sender is someone@hotmail.com, but the message comes from your server, which is not authorized to send emails for Hotmail.

This will be considered a forgery by Gmail, and you risk getting your public IP address into Gmail's blacklist.

To fix this problem, configure your email server to change the Sender's address in the email address before forwarding it. Click here for details.

By default, most firewalls do not block outbound traffic. This leaves port 25 open for misuse, and a compromised machine inside your LAN will end up sending outbound emails without your knowledge.

Therefore, it is important to block outbound traffic for port 25 from every other machine but the one where your email server runs.

User credentials are constantly exchanged between email clients and servers. Therefore, encrypting this communication is essential. Moreover, avoid using self-signed certificates and adding exceptions in your client for those certificates. Often, malicious users deploy proxy servers with their own SSL certificate and hope the end-user will ignore the certificate warning, leading to compromised credentials.

Many email servers, such as Xeams, can log messages if a user connects to the system without encryption, allowing administrators to take proactive action to prevent this from happening again.

Besides filtering emails, many spam filters can also report the outbound email activity. Such reports are very useful when detecting a compromised user. Of course, filtering outgoing messages will also prevent sensitive messages from leaking the company's IP.

Sophisticated email filtering systems allow end-users to maintain their while/blacklist addresses. These systems allow whitelisting individual email addresses as well as domains.

Administrators should ensure end-users do not whitelist public domains like Gmail, Hotmail, Yahoo, and others like that, which can be easily done if the email server is able to report such anomalies.

Another common but incorrect rule in an email filtering system is to whitelist their own domain. It is very common for spammers to put the recipient's address in the sender's field. Although such attempts are typically prevented by using SPF and DKIM, filtering gets skewed when incorrect whitelisted addresses are found.

Specifying an upper limit on the number of outgoing emails users can send in an hour is essential. This prevents an abrupt increase in the email flow in case an account is compromised.

It is common for companies to have multiple MX records with different priorities. A common strategy is to specify the higher priority MX to point to a spam filter and use a lower priority to go directly to the actual server, which is done to reroute messages if there is a problem with filtering.

Many spammers know this, so they deliberately target the lower priority MX with hopes that it will have less stringent filtering rules.