Product » A free email server for Windows and Linux » FAQ

SMTP Intrusion detection

By its very nature, email servers have a public face. Companies offer email services to their employees and customers in order to facilitate collaboration with each other across the Internet. Every company that has a public domain allows SMTP traffic through their firewall in order to accept email messages from other users on the Internet. This high visibility makes an SMTP server potentially more susceptible to attack compared to other services. Therefore, you need to make sure that your email server is well protected from outside intruders and diffuse any harmful attacks before they occur. This white paper identifies weak links that are inherent to email servers and how to pro-actively prevent any future attacks.

SMTP Protocol - from security perspective

Many administrators spend a tremendous amount of time making sure that their network is secure and no outsiders with malicious intent can harm their system. It is said that your network security is as strong as your weakest link. This section describes a few potential security risks found in the SMTP protocol.

SMTP is designed to send and receive emails from other SMTP servers or clients. When a user within your company tries to send an email to a friend on the Internet, his/her email client will forward that message to your SMTP server, which in turn will then forward it to the recipient.

Prior to the actual email data being sent, a few messages are exchanged back and forth between the client and server. Both parties use these messages to identify each other and to provide a set of instructions that are mutually understandable. It is this conversation that can be exploited by a malicious user to find out potential vulnerabilities in your SMTP server.

The table below outlines the communication between a client and a server using SMTP. The sequence numbers in this table is referenced in the remainder of this document to identify each step.

Seq. #	Source	Action	Description
1	Client	New Connection	Client opens a new socket connection on port 25
2	Server	Replies and sends the header	This header may include the name and version of your SMTP server.
3	Client	HELO or EHLO	Along with this HELO, the client usually sends the fully qualified domain name of the sender.
4	Server	Replies to HELO	Server replies back to the HELO command along with all of the extended verbs that it supports
5	Client	Authenticate	Client tries to authenticate the user. This is not a required step and may not occur
6	Server	OK	Server sends an OK if the user is authenticated, otherwise returns an error code
7	Client	MAIL FROM	Client sends the email address of the sender. This may be empty.
8	Server	OK	Unless the email address violates some system policy, the sender's email address is generally accepted.
9	Client	RCPT TO	Client sends a list of recipients who should get this message
10	Server	OK	Server may decide to reject or accept the recipient(s) based on the local policy settings. If the client is authenticated, it may even allow to relay someone else's message
11	Client	DATA	This is the actual email body, which conforms to RFC 822 specifications
12	Server	OK	Send an OK if the message is accepted
13	Client	QUIT	Client sends a quit message to disconnect
14	Server	Close connection	Server disconnects from the client.

How to identify a malicious user?

Many administrators find out about an intrusion after their system has been exploited. Before malicious users discover a weakness in your system, they must find out what is available for them to exploit. They discover these weaknesses by sending unexpected messages and/or inspecting the data returned by the server. In most cases an actual email is never sent. Consequently, it is very difficult to find out if such activity is taking place against your server. The remainder of this article explores possible intentions of a user if a connection is terminated without sending an email.

Termination after socket creation

This corresponds to sequence number 1 and 2 in the table above. Possible causes of this type of termination are:

• Network problems on either side.
• User wants to find out what server are you using. Many SMTP servers as well as some network firewalls provide a mechanism to return a modified string that does not reveal the server's true identity. Therefore, spammers usually do not rely on this information.
• Your server is experiencing a "Denial of Service" attack, which occurs when a user opens too many connections on your network but does not disconnect them. As a result, your server refuses a connection to a legitimate user trying to send an email.

Termination after Hello

This corresponds to sequence number 3 and 4 in the above table. Possible causes of such a termination are:

• Network problems on either side.
• User wants to find out about the extended features supported by the SMTP server. For instance if you are using MS Exchange, it will return XEXCH50 as one of its supported features. If a malicious user discovers a security hole in MS Exchange he/she can exploit this information in order to get into your system.
• Your server is experiencing a "Denial of Service" attack.

Termination after Authentication

This corresponds to sequence number 5 and 6 in the above table. Possible causes are:

• Network problems on either side.
• Someone is trying to guess user id/password on your system. In this case the user is actually only guessing for a password - they already know the user id, which is usually is the first part of an email address.

Termination after specifying a recipient list

This corresponds to sequence number 9 and 10 in the above table. Possible causes are:

• Network problems on either side.
• Someone is trying to find out if your SMTP server is an open relay.
• This can also be an indication of a dictionary attack, which happens when a malicious user sends an email to an invalid user names hoping that a small percentage will get through.

How can you protect your server?

Here are some suggestions on how to protect yourself from such attacks.

• Use a tool such as Xeams that can determine and report such activities. Xeams provides an intuitive way of capturing the IP addresses of connections where these clients originate from and suggests solutions for resolution to the problem.
• Frequently check your firewall logs to make sure no unwanted activity is taking place.
• Have your users change their password on a regular basis.

Summary

Many SMTP servers are capable of logging activity and traffic patterns. Although it is a very good practice to review these logs, they are not enough of a deterrent to set off an alarm before an attack is about to occur. Therefore, a more proactive and user-friendly approach is required to identify potential threats. Xeams's built-in Intrusion Detector provides a comprehensive solution to alert and identify possible intrusions into your network via an SMTP server before they wreck havoc.