Setting Up Your Own Email Server With hMailServer

Setting up your own email server with hMailServerSubmitted by Dominic Ryan on Wed, 2007-04-11 15:55.

Categories: 

How To Guides

Email

There are a many email servers for Windows out there that range in price depending on functionality and

the software vendor. However, it is not often you find an email server for Windows that is open source

(free to use and modify via the GPL license), feature rich, stable and very well supported. There are some

notable free offerings such as MailEnable Standard and Windows 2003 Server even comes with a basic

POP3 email server, but it seems that they are always just crippled enough to make you want something a

bit more. This is where hMailServer comes in as it offers just about everything you could want in an email

server. SMTP with advanced routing abilities, POP3, IMAP4, distribution lists, powerful anti-spam tools,

real time anti virus protection, PHP/COM based web administration, log file analysis, configurable server

messages and rules, advanced performance options. The list goes on, and all this in a RFC compliant

package with an easy to use GUI interface.

In this guide we'll cover the basics of what you need to do to configure your email environment properly

using hMailServer 4.3.1 as your email server. This includes;

Configuring you DNS records

A (Address) record

PTR (Pointer Record) record

MX (Mail eXchange) record

Considerations for hosting from home

Installing hMailServer

Select installation path

Select installation type

Select built-in or external database

Select program group for start menu access

Configure hMailServer

Create domain

Create postmaster account

Create abuse alias for postmaster account

Set SMTP host name

Configure RFC compliance

Configure SMTP relay options to prevent open relay

Test your email environment

Test against dnsreport

Test for open relay

Configuring your DNS records

Before we install your mail server it is a good idea to ensure you DNS records are correctly setup for email. DNS is vital to the operation of the Internet, and in a nutshell what it does is create a link between a human friendly domain name (e.g. example.com) and an IP addresses. When a user enters a web address into their browser, that domain is then resolved to an IP address which is then used to communicate. When hosting your own email server there are three very important types of DNS records to consider, and these are A, PTR and MX records.

An A record, or Address record is one of the most common types of DNS records you'll come across. Its purpose is to simply create a link between an IP address and a domain name. For example for the domain name of example.com you would have an A record that contained the IP address of the server that was hosting example.com. With A records the domain must be unique, but the IP address does not. This means you can have as many domain names pointing to the same IP address as you want, and is used extensively these days for both Email and Web hosting. In regards to your Email environment, it is best to create a separate A record that identifies your email server. This can be something like mail.example.com or similar. You'll need to create an A record for each server you plan on using as a mail server.

A PTR, or Pointer Record is unique in DNS in that you can only have one PTR address per IP address. This is because the function of a PTR record is to resolve a human friendly name from an IP address, instead of the other way around. Because you are resolving a domain name from an IP address, there can only be one authoritative record. If possible it is best to set your PTR record of the IP your email will be hosted from to be the same as your A record (e.g. mail.example.com), and you may need to contact your hosting company to do this for you.

Finally we have the MX, or Mail eXchange record which is used by other mail servers to direct email to the right place. Unlike an A or PTR record an MX record is not associated with an IP address in anyway, but instead contains the human friendly name of the A record you wish to use for your mail server. This might sound a little redundant as all it is doing is pointing from one record to another, but MX records also have one other important function. This is to establish the pecking order of your email servers by using a preference field in which you can enter a numerical value (the lower the number, the more important that server is) to define in what order other email servers should contact your email servers. If the email server with the highest preference (lowest number in preference field) is not contactable, then incoming email servers will simply use the server identified by the MX record with the next highest preference.

Considerations for hosting from home

Before we go on it is important to note that if you are wanting to host your own email server from home

over a standard ADSL or Cable connection, then there are a few things that may make your environment

a little more complex. These are;

You'll most likely have a dynamically allocated IP address, meaning that the A record for your

mail server will need to be updated everytime your IP address changes.

You most likely will not be able to set a PTR record for your IP address, especially if you are

using a dynamically allocated IP address.

A lot of email servers block email originating from dynamic IP addresses.

A lot of ISP's block the standard TCP/IP port numbers used by email servers on home

connections.

You ISP may well have prohibited hosting your own email (or web) server over your home

broadband connection in their terms of service or acceptable usage policy. 

With exception to the last few points, these issues can usually be pretty easily be overcome. To update

your mail server A record every time your IP changes you'll need to use a Dynamic DNS service. These

services are all over the web, and will require you to install a Dynamic DNS client on your server which

updates your DNS records everytime your local IP address changes. You'll probably won't be so lucky in

regards to the PTR record as ISP's usually have their own and will almost never change these for you. A

lot of email servers will check for the existence of a PTR record for an IP before accepting any email from

it, so as long as you do have a PTR you should be ok. If you are hosting your email server on a dynamic

IP address then you'll need to look at using an SMTP relay server (sometime called smart relay) which is

usually your ISP SMTP server. With this cause all outgoing email from your server is sent out through

your relay server which should be set up properly and will allow you to send email to servers that don't

allow incoming email from dynamic IP addresses.

Among the trickier problems to solve is that a lot of the time ISP's will block commonly attacked ports on

home broadband services to try and stop (or slow at least) the spread of viruses and prevent the

saturation of their network. Some ISP's will let you remove these blocks if you ask, or via a web interface.

If your ISP has specified in the TOS or AUP that it does not allow the hosting of services over your

connection, then your are bang out of luck. My suggestion is to look elsewhere as there are plenty of

ISP's that will give you unrestricted use of your connection (as it should be)

Installing hMailServer

That is most of the heavy stuff out of the way. From here on in it is all installing and configuring hMail, which is all done via a GUI interface meaning I can use screen shots to do the talking and is hopefully easier to follow. Click any of the images to get a pop-up window displaying a full sized image.

After downloading the latest stable hMail server installer package (currently 4.3.1) from the hmailserver website, double click on it to initiate the install process as shown below in figure 1.

Figure 1

Select the installation path for hMail as shown below in figure 2.

Figure 2

Select a full or custom install of hmail (full recommended) as shown below in figure 3.

Figure 3

Select wether to use the MySQL server built into the hMail distribution, or use an external database. If this is a dedicated email server then it is recommended to use the built in database server as shown below in figure 4. However, if you already have MySQL installed (or are planning to) or would like to use Microsoft SQL server then select the external database server option.

Figure 4

Set the start menu program group for hMail as shown below in figure 5.

Figure 5

Confirm your settings as shown below in figure 6.

Figure 6

Click install and hMailServer will be installed as shown below in figure 7.

Figure 7

Once installation is complete, make sure the "Run hMailServer administrator" option is checked as shown below in figure 8 and click finish.

Figure 8

Configuring hMailServerWith the installtion of hMailServer successfully completed, the next step is configuration. The configuration steps below show you how to add a domain, add an account, create an alias, setting the server host name, configure RFC settings and configure SMTP relay options to prevent open relay. Start by clicking the Add domain button as shown below in figure 9.

Figure 9

Enter the domain name as shown below in figure 10, and then set the catch-all address. If a mail is sent to an address on your domain that does not have a POP account or alias, then it is redirected to the catch-all address. In this example we have set the catch-all address [email protected]. From here you can also set the global maximum mailbox size as well as the maximum message size for your domain. Once the domain is created you are also able to access several other tabs to set global settings, but they are not covered in this guide.

Figure 10

With the domain setup, it is now time to create accounts. To be RFC compliant all domains should accept email to a postmaster account, and as we have set postmaster to be the catch-all we will now set up an account for it. Place the name of the account in the Account address feild, and then enter a password as shown below in figure 11. You can also set individual mailbox and message size, as well as many other options not covered in this guide such as Active Directory intergration, auto-reply, forwarding, signature and fetching of email from external accounts.

Figure 11

Sometimes it is not practical or desirable to setup an account for every email address you want, and in this case it can be handy to uses an email alias that points to an existing account. In this example we will create an alias of [email protected] that points to [email protected]. Simply enter the alias you'd like to use (in this case abuse) in theRedirect from feild and enter the account and domain ( in this case postmaster and example.comrepsectively) in the To feilds. Like the postmaster address, domains are also required to accept email to the abuse email address to be RFC compliant.

Figure 12

Now that we have a domain, account and alias setup lets look at selecting what email services we want to use. Using the navigation window in the left, select and expand the Settings item and then select the Protocols option. Make sure you have at least the SMTP and POP3 servers ticked, as otherwise you

will not be able to send or recieve mail. You may not wish to use the IMAP server, but you will need it if you wish to provide webmail functionality to your users.

Figure 13

Next step is to set the server host name. In the navigation window on the left, expand the Protocolsitem and select SMTP. This can be very important as some email servers will not accept email or mark it as spam if the host name does not match the hostname specified in the MX record we set earlier. In the Host Name field enter the full host name you specified in your MX records. In this example we'll use mail.example.com. Note: If you are looking to host your own email server over your home broadband connection, then you will want to enter the name of your ISP's SMTP server in the Relayer field. If your ISP requires authentication, then you'll also need to provide those details in the fields below.

Figure 14

With the host name set we will now set some extra RFC compliance settings. From the page you are on, simply click on the RFC Compliance tabe at the top. It is important that your email server be RFC compliant as otherwise it is likely that many domains will mark your email as spam, and that is if they

accept it at all. One of the RFC requirements for email servers is that they accpet a null sender address. You can enable this ticking the Allow empty sender address option. It is also a good idea to enable the Allow incorrectly formatted line endings option as several popular email server packages out their vary slightly in the way they terminate email messages, and without this option set you email server may not be able to recieve emails from them

Figure 15

The final step in the basic configuration of your email server is to enure it is not an open relay. An open relay is when a server enables mails to be sent through it to other domains on behalf of domains that do not exist on the local server. Being an open relay is a very quick way in which to get yourself blacklisted, and it can be near impossible to get off these blacklists once you're on it. Luckily hMailServer makes in very easy to prevent this, and in fact by default you should not have to change a thing. Just to be sure though it is always best to check the settings. In the navigation window to the left, select the select and expand the Advanced menu item from under the Settingstree. From here select the IP Ranges option and then select the Internet option. All you have to do here is ensure that the Local to local, Local to external, and External to local options are ticked from under the Allow deliveries from section. Finally also ensure that the To remote accounts option is ticked from under the Require authentication for deliveries section. All these options are shown below in figure 16.

Figure 16

Testing the configuration

Almost done now, the only thing left to do is test your email environment to make sure everything is

configured correctly and ensure it is not an open relay. First stop is to plug your domain in

atDNSReport.com which will give you a good overview of how well your email system is setup. If

DNSReport finds any issues of concern it will notify you and offer advice on what needs to be fixed.

There are several tests available for free on the internet for testing your email server for open relay, and

the ones I'd suggest using are the tests at abuse.net and aupads.org. Once you have passed these

tests then you should be all clear to go ahead and start creating other account and finally sending and

recieving email from your very own email server.