How to Self-Host an Email Server
Running your own email server is one of the more demanding self-hosting projects you can undertake. Unlike a web server or database that only communicates with known clients, an email server interacts with thousands of other mail systems worldwide, and each of those systems has its own criteria for accepting or rejecting your messages. The steps below address the technical requirements, the common pitfalls, and the ongoing work needed to keep a self-hosted email server running reliably.
Choose Your Server and Hosting Provider
The hosting provider you choose directly affects your ability to send email. Many cloud providers, including AWS, Google Cloud, and Azure, block outbound traffic on port 25 by default and require you to request an exception before your server can send mail. Other providers like Hetzner, OVH, Contabo, and Vultr allow port 25 by default, making them more straightforward choices for email hosting.
The server's IP address matters as much as its specifications. Ideally, you want an IP address that has not been used for spam in the past. Check your prospective IP against major blocklists using tools like MXToolbox or multirbl.valli.org before committing to a server. If the IP appears on blocklists, request a different one from your provider.
For hardware specifications, a VPS with 4 to 8 GB of RAM, 2 CPU cores, and 40 to 100 GB of SSD storage handles most small to medium deployments comfortably. Storage needs depend on retention policies and attachment sizes, so plan for growth. A dedicated server is not necessary for most deployments, as modern email platforms run efficiently on virtual machines.
Set the server's hostname to your mail subdomain before installing any software. For example, if your domain is example.com, set the hostname to mail.example.com. The hostname should resolve to the server's IP address through both an A record (forward DNS) and a PTR record (reverse DNS). Many hosting providers offer a control panel where you can set the PTR record, or you may need to contact their support.
Register a Domain and Configure DNS
Your domain's DNS records tell other mail servers where to deliver mail for your domain and how to verify that messages from your domain are legitimate. Missing or incorrect DNS records are the most common cause of delivery problems for new email servers.
Start with the essential records. Create an A record pointing mail.example.com to your server's IP address. Create an MX record for example.com pointing to mail.example.com with priority 10. Verify that your hosting provider has set the PTR record for your IP address to match mail.example.com.
Next, add the authentication records. The SPF record is a TXT record on example.com that lists authorized sending IPs: v=spf1 mx a -all. This tells receiving servers that only your MX and A record IPs are allowed to send mail for your domain. The -all directive instructs receivers to reject mail from unlisted sources.
DKIM requires generating a key pair, which your email platform handles automatically during setup. After installation, the platform provides a DNS TXT record to publish, typically at a selector subdomain like dkim._domainkey.example.com. This record contains the public key that receiving servers use to verify message signatures.
The DMARC record ties SPF and DKIM together. A reasonable starting policy is v=DMARC1; p=quarantine; rua=mailto:dmarc@example.com, which tells receivers to quarantine messages that fail authentication and send aggregate reports to your specified address. After monitoring reports for a few weeks and confirming legitimate mail passes, you can tighten the policy to p=reject.
Install Your Email Server Platform
With DNS configured, install your chosen email platform on the server. The major platforms have well-documented installation procedures. For Mailcow, this means cloning the repository, running the configuration generator, and starting the Docker Compose stack. For iRedMail, download the installer, run it on a clean OS installation, and answer the interactive prompts. For Mailu, use the web-based configuration wizard to generate your Docker Compose files, then deploy them.
Whichever platform you choose, install it on a clean server dedicated to email. Running other services alongside your email server creates security risks, resource contention, and configuration conflicts. The email server should be the only application on the machine, with the exception of supporting infrastructure like the firewall and monitoring tools.
During installation, configure your domain, create the initial administrator mailbox, and verify that the platform generates DKIM keys for your domain. Most platforms display the DKIM DNS record you need to publish, so add it to your DNS if you have not done so already. Allow time for DNS propagation before testing, as DKIM verification will fail until the record is visible globally.
Configure Email Authentication
After installation, verify that all authentication mechanisms are working correctly. Send a test email to a Gmail address and examine the headers by clicking "Show original" in Gmail's menu. Look for spf=pass, dkim=pass, and dmarc=pass in the Authentication-Results header. If any of these show fail, your DNS records need correction.
Use the mail-tester.com service to get a comprehensive score of your email configuration. Send a message to the address shown on their site, and they will analyze your SPF, DKIM, DMARC, message content, and blocklist status. A score of 9 or 10 out of 10 indicates a properly configured server. Scores below 8 usually point to specific configuration issues that the report identifies clearly.
Check your IP address against major blocklists. Even a clean IP should be verified against Spamhaus, Barracuda, SORBS, and SpamCop. If your IP appears on any list, follow the delisting procedure for each service before sending production email.
Secure the Server
Email servers are high-value targets for attackers because compromised email accounts can be used for phishing, spam, and social engineering. Proper security configuration is not optional.
Configure the firewall to allow only the ports needed for email operations: 25 (SMTP), 465 (SMTPS), 587 (submission), 143 (IMAP), 993 (IMAPS), 80 (HTTP for Let's Encrypt), and 443 (HTTPS for webmail). Block all other inbound ports. If you do not use POP3, close ports 110 and 995 as well.
Install and configure Fail2ban to monitor authentication logs for both SMTP and IMAP. Fail2ban automatically blocks IP addresses that make repeated failed login attempts, protecting against brute-force attacks. Most email platforms include Fail2ban configurations or equivalent built-in protection. Verify that TLS is enabled on all services and that the server uses strong cipher suites. Disable TLS 1.0 and 1.1, and prefer TLS 1.3 where supported.
Test Mail Delivery
Before directing real traffic to your server, test delivery to the major email providers. Send messages to accounts on Gmail, Outlook.com, Yahoo Mail, and any other providers your recipients commonly use. Check whether messages arrive in the inbox or the spam folder.
If messages land in spam, the cause is usually one of three things: a blocklisted IP address, failed authentication checks, or low sender reputation. New IP addresses start with no reputation, and some providers treat unknown senders cautiously at first. Sending a moderate volume of legitimate mail and having recipients mark it as "not spam" gradually builds your reputation.
Consider using an SMTP relay service like Amazon SES, Mailgun, or Postmark for outbound mail during the reputation-building period. These services provide high-reputation IP addresses for outbound delivery while your inbound mail still goes directly to your self-hosted server. Once your server's IP has built sufficient reputation, you can switch to direct delivery.
Set Up Monitoring and Backups
A self-hosted email server requires active monitoring. At minimum, monitor disk usage (full disks cause silent mail rejection), service health (Postfix, Dovecot, and Rspamd should always be running), mail queue size (a growing queue indicates delivery problems), and TLS certificate expiration.
Tools like Uptime Kuma, Monit, or Prometheus with Grafana provide dashboard visibility into server health. Configure alerts for critical conditions like services going down, disk usage exceeding 80%, or the mail queue growing beyond normal levels.
For backups, capture three things: the mail storage directory (typically /var/vmail or a Docker volume), the database containing account and configuration data, and the DKIM private keys. Test backup restoration periodically to verify that backups are complete and functional. Storing backups off-server, on a separate VPS or object storage service, protects against hardware failure on the mail server itself.
Self-hosting email is achievable for anyone with basic Linux administration skills, but it demands careful attention to DNS configuration, authentication records, and ongoing maintenance. The biggest risk is not the initial setup but the ongoing commitment to monitoring, security updates, and reputation management. Plan for this commitment before deploying, and your self-hosted email server will serve you reliably for years.