How do you protect your Mailserver from DDoS Attacks?

Which part of your email system is being targeted? What specifically are they targeting? Are they just making connections to a port, are they sending enormous amounts of spam, are they trying to brute a password?

Inbound mail traffic should be routed through lightweight MX servers. It's a common mistake to put an MX and a mail store on the same system. You can deploy new MX endpoints all day long and just update your DNS for it and email will still work pretty well.

Although there are perfectly reasonable arguments against it, Fail2Ban or similar can shut down nuisance traffic on a mail store. You should beware though that it's difficult to ensure that Fail2Ban or other active-response log monitoring can't itself be abused to ruin your day.

If your mail store is just getting hammered, it's a tricky problem to solve without a lot of resources. If you're the only user on your system, there's no reason to publish a dns record for your mail store, so move it to a new ip and update your settings. If you can't do that and Fail2Ban can't resolve it, you get to start thinking about things like distributed mail storage. I've been wanting to check out dbmail for years (https://github.com/dbmail/dbmail/), maybe you can give that a whirl and link a writeup of your experiences with it.

I have experimented a bit here and there with dovecot-on-mysql and multi-master percona and all that and it's not fun or reliable.

If you're getting mind-blowing amounts of spam -- especially if it's newsletter signups -- it's possible you're being mailbombed. That sucks, there aren't a lot of good solutions for that right now, even Gmail users can be victimized by it. It seems to often be associated with some financial fraud, probably because those suspicious activity notifications kinda disappear when you're getting 20,000+ messages a day. I'm working on some software for this, it's in limited testing now but still really rough. Email is hard to write good software for.

I'll assume that, because rebooting fixed it, this wasn't a pure traffic problem, but just a general system overload issue. You might want to (1) collect traffic origin statistics and (2) take a good look at some more efficient mail-reception servers (I don't think your outbound SMTP server was hit by DDoS).

I'd suggest using a hosting that include DDOS protection, like OVH or Hetzner. You should then combine that with a proper email gateway, proxmox mail gateway is free and it's very easy to install.

Cloudflare recently introduced a product that seemingly can protect arbitrary TCP-based traffic (https://www.cloudflare.com/products/cloudflare-spectrum/), haven't used that myself though, it seems to not be part of the standard offering for now.

Some of the simpler DDoS attacks can be mitigated by kernel settings and iptables (see e.g. https://javapipe.com/blog/iptables-ddos-protection/), but that won't help you much against larger attacks.

DDOS comes in several flavors.

Most common one is just to dump too much traffic, so network can't handle it.

Others are less common but can cause server to become unresponsive due to exhausting some other resource than network (cpu, file descriptors, huge amount of swap, ...)

For the first type of attack you can't do much on your own you have to either work with your isp (or CDN for web traffic)

If your server is still reachable over network but unresponsive that means you are suffering from second type. That you can usually do something about.

You could put firewall with rate limiting (sometimes called traffic shaping), connections to your mail server.

Setting up firewall is not something I can guide you in a comment so google for it.

Good free one is pfsense

I have a few mail servers and I have not had your problem but this could be because I use fail2ban, and have it drop requests above a certain threshold in a given time frame.

Depending on how these ddos attacks are getting sent this might help.

Check if your box has been compromised and if it used as spam relay or amplification box.

Use iostat, netstat, lsof, top, strace. Locate problematic process, incoming and outgoing traffic, unexpected open ports, check mailboxes user and system folders....

In any case use the usual tools to narrow the problems by process, socket status, user, ips, scenarios.

If the problem fade with a reboot you can be the target or being part of a botnet.

There’s little you can do if you never had such problems in the past. DDOS is hard to battle with; you could get away by moving your box to another provider, but the IP reputation and other misconfigurations & stuff might bite you back causing a huge pita.

We rely on Mailinabox, which has fail2ban but the server being on a DigitalOcean network, they claim to offer some kind of DDOS protection.

imho.

* don't reboot a server until its really necessary - like a kernel upgrade etc.

rebooting is never a solution, this is just pointless "panic mode":

if its still possible: investigate why the system becomes slowed down ... a hardware-problem/ram or other resource-constaints or "real" dos-problems like out-of-tcp-connections etc.

* for this: use proper monitoring - mainly graphing - as simple as munin or more complex like prometheus & grafana

hey ... its just a small mailsystem you don't need any of the latest and greatest paid service for this.

* personally i operate a small mailsystem since the ancient times of the internet - aka 90ties - sendmail/qmail/qmail-ldap and atm exim-ldap. around 300 mailboxes - more or less my friends & familly.

for example: i'm using dynamic blocking similar to fail2ban for smtp-auth brute-force - implemented in bash/python; spamassassin & clamav for spam - custom config: mainly blocked most of the "crap" TLDs like .icu etc...

Can you explain what you mean by DDoS attack in the context of mail server? Are you just getting a lot of network traffic (for example, just open connections on smtp/imap ports) or are you getting all sorts of spam (like an email bomb)?

I found a ready-made image with some good insights https://gioorgi.com/tag/mailserver/ But DDoS is not easy to manage

AWS ELB has some level DoS protection out of the box (e.g. SYN flood), plus you can probably add another layer by manipulating security groups in response to traffic.

You will lose your old IP address, but the DNS should be fully functional.

I never had any problems with DDoS of our email servers. If we have one, we're gonna

1) check that our MTA is up to date 2) check that we got fail2ban rules for POP&SMTP failed logins 3) use a haproxy and ban by subnet/country

If you're getting overwelmed with bounces for mail you didn't send, setup SPF, DKIM, and DMARC so responsible mailservers can reject spoofed messages early and not bounce things as much.

Do you notice any patterns in the incoming traffic? Any pattern you can distill will lead you to your final fix. Otherwise it's just guessing. Good luck!

I'm not an expert but can't you setup network level monitoring and employ ipfiltering when threshold of network usage is crossed.

Use something like ProxMox Mail Gateway to let it take the chunk of traffic