Protecting your Sites/Services from Unwanted Traffic?

Question

So... If PoW (proof-of-work) schemes like Anubis ultimately don't' work in practise, because Bots/Crawlers are increasingly using headless browsers, are able to solve captures, proofs, etc;What options do we have realistically to filter out or block unwanted traffic?What clever schemes can we come up that don't rely on centralised serices like Cloudflare?

wanderingpixel · Accepted Answer

It depends on what you define as unwanted traffic.
Personally I do not have an issue with bots as long as they behave and are not straight up malicious, so I rely on a combination of rate limiting, a fine-tuned OWASP CRS ruleset and an aggressive Fail2ban enforcement (hit 2 triggers and you get a 24 hour ban, 2 bans and you get banned for 30 days).
My sites also make extensive use of static elements and caching.

mstaoru · Answer

Browsing is absolutely intolerable these days, with almost every website having a block screen, "we're checking your browser...", stuff like that. Lately I just close these websites immediately.