By the time we looked at it properly, every one of our servers was running ThreatCrush, the open-source security daemon we build. It caught attacks and banned them. But each box was its own island. Our dashboard listed detections, yet nothing showed the whole fleet at a glance, so to see what a server saw you SSH'd in and read its logs. Mostly nobody did.
Then we upgraded the fleet by hand this week and found what that had cost us:
One box was several releases behind. It had missed the event pruning we shipped since, its state database had grown to 138 GB, its log to 60 GB, and the daemon could no longer start.
That same box had been trying to ban Cloudflare's edge IPs, because its nginx logged the CDN's address instead of the visitor's. The bans only failed because of an unrelated permissions bug.
An older release once banned every machine that logged in over SSH, including our own. We found out when our servers dropped off our own network one at a time.
What fixed it wasn't a new tool. It was three boring things:
One view. Every detection from every box, placed on one live world map: in the web dashboard, and in the terminal with
threatcrush map. If a box goes quiet or starts banning something odd, you see it.Updates nobody has to remember. The daemon now updates itself. It waits until a release has been public for two hours, adds random delay so the fleet doesn't restart all at once, and never jumps a major version on its own.
A cap on anything that grows. Logs rotate daily or at 200 MB, the event store prunes itself, and the installer sets up rotation even if you never install the service.
Results so far:
In the first hour the map was live, our three public servers took 108 attacks from 92 different sources, mostly web scanners. The US was the biggest source by a wide margin.
One server's terminal map showed 88 sources and 5,095 events in a few minutes.
Upgrading the fleet took one manual pass. After that, a new release reaches every box about two hours after it's published.
Cleaning up the broken box got 197 GB of disk back.
If you run more than a couple of servers, set up three things: one place that shows every box, updates that happen without you, and a limit on everything that grows.
Disclosure: I build ThreatCrush. The CLI is MIT licensed and installs with
curl -fsSL https://threatcrush.com/install.sh | sh. The map on our homepage is live attacks on our own servers, with screenshots of the terminal version here: https://threatcrush.com/blog/live-attack-maphttps://redd.it/1x80hht
@r_SelfHosted
