It started with a message from my developer. "Site's down, restarting it now."
Okay, fine. These things happen. I didn't think much of it.
Then it happened again the next day. Then twice in one day. Then I started getting alerts at 6 in the morning telling me the site was down, only for it to come back up on its own ten minutes later before I'd even had my coffee.
I'm not a coder. I've said this before on this blog and I'll keep saying it because it's the whole point of this space — I don't read server logs for fun, I don't know what a CPU spike actually means in practice, and until a few weeks ago I genuinely did not know the difference between my website "being down" and my server "being down." Turns out those are not the same thing. Turns out a lot of things are not what I thought they were.
So here's roughly how it went.
First theory: too many people on the site at once. We run New Projects Tracker, and while we don't get massive traffic, maybe something had changed. Maybe more people were logging in than usual. Seemed reasonable. We checked. Nope. Same usual handful of users, nothing unusual there.
Second theory: maybe it's the public pages. We have some pages open to anyone, no login needed, mostly for search engines to find us. Maybe someone was hammering those pages. This felt closer to right, so we went down that road for a bit — looked at removing some filters, capping how many pages someone could click through. Did that. Felt good about it. Site stayed up for about a day.
Then it went down again.
Third theory: bots. This one turned out to be partially true, and I'll get into exactly what we found in the next part, because it's honestly a little wild — there were specific AI crawlers hitting one part of our site over a hundred thousand times in a single day. A hundred thousand. From one source. That is a real thing that was actually happening, not an exaggeration for the blog post.
We fixed that too. Blocked what needed blocking. Felt very good about that one. Surely that was it.
It was not it.
The site went down again a few days later. Same pattern — CPU pegged at basically 100%, no memory left to spare, server unresponsive, automatic reboot. Like nothing we'd done had mattered at all.
This is the part that was genuinely confusing as a non-techie. Every single thing we fixed was a real problem. We weren't imagining these. The bots were real. The exposed pages were real. We weren't chasing ghosts — we were finding actual issues, fixing them, and the server kept crashing anyway. That's a strange feeling. You start to wonder if there's something fundamentally wrong that you're just not equipped to see.
There was. I just hadn't found it yet.
What I didn't know yet — and what I'll get into in the next part — is that while we were busy fixing the things we could see, there was something else going on entirely underneath all of it. Something that had nothing to do with bots or traffic or any of the things you'd normally suspect first.
This turned into a four-part thing, because that's genuinely how long it took me to get to the bottom of it.
- Part 1 (this one) — the chaos. Things going down, no pattern, no clue.
- Part 2 — what I actually found when I started digging properly. Spoiler: it wasn't one problem, it was several, layered on top of each other.
- Part 3 — the thing that turned out to be the real cause. I would not have guessed this one in a hundred years.
- Part 4 — what actually fixed it, how long it's stayed fixed, and what I'd tell anyone running their own website to go check right now, today.
If you run your own website and it has ever gone down for no reason you could explain — stick around. This might save you some weeks of confusion.
Next part coming soon.