Twenty minutes of HTTP 500, and the alert that never came
The Redis instance holding user sessions ran out of memory and every page started returning a 500. The part worth writing down is why nothing paged anyone.
01The problem
Every page returned a 500, login included, so nothing about the system was usable for twenty minutes. The Redis instance holding user sessions had run out of memory: its snapshot-to-disk job forked the process every five minutes, and a fork briefly doubles what the process needs.
02What I did
I capped Redis memory with an eviction policy, dropped the persistence mechanisms it did not need in that role, and gave it a restart policy so the same failure would clear itself next time. Then I went looking for the second problem, which was that no alert had fired at any point.
03Where it landed
It has not happened again. The reason nothing fired is that the monitoring stack was running on the same host it was monitoring, so it died with everything else. Where the monitoring runs is now the first thing I check on any system I take on.
04Evidence
Timestamped system logs across the incident window.
Ask for it on a call and I will open the dashboard.