Server outage affecting all services
Resolved
Nov 7, 2026 at 7:41pm UTC
Resolved. All services on this page were slow or unavailable on 7 November, from about 18:22 to 19:38 UTC. The evidence points to a fault on the physical Oracle Cloud host running our server, and moving to different hardware fixed it. The server's disk stayed attached, so nothing had to be restored from a backup.
What happened and what we did (all times UTC):
- 17:55: Server CPU jumped to 100% and stayed there.
- 18:22: Our monitors began reporting services as unreachable.
- 18:20 to 18:50: Restarted the server several times, including a full stop and start. This didn't help, and each boot was extremely slow.
- 18:57 to 19:20: Investigated. Boots were slow before any of our services had started, and starting even tiny programs took around 100 times longer than normal. No recent software updates lined up with the problem, which pointed to the host rather than our software.
- 19:24: Moved the server to different hardware in another fault domain.
- 19:32: Server back online and running at normal speed.
- 19:38: All monitors recovered.
We're now reviewing backups, resource limits and alerting so we can catch and recover from this faster in future.
Affected services