Rubenerd

The entire Sydney Trains system went offline last Wednesday, stranding thousands of commuters (like me!) just before the afternoon peak.

Richard Chirgwin reported for iTnews:

The outage halted all trains at stations, because the communications network is critical to communication between drivers, guards, and the rail network’s management centre.

The head of Sydney Trains fronted a press conference to explain:

Transport for NSW believes a failed network switch caused yesterday’s hour-long communications outage, compounded by the system’s failure to automatically switch to a backup network.

“The system has the redundancy to automatically switch across to a backup. That should have occurred immediately … [but] didn’t occur."

Mistakes happen, and automated failovers often aren’t. Disaster recovery strategies are like backups: only tested ones can be considered functional.

Here’s hoping they switch (HAH!) to a better process.