Floor plan and rack elevations, hot and cold aisle containment, dual UPS with battery plant and maintenance bypass, independent A and B electrical paths from separate panels and transformers, and overhead busway power distribution.
The network path used diverse carrier circuits and aggregated links into redundant next-generation firewall pairs running in high availability with hardware bypass, so that a failure of the inspection layer degraded to open rather than to down. Security controls that take the network with them when they fail do not survive contact with an operations team.
Data center infrastructure management was deployed across the estate: the full power chain, UPS output across both feeds and generator power, underfloor water detection, humidity, and instrumented hot and cold aisle temperatures to confirm the containment was performing the way it was drawn.
Diagnosed by substitution
The longest incident thread in the record ran from October 2020 into January 2021 across this site: the Sheriff’s Office workstation image, a Fire and Rescue dispatch application, a state law enforcement query dependency and general site slowness, all live at once and none of them cleanly separable. Four parallel workstreams, four theories.
A dispatch machine from a different building was swapped in. The symptom resolved immediately, which moved the fault off the network and onto network interface drivers and firmware differences between builds. The substituted machine was then left in service a further twenty-four hours before the finding was accepted.
When four theories are live, the cheapest instrument is usually a known-good unit rather than another capture.
Outcome
A recovery site whose availability did not rest on a single power path nobody had drawn.