IT‑IS‑Networking
Projects › Utility, then local government

Reliability Engineering Practice

A reliability toolchain bought for one utility engagement in 2000, carried into a city estate, and still running twenty-three years later — availability as a computed number rather than a drawn assertion.

The toolchain was bought in 2000 for the JEA SCADA engagement and then came along to the City of Jacksonville, where it stayed in use for eighteen years. CARE for system-level reliability and availability modeling, reliability block diagrams for series and parallel path analysis, parts-count mean time between failures prediction against published reference libraries.

At the city it was applied to model enterprise systems for robustness and recoverability — establishing where redundancy genuinely existed and where it only appeared to — and those findings fed disaster recovery and continuity design. The discipline was written into the job rather than left to preference: the Enterprise Technology Architect specification of August 2012 names uptime availability, fault tolerance and performance for mission-critical systems as a standing responsibility. The toolchain was still in use in July 2023, three months before the citywide network study that assessed manufacturers on availability and mean time between failures.

Naming the shared terms before the bad night

In November 2019 five monitored dependencies were designated backbone-affecting: carrier DNS, the financial system IPSEC tunnel, the commodity internet provider, the hosted mail service, and the state regional data center connection. An alert on any of them escalated immediately and by name rather than through the ordinary rotation.

That is single-point-of-failure enumeration applied to operations instead of to design. These are the elements whose failure propagates to the whole estate, identified in advance so that staff could tell a systemic event from a local one without spending the first two or three hours of an outage establishing which one they had. It bypassed standard escalation procedure, which drew an objection, and the justification offered was diagnostic speed under outage conditions.

The airport dispute, settled by removing the shared element

In June 2019 the regional airport authority’s network architect rejected a computer-aided dispatch and radio connectivity proposal on reliability grounds. He was right to.

Two circuits from one carrier are drawn in parallel and behave in series, because the carrier is a common term in both branches. He had the empirical basis for it too: a run of fiber cuts from roadway construction near the facility, where that carrier was the only provider with fiber into the building.

The counter-design answered the objection on its own terms rather than by buying more capacity. The primary link failed over between two geographically separate headends, so the headend stopped being a shared element. The secondary was deliberately not a second circuit of the same type from the same carrier — it used the municipal fiber path with independent physical routing, which removed the carrier as a common term. And the secondary was argued as preferable because it terminated at a single site, on the grounds that a simpler path with genuine independence beats an elaborate one carrying a hidden dependency.

That is reliability block diagram reasoning conducted inside a procurement argument, which is where it is least expected and most useful.

Outcome

Availability treated as something to compute before building rather than to report after failing.