Architecture of the SCADA system and its connectivity for JEA, the municipal electric and water utility, across every water treatment and wastewater facility it operated — 24 plant sites — with separate control centers for each of the two domains and backup capability between them, so the control systems stayed up when one of them did not.
This is the work Coleman Technologies made the hire for, in October 1999, and it opens a relationship with the utility that ran twenty-five years.
Why the redundancy went in first
On the treatment side the stakes are physical rather than informational. Hydrogen accumulates, and if the ventilation fans stop and nobody knows they have stopped, the building becomes an explosion risk.
A SCADA network carrying that kind of signal is not a monitoring convenience — it is what stands between a sensor reading and somebody getting hurt, which is why the redundancy went in before anything else did.
The analysis came before the drawing
The engagement began with reliability analysis rather than with a design. I was hired first to run formal analysis of the whole system in CARE, the reliability engineering package from BQR — then to design the SCADA network against what it returned, then to implement it alongside JEA staff.
The model calculated mean time between failures and availability, and identified single points of failure device by device and path by path across both domains, from each treatment facility back to the control center that watched it. What it surfaced — the weak links, and the shared components sitting behind paths that looked redundant on paper — went into the design as requirements rather than into a report as observations. That is the whole reason the sequence was worth running in that order.
Drawn redundancy is a claim. Computed availability is a number somebody else can check, and on an estate where the signal being carried is whether the ventilation fans are still turning, the difference between the two is not academic.
Separation, before it had a name
The estate was engineered with a deliberate separation between the control network and the corporate one: distinct address ranges for the point-to-point transport, for the PLCs and SCADA servers, and for the corporate network, so a corporate machine had no path to a controller by accident.
That is Purdue-model zone separation and an IT/OT boundary. It was designed in 2000, roughly two decades before the industry settled on the vocabulary for it. The reasoning did not require the vocabulary — a machine that has no business reaching a controller should not be able to reach one, and the cleanest way to guarantee that is for the address plan to make it impossible rather than for a rule to forbid it.
One hierarchy, agreed first
The estate also got a tag and asset naming standard, authored before the systems that would have to honor it were built. One hierarchy, which supervisory control, the historian and reporting all referred back to, so a point added at one plant was intelligible everywhere else rather than needing a private map in each system that wanted to read it.
This is the same argument as the address plan above, applied to names instead of numbers. Both work by making the wrong thing impossible rather than forbidden, and both are nearly free to do first and brutal to retrofit — reconciling two naming conventions after the fact means reading every tag by hand on a live plant, with no way to prove the mapping is complete.
The vocabulary arrived two decades later: ISA-95 asset modeling, and latterly the unified namespace.
What it ran on
Control was split by domain and by location rather than pooled into one room. Wastewater ran primary from the JEA System Operations Center on the west side of Jacksonville; water treatment ran primary from Ridenour, near Beach Boulevard. The System Operations Center also stood as backup for water treatment, and the utility’s tower building stood as backup for both. Each site carried its own Siemens SCADA server.
The two were engineered as separate systems throughout rather than as one estate on one control path, and the separation was driven by personnel and by proximity to the facilities each center served rather than by a technical preference.
It ran over a SONET and ATM core:
- Lucent DDM-2000 at OC-12 and OC-3
- Cisco LightStream and AXIS shelves carrying LANE, with permanent virtual circuits engineered end to end
- Cisco 7513 and 7206 routers
- A leased 64K ISDN path held in reserve as an out-of-band backup
At plant level, Siemens S7-300 and S7-400 controllers ran tank and well pumps, with radio links reaching the sites that fiber could not.
The ISDN line is the detail worth keeping. It was slow and it was never going to carry the estate — it existed so that when the transport was gone, there was still a way in to find out why.
Outcome
Two control domains that each survived the loss of their primary site, on a network where a corporate machine had no accidental path to a controller.