Local

‘Dark fiber’: How engineers say a fragile system brought Florida 911 centers to their knees

LAKE COUNTY, Fla. — One CEO called it the fiber industry’s “dirty secret.” At a different company, an engineer described used words like “outdated” and “fragile.”

Time and time again Wednesday, people who design, build and operate the infrastructure that 911 centers rely on to respond to emergencies and save lives described the state of America’s networks as a looming disaster.

“It’s amazing people can make phone calls,” the CEO said, speaking under the condition of anonymity. “It’s amazing we don’t have more catastrophes than we do.”

The scrutiny over lines buried along the sides of Central Florida’s roads began Tuesday, after a construction crew accidentally severed a fiber line and knocked six counties’ 911 systems partially or fully offline.

Residents of Marion County were asked to text instead of call. Lake County told its people they might have difficulty getting through. Osceola County’s emergency management director, reached before it was known the issue affected the county, quickly excused himself from the call.

Connectivity was restored to all counties overnight. As operations resumed their normal ebb and flow and county staff patted themselves on the back for handling the chaos, questions began to bubble up.

“Was there a plan?” Lake County Commissioner Tim Morris said. “If the system did go down, what was the plan? If there was a plan, why didn’t it work?”

Fragile fiber

The line at the center of the maelstrom ran along the west side of Highway 27 and was marked, ironically, by a sign warning workers against digging in the area just feet from where it was severed. It belonged to Lumen, formerly known as CenturyLink, one of the Orlando area’s biggest cable and fiber providers.

Lumen’s maps show the line is just one of two running north-south in rural Central Florida, connecting Ocala to the Orlando metro and Osceola County.

Engineers called attention to the lack of redundancy, calling it a bottleneck that the CEO described as “dark fiber.”

“Every company leases fiber from each other,” the CEO said. “It’s all interconnected. Too much traffic, too little lines.”

He called it a calculated risk by the companies. The cost to lay new fiber lines, he said, was $1.5 million per mile, which would price an Ocala-Kissimmee cable at an eye-popping $130 million.

When a cut does happen, he said the systems are designed to auto-route to alternative channels to avoid disruptions. However, he explained that it doesn’t always happen and switches are sometimes done manually.

“It’s a very fragile network,” he summarized.

Until recently, that fragility has been paired with outdated technology, according to a 911 engineer who works for a county outside of Central Florida.

He explained traditional 911 systems used a central building that would receive calls for an entire region. Those calls would be routed to each county’s 911 center through the underground cables, but in rural areas it might only be a single line serving multiple counties.

That’s why this engineer moved his county to NextGen 911, a modern, internet-based network that is prized for its redundancy, using different networks and data centers across the country to route calls in order to create a “spider web” that’s too complex to be broken.

“There’s no single point of failure,” the engineer said. “If something does happen, it takes nanoseconds to switch.”

Sumter County’s 911 center did not go offline, even though it’s sandwiched in between counties that experienced issues. Sumter was the first county to get NextGen 911 in Central Florida.

Assistant County Administrator Stephen Kennedy said when his county moved to NextGen, they required Lumen to install two independent lines to their critical buildings for an added cost of $75,000.

Lake and Osceola Counties also use NextGen 911, and it’s not clear why their facilities ran into problems by the cable cut. Lake County’s calls, however, were immediately routed to Sumter, which had radios its telecommunicators used to speak to the dispatchers in Lake with details of incoming calls. They also had people text 9-1-1 and received 64 texts compared to their normal one or two.

“Our plan worked flawlessly and we’re so proud of it,” Lake County Public Safety Director David Kilbury said. “Kudos to Sumter County that the surge of calls that came into their center to stay operational… is a testament of the professionals that are behind the microphones.”

He added that he was sure conversations would be had about how to prevent another network failure from happening in the future, though he said that was beyond his position.

Lumen leaders declined to go into detail about their network infrastructure when reached Wednesday.

“While the 911 centers and core 911 network remained operational—and are designed with redundancy—some voice calls were unable to connect to the 911 network,” Mark Molzen wrote in a statement. “Our teams responded immediately and worked around the clock to restore service as quickly as possible. We take service reliability and public safety seriously and are thoroughly reviewing this incident to identify opportunities to strengthen our network resilience and minimize the impact of third-party disruptions on essential services like voice and Internet.”

Funding imbalance

More counties would’ve switched to NextGen 911 sooner – and possibly with more enhancements – if they had the funding, the engineer said.

Florida taxes wireless customers 40 cents to pay for 911 systems, which the engineer said was the second lowest rate in the nation and a rate that hadn’t been changed in decades.

While the engineer said his county had the funds and he had the technical expertise to help build a more robust system, others did not.

Grants used to help small counties pay for the upgrades, he said, but those have since run out.

“Everyone is doing the best they can,” the engineer said. “I wouldn’t be able to fault the 911 coordinators in those counties.”

He then excused his language.

“I feel like writing a story on this would get (stuff) done.”

Click here to download our free news, weather and smart TV apps. And click here to stream Channel 9 Eyewitness News live.

0