KU still tallying total cost of massive internet fiber cut, weighing options for preventing a repeat outage

This aerial photo shows Kansas University's Daisy Hill and the ongoing construction work on KU's 50 million Central District redevelopment project as of April 13, 2016.

Kansas University has yet to tally the total cost of an internet fiber cable cut that crippled campus earlier this spring, but some sources indicate it may be in the millions of dollars.

Meanwhile KU Information Technology is looking at how to prevent such an outage from occurring in the future, and University Senate representatives say funds must be secured to build a backup system.

“This simply cannot happen again,” the University Senate’s Academic Computing and Electronic Communications Committee wrote in its end-of-the-year report. “Specifically, a single communications cut cannot bring down the entire Lawrence campus and parts of Edwards campus.”

Early in the afternoon of March 29, construction crews inadvertently cut through a critical section of fiber between the Price Computing Center and the Ellsworth Data Center on Daisy Hill.

Suddenly, KU found itself in a major lurch.

The cut shut down internet access all over KU’s main Lawrence campus as well as wireless internet at the Edwards Campus in Overland Park. Buildings and departments that have converted landline phones to internet-based Skype for Business lost phone service (as did several buildings using landlines, as some of those lines were cut, too).

Because connectivity to university servers was out, KU websites and applications such as Blackboard also went down, as did state testing for thousands of K-12 students across more than a dozen states that rely on the KU-based Center for Educational Testing and Evaluation.

The majority of internet service was restored by 10 p.m. the night of the fiber cut, which KU officials tout as a mammoth achievement given the severity of the fiber cut.

Internet access in several buildings — including the engineering complex — took two to three more days to fix.

Restoration took a lot of manpower on the part of IT.

“Several of our staff worked around the clock to restore services; some didn’t even go home to see their family or shower those first couple of days,” KU Chief Information Officer Bob Lim wrote in a May memo to campus.

What happened?

Lim called the fiber cut “unfortunate” and said it could have been avoided.

“It was the result of miscommunication among external contractors, even though for some time we had consistently communicated the existence and location of buried cables,” Lim wrote. “As a result of the incident, several groups across campus and the contractor have already implemented additional procedures to prevent a similar incident in the future.”

KU officials are not publicly naming the company whose workers cut the fiber.

They said KU, however, will not be footing the repair bill, whatever it ends up totaling.

“The total cost will represent a combination of materials, staff time and other expenditures that are still being calculated,” university spokeswoman Erinn Barcomb-Peterson said. “It remains to be determined who will pay for the repairs, though we know it will not be KU.”

KU IT was estimating a “seven-digit” cost to restore the fiber alone, according to the University Senate committee report, based on information the committee received from Eric Freeze, KU’s deputy technology officer.

Freeze told the committee that 3,000 strands of fiber and 2,000 copper pairs were cut that required patching, according to the report.

Barcomb-Peterson said that while the restored link is now 100 percent functional, KU will ask to have it replaced with a smooth run without splices, as it was before the cut.

Backup plans

While university officials will continue to deal with costs and likely insurance companies about who will pay for the fiber cut, KU IT is now focused on prevention, said David Day, director of external affairs for KU IT.

Because of cost, he said, that involves balancing the need for backup systems with investing in new or enhanced IT services.

“Building in redundancy has a cost, and you have to calculate that against the possibility of something remote happening,” Day said. “That’s what has always been weighed in the past and is being re-looked at now.”

Day said it appears likely that KU IT will invest in redundancy — or a backup system — of the Price to Ellsworth fiber line.

“It’s such a critical pathway,” he said. “It’s been proven now that it can be compromised, and unfortunately that opens your eyes.”

Campus’s increased reliance on the internet is likely to factor in.

“Even 10 years ago losing internet connectivity wouldn’t have been so critical,” Day said. “The need for redundancy and maintaining that connectivity has grown exponentially, as it has become such an important part of our lives.”

In one example, while only a few buildings and departments are using internet-based phones now, KU plans to eliminate nearly all campus landlines and have the entire university making calls via Skype for Business within two years.

In the classroom, many KU faculty rely on the internet to teach, said University Senate President Joe Harrington, a professor of English.

Harrington noted that on the day of the outage he ran into a colleague whose entire class plan for the day was based on material from websites, sending her scrambling to come up with an alternative lesson.

“Most everything we do relies on internet connectivity, particularly in the sciences and engineering, but even in the humanities,” Harrington said. “So having a fiber cut like that really, in effect, shuts down the entire university… We take it for granted, but when we don’t have it, it really does put us out of commission.”