Archive for Monday, May 3, 2010

Dead batteries cited in KU Internet outage

Dead backup batteries were the cause of an outage that left the KU campus without internet and e-mail access last week. The batteries have been replaced and the system is back up and running.

May 3, 2010


Blame dead batteries and a faulty warning system for leaving the KU campus without Internet or e-mail service for most of a day last week.

A team of experts identified the problem and made other changes designed to prevent the Internet outage — dubbed by some in the Twitterverse as “Apocalypse2010” — from happening again.

“The system is fixed,” said Jack Martin, a KU spokesman, estimating the cost of work at $13,000. “New batteries are installed. We are back up and running on regular power.”

Internet service and KU e-mail accounts had stopped Thursday, when a massive surge protector — designed to protect more than $8 million in computer servers and other equipment — had tripped.

The system had been designed to transfer power to backup batteries or a backup generator, he said, but neither contingency occurred.

“The batteries had not been charging, and as a result were dead,” Martin said. “The system had not been telling us that they were dead, like it was supposed to.

“When there was a momentary power fluctuation, there were no batteries that could keep the system going.”


gccs14r 8 years, 1 month ago

I wonder if their batteries are still in an 85° pump room, being kept just below their maximum permissible operating temperature by a giant homemade box fan like they were 15 years ago? I could see a situation where the UPS control module sees the high battery temperature and decides that they're too hot to accept a charge, so the batteries sit there and run down over time.

bearded_gnome 8 years, 1 month ago

what the story doesn't tell us: obviously there was human failure in the process! why weren't the batteries being tested/monitored under nondisaster conditions? why wasn't a human checking the charging circuitry? *why wasn't there a full-on drill/test of the equipment done at an off-time to test the entire system, say at 2am sunday morning?

Mark, your article only tells half the story!

I tol my cuzin not to be a pluggin outlt bars into other outlt bars! he don know bout dizaster recocvry and complecated stuff lik that! Marc Fagan ain gonna talk about my cuzin: he caint talkbout it.

Jimmy Joe Larry sed wen KU web interstrcture went down, there wuz bolts of lectricity six feet long!
wuz reel kewl! but now, his boses at KU gonna kik his @$$ outof work.
Donna Sue's already lookin for work, and Jimmy Joe Larry's got to mind the kids now ta hom!

mickeyrat 8 years, 1 month ago

^^^ And nothing ever goes wrong at privately-owned corporations.

gl0ck0wn3r 8 years, 1 month ago

Sure, it happens - but in a private industry company with a budget as large as KU's, someone would lose their job. KU IT has a long history of being a joke and this incident does nothing to disprove it.

Majestic42 8 years, 1 month ago

ResNet is every ISP in ten years. Every time I or someone I knew called them with a problem, this is their script. 1. Is your computer on? 2. Is it plugged into the internet? 3. Both? Well then I can't help you. Sorry.

gccs14r 8 years, 1 month ago

KU has what, a $600 million maintenance backlog because of chronic underfunding? Given a choice between building a climate-controlled facility to house the UPS and rebuilding Mallot, they'd probably much rather rebuild Mallot, but that assumes they're ever given the money to do either one.

Liberty275 8 years, 1 month ago

They can't charge a battery but they can tell us the weather forecast for the next 100 years.

Fugu 8 years, 1 month ago

Faculty and staff serve two completely different functions at a university.

Fugu 8 years, 1 month ago


I used to work there for several years. The UPS room is climate controlled and is kept between 69-71 degrees. However, I know what giant box fan you are referring too. We'd break the baby out whenever we lost a feed and an air conditioner or two went out.

gccs14r 8 years, 1 month ago

That's a plus. When I was there, the UPS control unit and all the batteries were on the far east end of the main air handler room on the north end of the computer center. On the opposite wall was the entry point for all the copper trunks for the 5ESS phone switch that was on-site. There was a stick thermometer hanging from the ceiling on a string, just about head height, and at every shift change someone had to go read the thermometer to make sure it wasn't any higher than 85. It was always 85, unless someone from F&O left the double doors open leading to the great outdoors. I hated walking past the compressor that was just inside of the security door for that room. That thing was [b][caps]LOUD[/caps][/b].

eotw33 8 years, 1 month ago

I see a duracell commercial in the making...

bearded_gnome 8 years, 1 month ago

well, if it is the maintenance backlog, funding shortage, then that's part of the story Mark didn't tell in his little story!

however, I think this might have less to do with the maintenance backlog/funding, than a failure of management to do prudent planning, and properly check whether the "backup systems" are functional.

gl0ck0wn3r 8 years, 1 month ago

Classic KU IT management - using the word management very, very loosely.

In the real world, someone would lose their job for this... but at KU it will, at most, provoke a memo.

mdrndgtl 8 years, 1 month ago

These things happen, don't let it pee in your cereal.

Commenting has been disabled for this item.