Archive for Thursday, April 29, 2010

Machine failure blamed for knocking out KU website and e-mail; most service expected back tonight

A blown surge protector meant server outages for the whole day, frustrating students and professors alike.

April 29, 2010, 11:26 a.m. Updated April 29, 2010, 9:49 p.m.


Access to most Internet services hosted by Kansas University was restored Thursday evening after being down for much of the day.

An electrical problem at KU’s Computer Center had limited access to e-mail and websites hosted by KU.

KU’s website had been down most of the late morning and afternoon because of the issue, but was functioning again shortly after 5 p.m.

Blackboard, an online course management system that provides documents, assignments and other communication between teachers and students, was still down late Thursday, but KU officials hoped to have that system back up and running by today.

Jack Martin, a KU spokesman, said that a machine that served as a sort of surge protector and for battery backup for KU’s data center had failed.

Because that piece of equipment served as the link between KU’s data center and the electrical grid, equipment that provided e-mail and connective services to KU lost power shortly after the initial equipment failure, Martin said.

Resolving the problem was a slow process, Martin said, as turning the system back on is not as simple as just flipping a switch. Services were gradually restored throughout the day Thursday.

The university first reported the outage about 11 a.m.

An afternoon without Internet and e-mail service was causing problems for a community used to being constantly plugged in.

“Panic” was the word Jessica Nadeau, a Topeka senior, used to describe it.

Nadeau was sitting in front of a black computer screen in the Budig Hall computer lab about an hour and a half before a 2:30 p.m. exam. She was taking an old-fashioned approach, looking at a textbook and handwritten notes scribbled in a spiral notebook.

“I’m trying to study for a wicked-hard test, and my study guide is online, all nice and filled out, so I’m just trying to go through my notes,” she said. “Not a good day.”

Martin said the provost’s office would be asking faculty members to be flexible with students affected by the outage and to make accommodations when appropriate.

For KU employees, too, not being able to connect in the usual way forced people to improvise.

Joe Harrington, associate professor of English and the department’s director of graduate studies, had Microsoft Word open at his desk.

He was typing in e-mails in Word documents that he would be sending to a number of recipients later.

“It makes you wonder what things would be like if it was out for a long time,” Harrington said. “We’ve become so dependent on it. I mean, it’s really amazing how many things I can’t do because I can’t get on e-mail, I can’t get on Blackboard and I can’t get on all the other things that I use for research and teaching.”

The KU Bookstores announced they would be closing because of the network outage shortly before 1 p.m. A sidewalk sale scheduled for Thursday was pushed back to today in the KU Bookstore at the Kansas Union, from 8:30 a.m. to 4 p.m.

KU had about half its normal Internet traffic flowing in and out in the early afternoon hours, said Cort Buffington, executive director of KanREN Inc., which provides Internet service to the university. He said he had offered to help with the issue in the morning when it arose, but mainly tried to stay out of the way of those who could fix the problem.


boothillbilly 7 years ago

No, it means you can't access the library catalog, databases, email, etc. Grow up g_rock.

somedude20 7 years ago

This comment was removed by the site staff for violation of the usage agreement.

MyName 7 years ago

Actually, the internet connection is fine, it's just affecting the KU web/email/database servers.

somedude20 7 years ago

thank god for that as there would have been a lot of idle hands. let the Billy Squier(ing) begin

RogueThrill 7 years ago

It's the DNS servers. If you already have all of your frequently visited site DNS translations cached from first thing in the morning browsing you'll be fine. Don't reboot though.

average 7 years ago

If you're techy enough (and not locked down from doing so), you could temporarily set your DNS to something besides the traditional two at the computer center. and are Google-sponsored public DNS servers (and easy to remember at that).

Although most places at KU that auto-configure (DHCP) have some off-site DNS server in third place, so, while lookups are slow, they work.

Liberty275 7 years ago

All good nerds have their own caching DNS server.

terrapin2 7 years ago

Oh really? And who are you who are so wise in the ways of science?

monkeyspunk 7 years ago

It's not just the DNS servers, though the lack of DNS would explain the rank and file user's inability to access the internet.

If they have lost all of their central systems, then it can be only a couple of things. First, they could have a power outage, but in that case connectivity would be completely nonexistent. Second they could have a physical issue, such as fire or cut wiring. Again not likely as connectivity still exists. What is most likely the issue is that they are having an issue with their virtual infrastructure and things are not coming back up as they had hoped. If they have virtualized their web, email and database servers and they lose their storage system or the file system on the storage device, they can be brought to their knees.

average 7 years ago

The pager alert said power and I'd believe it. A redundant network is fairly do-able. If you've got even two fibers coming to a campus, I'd send them to two different buildings. Backhoes happen. The building switches are likely interconnected in some sort of ring to get to the other fiber. Just that central services (including DNS) are centralized.

kuoirad 7 years ago

KU's network infrastructure is not at the Computer Center...

kusp8 7 years ago

KU's free anti-virus', Sophos, updating servers are also down. So...basically as long as you've updated within the past month you should be fine. If not then you'll need to quit surfing so much porn. haha.

monkeyspunk 7 years ago

But but but...

The internet is for porn!

Shardwurm 7 years ago

How will the professors conducting 'research' check their stocks?!??!!?

classclown 7 years ago

I thought it was pretty quiet around here. Now we know why.

MyName 7 years ago

Maybe the people know WTF they're doing, but just don't have the parts.

Kind of like how your mechanic can fix your car, but you still gotta wait a week for him to get ahold of the part to that Chevy Nova you're still driving.

tnfats 7 years ago

Use Sunflower? That's like saying "Dang, the batteries in my remote have died, I knew I should have gotten DirectTV"

MyName 7 years ago

Did you even read the part in the thread where it was mentioned that the network still works, it's just the building with the servers that is having problems?

Maybe I should translate this technical stuff for you: the tubey things are fine. People at KU can still send stuff up and down the intertubes. The problem is that the place that stores the KU stuff is broken so KU can't send its stuff through the intertubes...

Tom Miller 7 years ago

EXTREMELY well described, and in exactly correct techie language! 2FF!!!

jhawks1234 7 years ago

How is Anna going to be able to post her latest blog entry with no internet on campus ;)

Ricky_Vaughn 7 years ago

She'll have to call daddy and have him call their family IT guy!

terrapin2 7 years ago

As Dan Akroyd's character said in the 1985 movie Spies Like Us, "We mock we do not understand". It is rather hilarious to see a bunch of you spewing half-baked theories about the technology issues impacting KU at the moment.

Some employees directly affected by the outage were sent home but were told they might have to use vacation time to cover the afternoon if administrative leave isn't approved.

MyName 7 years ago

Well yeah, because clearly the people who work at KU live in a magical place where they don't have to pay taxes. Or work.

And never mind the fact that the state's contribution to KU's budget is hovering around 20%. It's still going to make a "huge" difference.

But hey, wishing ill on others for no good reason is obviously good for your karma. So "Karma lives" is an awesome slogan for you.

whats_going_on 7 years ago

this is serious business. Don't take it too lightly, peoples.

monkeyspunk 7 years ago

I hope you are being sarcastic, because it really isn't serious business. They host very little in the way of what could be considered critical data or services. If they did, they would have a Disaster Recovery system in place for those services.

People are inconvenienced and that is about it.

On the bright side, there might be a job opening at KU that I would be qualified for in the upcoming days. =D

lawrencian 7 years ago

wow, people got to go home because of this? I wish that was true in MY office!

afraidnot 7 years ago

Any more information LJWorld? Like when is it going to be back up?

Fugu 7 years ago

I heard it was a power issue.

monkeyspunk 7 years ago

Again, doubful that it is a power issue. They most likely have redundant or generator power. Also, power would knock out infrastructure devices switches and routers and firewalls. Connectivity would be down, and people are reporting that they can get out as long as they have DNS.

Fugu 7 years ago

They do have redundant power, however last time I checked, they were doing a major revamp of the power infrastructure, which included a brand new generator. They were having issues getting it fully operational. Also, much of the network infrastructure is housed outside of the Computer Center. I'm no expert though..

monkeyspunk 7 years ago

Ah, then that makes a ton of sense if they are working on it. Hopefully no damage was done.

kuoirad 7 years ago

The network infrastructure for campus is not at the Computing Center.

birdie138 7 years ago

Except that a lot of it (DNS, Firewall, content servers) is.

kuoirad 7 years ago

While the DNS servers may be/are at the Computer Center, they're not typically thought of as "network infrastructure". That would be the main campus router(s) and the fibers that constitute KU's connection to the Internet (commodity and I2 both, plus the tie-in to Sunflower, I believe). Those are up on Daisy Hill with NTS.

I'm not quite sure there's a campus firewall. If there was, that'd probably be up at NTS's Daisy Hill location as well.

OldEnuf2BYurDad 7 years ago

It can't be too serious. I just saw my wife on Facebook, and that's all that matters.

RoeDapple 7 years ago

The internet is down? We're all gonna DIE!!

Ricky_Vaughn 7 years ago

Anyone remember that South Park episode when the internet goes out and they travel to California looking for internet...?

jhawks1234 7 years ago

We're headin up Californiway...We's heard ders some internet thataway

Joseph Jarvis 7 years ago

So if it's a power issue, and they have backup generators, why didn't systems fail over to them?

Joseph Jarvis 7 years ago

@Andy Hyland: Has LJW contacted anyone at the computer center or sent a reporter there? Just curious.

valgrlku 7 years ago

The text message we got from KU at 3:10 reads: "Backup power being used to restore services. Most services online by 9 p.m."

I still don't understand how an electrical issue caused this problem, especially if there is/was backup power available.

ahyland 7 years ago

All —

Another update will be posted shortly, as we've been working to get at these problems. Apparently the problem was with the "uninterrupted power supply" or UPS, which serves as a sort of surge protector to the data center. KU is restoring power as it can, operating on generator power. It's a complicated issue, and, as we speak, I understand technicians are not only working to resolve the problem, but in so doing, also have to run diagnostics to find out just what happened in the first place.

Check back soon (maybe even by the time I finish typing this). We're obviously doing what we can to get information as it becomes available.

Andy Hyland KU reporter

true_patriot 7 years ago

Would be interesting to know the primary cause rather than proximate cause - what was causing the kind of spikes on the grid that could cause the UPS to fail? Does Westar have any ideas on that? Did this very windy day have some role?

gccs14r 7 years ago

I was there long, long ago when the main UPS firmware thought someone pressed the EPO button on the front cabinet and shut down the entire protected load. We didn't even start bringing things back up for almost nine hours, and that was after flying a rep from the UPS company down from Minneapolis so he could look at the system to make sure it'd be OK to restart it. We found out about a month later that it was a firmware bug, but that didn't keep management from grilling all of us who were on duty as to who was doing what where when the power went out. They never apologized for being accusatory, either.

true_patriot 7 years ago

Yeah, i'd be real concerned about turning stuff back on until i knew for sure the power wasn't dirty. Scary stuff.

Joseph Jarvis 7 years ago

So you're saying this is the second time the UPS has been a single point of failure?

gccs14r 7 years ago

That we know of. Twice in fifteen years is pretty good uptime, except for the outages being in double-digit hours.

lurn 7 years ago

I like turtles with internet

Boston_Corbett 7 years ago

Am I the only one struck with the irony that a device/box/unit which is designed to provide uninterpretable power/continuity is being blamed for actually creating a 12-hour interruption when there doesn't appear to be a campus power problem.

I'm having a hard time understanding how that can be.

I don't care if the issue is "complex." I hope the LJW pushes for and prints a more detailed explanation.

Phillbert 7 years ago

Ironic? Yes. Somehow nefarious? No.

Things break.

Betty Bartholomew 7 years ago

It was aliens.

Seriously, though, as somebody who works on campus, this has caused a severe problem in my office - almost all of the resources we access are on network drives and intranet servers that can't be accessed. I know other offices have been the same way. And students who have been trying to enroll, clear holds, or turn in homework today haven't been able to do so (though I understand many professors granted extensions.)

We're all looking forward to our IT people getting us back up and running.

walleye9898 7 years ago

So they have a "SUPS" (sometimes uniterruptable power supply). Must have been State contract or low-bid!

The UPS serves as a buffer/surge and short-term power provider between the power (grid or generator) and the equipment. If the buffer doesn't work or fails (A UPS has switches, circuits and wires in it which can fail) then the power doesn't get to the equipment.

Go home and look at all your 6 plug-in outlets. If the main connection and or surge/breaker in it fails, then your 6 devices plugged into it don't work.

A comp center is a much bigger, uglier, and extremely more complex system than a 6 prong outlet. You can't just qucikly re-route or re-connect. Also, every system undergoing a hard shut down has to go through system diagnostic or review before it can be released.

If you want to be live 24/7 then you have to have redundant systems. The issue is money!!!!

Technology is great, when it works!

Use the down time to get out from behind the computer and revitalize actual communication, read a book or go outside and enjoy the weather!

For all you KU Comp Center folks busting your hump to get systems back on-line --- THANKS!

Bob_Keeshan 7 years ago

Boston_Corbett (anonymous) says… Am I the only one struck with the irony that a device/box/unit which is designed to provide uninterpretable power/continuity is being blamed for actually creating a 12-hour interruption when there doesn't appear to be a campus power problem.

Are you questioning whether the box provides "uninterpretable" power? If so, I can tell you that it does.

Not even C3PO can talk to one of those things.

OldEnuf2BYurDad 7 years ago

The uninterruptible power supply has interrupted their power.

true_patriot 7 years ago

Looks like their website is back online now - they must have made good progress.

JoRight 7 years ago

This is essentially what happens when you underfund an IT unit.

This is also true for not only KU, but for a lot of businesses / corporations out there.

I can guarantee this will raise a lot of questions / cries for help that the IT there has brought onto deaf ears. If any good were to come out of this situation, I'd hope the powers that be would be there to answer these questions & resolve these issues.

Anyone that has ever set foot in any IT department knows this all too well & can only sympathize for the people involved in a situation like this. You can't expect a battleship when you buy a crew & boat that of a Somali pirate boat.

gutenberg 7 years ago

Remember, there are only two types of computer users: those who have already lost data and those who are going to lose data sometime in the future. Please, backup your backups. And remember, printed type and graphics on paper are durable forms of data backup, too. -Johannes Gutenberg, txtng since 1439

persevering_gal 7 years ago

Blackboard still isn't working. I guess I won't be doing that quiz tonight.

kusp8 7 years ago

I have pretty good information that all services won't actually be up until after 9. I hope I'm wrong, but I don't think I am.

apenw322 7 years ago

Any guesses on when Blackboard might be up? I've got a test at 10am tomorrow, and about 80% of my study materials are on Blackboard...guess I'll just be waiting up until it's back online.

kusp8 7 years ago

This evening sometime is my guess, or at the latest tomorrow morning before 8AM.

Joseph Jarvis 7 years ago

I hear you. I have a term paper due tomorrow. The electronic database on I need requires KU user authentication, which is still down. 9:00 has come and gone... sigh

twoleftfeet 7 years ago

My husband has been helping to get this problem fixed for most of the day....It's 9:30 and I haven't heard anything from him in several hours. Which makes me worry it's still gonna be quite a while. Here's hoping I'm wrong.

twoleftfeet 7 years ago

Never mind, he just walked in the door. He says he's pretty sure Blackboard is back up, but the Kyou portal still isn't working. There are still 20-30 people there working on it and he says they don't know when it will all be fixed. That's all I know. What a mess!

apenw322 7 years ago

I'm still getting an error message from Blackboard. It says: Blackboard is unavailable on Thursday, April 29, 2010. I'll keep checking, though! Thanks for the update!

LogicMan 7 years ago

"It was aliens."

It was not, and I resent the nonnative-species-ist accusation.

devobrun 7 years ago

twoleftfeet: Did you smell his breath?

9070811 7 years ago

I still can't access Blackboard and I might have a heart attack.

RoeDapple 7 years ago

Zombie Apocalypse has begun! Run for your lives!!

Ms_P_Lane 7 years ago

Blackboard is still not working. Three term papers due next week, people. Come on!!

JonKU 7 years ago

Maybe it's just me, but your lead sounds awkward.

"An electrical problem at Kansas University’s Computer Center is limited access to e-mail and websites hosted by KU."

I would suggest 'has limited.'

Cody Ochs 7 years ago

Walleye9898 appears to know knows his/her stuff. I cannot imagine how long a file system check would take on just one of the arrays KU uses, let alone all of them. I have been in situations like this, and it is not fun at all. Kudos to IT for getting systems back up so quickly.

gccs14r 7 years ago

To the students,

This is just one of many reasons why you don't want to wait until the last minute to write a paper or study for an exam.

bearded_gnome 7 years ago

Dernit! I bin tellin' cuzin Jimmy jo larry [who werks up at Knsas univrsity comptr center] not to go pluggging in outlet bars into othr outlet bars!
now, he's reely gon and dun it! I herd there wuz bolts of lectricity six feet long wen his crosswiring shorted.
I supose this meens cuzin is gonna lose his job in charge of KU's web interstructre, and than how's he gonna be supportin' the wife donna sue and the littlins?
I done tol him ther's a bettr way to fix them lectrons and seecure the lectricity for the puter center. but did hE listen? naw!

stewpid cuzin.

bearded_gnome 7 years ago

The KU Bookstores announced they would be closing because of the network outage shortly before 1 p.m. A sidewalk sale scheduled for Thursday was pushed back to today in the KU Bookstore at the Kansas Union, from 8:30 a.m. to 4 p.m.

---oh man! and I really need some sidewalk, too.

thanks KUSP8. if you see my cuzin, jes send him hom now!

gphawk89 7 years ago

It's a bit worrisome when something as simple as a sidewalk sale gets postponed because of problems with a network. Have we become so dependent on technology that we can't even keep track of inventory on paper for one day and maybe do some simple arithmetic?

"An afternoon without Internet and e-mail service was causing problems for a community used to being constantly plugged in." Welcome to the real world of IT, folks. After a few significant losses of data - or not being able to study for an exam because your study materials are online - you'll start appreciating local copies, backups, and even hard copies. I remember a long while back needing to reload Windows95 on my machine at work. The IT guy said something about not needing to backup my data because he made regular tape backups over the network and they were very reliable. I wiped the hard drive, reinstalled Windows, and tried to restore my data from the tapes. Nothing. All of the tapes were unreadable. Five years of data lost.

Commenting has been disabled for this item.