Archive for Tuesday, August 24, 2010

KDHE fetches documents from storage to help restore records lost in computer failure

August 24, 2010, 2:44 p.m. Updated August 24, 2010, 4:12 p.m.


— It was back to the salt mines for the Kansas Department of Health and Environment. All because of a hardware failure.

The agency announced Tuesday it had to retrieve 120,000 records, including birth and death certificates, from a storage place in a central Kansas salt mine to help restore data from a massive computer system failure.

KDHE Secretary Roderick Bremby has said that problems started with an Aug. 5 hardware failure, affecting 85 percent of KDHE’s servers and putting out of reach millions of records, from marriage certificates to immunization files.

It couldn’t have come at a worse time because many Kansas parents were trying to access records for their children to start school.

Bremby said the problems stemmed from failure of what is called the storage area network, which is described as a central component of KDHE’s network.

The storage area network, or SAN, was purchased from Xiotech Corp. KDHE revealed the name of the Eden Prairie, Minn., company after the Lawrence Journal-World filed a request for the information under the state’s Open Records law.

By Aug. 12, Xiotech and KDHE put in a new storage area network to replace one that KDHE says was 3 years old.

Efforts to recover the lost data have so far proven difficult, which has led the health department to turn back to stored paper records.

KDHE said birth and death certificates as well as other documents have been trucked to Topeka from a salt mine storage facility to help restore the state’s electronic database.

On its website, Xiotech describes itself as one of the largest privately held data storage companies in the world.

Brian Reagan, chief marketing officer with Xiotech, said the system failed because of a disk drive malfunction.

“The weak link in the chain was the disk drive,” Reagan said. “Things go wrong with technology.” He said it was a mechanical malfunction that sometimes happens.

Reagan said Xiotech was working to help correct the situation and he praised the employees at KDHE for their work.

But KDHE continues to struggle to restore the lost data and is notifying customers on its website of continued delays in getting records such as birth and marriage certificates.

Reagan said he understood there would be a news conference Thursday to announce that the problem had been fixed.

As far as liability, Reagan said that would be determined in the future. The main concern now is getting the system back together, he said.


thelonious 7 years, 10 months ago

I wonder if Kris Kobach will be able to prove he was born in Kansas and the US now? If the state of Kansas cannot provide an original birth certificate now, does that mean he can't run for office? Will the "birther movement" now expand to include questioning the citizenship of all Kansans whose original birth documents cannot be found?

On a serious note, this is a disturbing story in light of all of the calls around the country for "showing papers" and such - what would any of us do if the state fubar'd their record keeping to the extent that they could no longer produce or reproduce the original documents?

Sigmund 7 years, 10 months ago

Never heard of of them. They may be the largest privately-held company SAN companies in the country but their technology fails. Even if the network fails the data on disks should ALWAYS be preserved. Even if several disks fail any decent SAN solutions provides some form of RAID to prevent data loss.

Evan Ridenour 7 years, 10 months ago

A disk drive malfunction? That is why these kinds of systems are supposed to have redundancy. Drives fail all the time.

Someone was negligent here.

Noemon 7 years, 10 months ago

I wondered how KDHE's network could have changed so much that it had a single point of failure. I worked in their IT department from '98-2001, and at that time their servers were diverse enough in terms of hardware and operating systems, that the only thing that could have brought the entire network to its knees like this would have been a physical incident that destroyed the server room where most of the machines were housed--a tornado, say, or a fire.

When I quit working for them they were just beginning to talk to Xiotech about their SANs; I remember going to a couple of informational meetings and presentations about the hardware shortly before I moved on. Interesting to see that it bit them in the ass the way it did.

gccs14r 7 years, 10 months ago

Every SAN vendor I've ever talked to, when asked about a backplane failure, says "Those never fail." Ha. SANs are great, but if you're going to hang your entire operation off of one, you'd better have a duplicate data center somewhere else. A duplicate data center is a good idea anyway, but not everyone can afford to do that. KDHE should have had one, though.

tolawdjk 7 years, 10 months ago

They had a duplicate data center.

In a salt mine.

In a bunch of boxes.

gutenberg 7 years, 10 months ago

There are only two kinds of computer users: those who have lost data, and those who are going to.

prairierose54 7 years, 10 months ago

So much for things being backed up every night.

So much for the statement that "no data was lost".

When do we vote in a new governor????????? Then you'll see a change in IT manager knowledge

The part that stinks is that the warning signs were there and people were alerted but didn't do anything til a lot of damage was done.

Commenting has been disabled for this item.