Timeline of events
Aug. 5 -- Hardware failure; data restoration efforts begin.
Aug. 12 -- Storage area network (SAN) failed; restoration efforts cease.
Aug. 13 -- New SAN installed.
Aug. 14 -- Restoration efforts resume.
Aug. 16 -- KDHE Secretary Roderick Bremby has news conference to say that KDHE was “on road to recovery,” no data had been compromised or lost, and that 150 servers with 25 terabytes of data were affected.
Aug. 24 -- KDHE announces it has retrieved 120,000 records from salt mines in central Kansas.
Aug. 26 -- Bremby holds another news conference, also with representative of Xiotech, the vendor of the SAN. Bremby says Xiotech will pay for full cost of recovery and system upgrades and overtime wages.
Aug. 30 -- KDHE announces it has access to electronic versions of birth certificates and other vital statistics records.
Source: KDHE timeline and statements by Bremby
Topeka — The computer disruption that rocked the Kansas Department of Health and Environment for most of August is prompting a general review of computer systems in all state agencies.
Gov. Mark Parkinson said he wants agencies to check their systems, and Kansas Department of Administration Secretary Duane Goossen said a more in-depth look will be needed in the near future.
“There will be a bigger effort to figure out what happened,” Goossen said.
“What does it mean for other systems throughout the state, do other agencies have similar systems? These are legitimate questions. That kind of reviewing hasn’t been done yet, but it will be,” he said.
On Aug. 5, KDHE began experiencing technical problems that took down 150 servers, or 85 percent of the department’s total. Officials said the problems stemmed from a disk drive malfunction that caused the failure of a storage area network, or SAN, which is a central component of KDHE’s network.
KDHE workers had no electronic access to birth certificates, death certificates and other vital records that are commonly requested by the public.
A backlog of 8,000 requests for such records piled up, and the department had to retrieve records kept in storage in salt mines in central Kansas to manually process requests. The agency has temporarily doubled the staff of its vital statistics office, adding 50 employees from other parts of the agency.
It wasn’t until Monday, 26 days after the first problems arose, that KDHE said its last major information system was fully functional.
Now the questioning starts. Was there enough redundancy built into the system, meaning if one system failed is there an up-to-date copy of the data somewhere else.
KDHE uses tape as a backup to copy data and KDHE Secretary Roderick Bremby has said because of the massive number of records there is no need to build more redundancy for an event that Bremby said was simply never supposed to happen.
“We don’t anticipate this ever happening again,” he said.
But agency officials did say they are looking at improving the capability of restoring data more quickly.
“KDHE is looking at costs for a quicker solution,” said Kristi Pankratz, a spokeswoman for KDHE.
She also said that a KDHE vendor — OnTrack — was able to recover the data from hard drive disks and the tape backup was not needed.
Comments
LJWorld.com doesn’t necessarily condone the comments here, nor does it review every post. Read our full policy. Also, read about banned accounts and harassing comments.
sunshine_noise (anonymous) says…
and panic occurred when these people had to actually work at their jobs by manually search for documents, copy them and mail. Just like folks did before computers came along. Computers are really only a temporary solution and I think we've come to rely on them too much and have become too complacent in that dependency.
WorkinGirl (anonymous) replies…
Which is why we are reading the LJWolrd online.
BrentS (anonymous) replies…
Yep, because one day computers will just go away, like written language, power tools, and the automobile. We must be prepared for our neo-primitive future!
/sarcasm
OonlyBonly (anonymous) says…
I think there's no need for "questioning" anything.
It took 26 days to get the system recovered - there wasn't any redundancy other than the "original" records in the salt mine.
25 Terabytes, 120,000 records and "we don't need no steenkin' backup." What!?!?!
“We don’t anticipate this ever happening again,” he (Bremby) said. And I'm sure it won't - until the next time.
jrlii (anonymous) says…
I think it is a very good thing for agencies to look into their Storage Area Network recovery plans because, while SANs are perhaps the most reliable random access mass storage devices to date, when they go awry there is hell to pay.
Just the other day a SAN (a different brand than KDHE's) failed in Virginia and took down the Department of Motor Vehicles amongst others. Talk about a mess!
Eride (anonymous) says…
Storing paper records in a salt mine isn't redundancy.
Bremby should be fired if he really suggested that the system doesn't need any further redundancy when simple drive failures caused the entire storage network to be lost for a month.
just_another_bozo_on_this_bus (anonymous) says…
If they restored all of the data from backups, then they indeed had sufficient backups.
But backups aren't intended to prevent system failures. They're merely intended to keep data from being irretrievably lost when the systems do fail.
If there are reasonable measures that can be taken to prevent future system failures, they should be ascertained and implemented, but that may have very little to do with additional storage systems.
gbulldog (anonymous) says…
each "little god" must run their system just like each usd must continue to operate in kansas. each "little god" must protect their kingdom because if they do not, another "little god" will take over. as a result, taxes will continue to go up and service quality will continue to decrease.. and don"t forget, everyone must be "politically correct."
Boston_Corbett (anonymous) says…
Here's an idea: Check the batteries in those uninterruptibile power supply boxes.
Boston_Corbett (anonymous) says…
Yea, Hope_and_Change, we need fewer Government jobs. Maybe Governor Sam will out-source the running of state computers. You know, run it "like a business" and all...
Lets see how that worked out in Virginia...
http://www.washingtonpost.com/wp-dyn/...
(welcome back, Hope_and_Change. How many times have you been booted from the LJW boards?)
wastewatcher (anonymous) says…
Keeping computer systems up to date and having back up to prevent such happenings should be part of the day to day activities of the entire executive branch agencies. Looks to me like the Gov's people haven't been doing their jobs and now he is making a big deal out of what should be routine day to day work. What do you exoect out of Bremby, remember he is a Sebelius lackey who couldn,t balance his own checkbook so he chose bankruptcy while making over a hundred grand on the state payroll.
jrlii (anonymous) says…
Storage Area Network devices are amongst the most reliable ways to store data while keeping it accessible, so going to one was a pretty good idea.
What was lacking was a remote hot site.
And that's expensive. You have to replicate practically all of your functionality at a separate site, and have it ready to go at the drop of a hat. Or the malfunction of a controller.
Backup sites get short shrift, 'cause they are kind of like paying the preacher: It really only makes a difference when you are dead,.
gbulldog (anonymous) says…
the remote hot site is available to State agencies. The real question is why did Health & Environment not use it, or maybe they don't know how.