Archive for Wednesday, August 18, 2010

Mystery surrounds KDHE crash

August 18, 2010


— Nearly two weeks after a major state agency’s computer system failed, there is no official word on the cause.

On Tuesday, the Kansas Department of Health and Environment continued restoring systems from the crash that started Aug. 5 and ultimately affected 150 servers with 24.7 terabytes of data.

On Monday, KDHE Secretary Roderick Bremby held a news conference to talk about the situation.

Bremby asked for patience from the public and said no data had been compromised or lost.

Bremby said the problems stemmed from failure of what is called the storage area network, which is described as a central component of KDHE’s network.

By Aug. 12, a new storage area network was delivered and installed to replace one that KDHE says was three years old.

But why did the storage area network fail? KDHE said Tuesday, “Regarding the reasons for the SAN failing, we are reviewing reports now along with our vendor and consultants. We hope to have further info to release at a later date.”

At his news conference, Bremby refused to release the name of the storage area network vendor. The Lawrence Journal-World has filed a request for the company’s name under the Kansas Open Records Act.

KDHE says the system failure has not affected other state agencies, but has affected many local health departments that were unable to access data such as immunization records, the state’s disease surveillance system and child care licensing records.


sciencegeek 7 years, 10 months ago

Oh, for crying out loud, LJWorld! There's plenty of state employees who can tell you who the vendor is! Won't, of course, cause they need their jobs...

Sigmund 7 years, 10 months ago

It looks like the SAN "fabric" and not the SAN disks that are having the issue. This could include everything from core switches to the edge. If this equipment is three years old then it still should be covered under warranty and the vendor needs to be held accountable on the Service Level Agreement. Further, if it was delivered and installed by Aug 12th this should have been up and running several days ago.

If I had to guess this is EMC/Clariion or McData equipment and if the phrase "running out of buffer credits" rings a bell with anyone familiar with the situation, this is almost certainly a firmware compatibility issue with another vendors equipment and you just wasted hundreds of thousands of dollars.

But instead of speculation I am sure even Bozo would support an investigation to uncover the pertinent facts to determine what degree of fault lies with KDHE and what part lies with the vendor(s). If the KDHE is not being forthcoming with information to the press and need to file a freedom of information request, then it is time for the Kansas Legislature to commence hearings and put these clowns under oath.

just_another_bozo_on_this_bus 7 years, 10 months ago

"But instead of speculation I am sure even Bozo would support an investigation to uncover the pertinent facts to determine what degree of fault lies with KDHE and what part lies with the vendor(s)."

No objections here.

somebodynew 7 years, 10 months ago

Check the archives - - - -

Wasn't there a high ranking State official who left office early (not too long ago) to go to work for a (uh, let me think....) Computer company. One that happened to have a lot of contracts with the State ?????

Don't have any details, but is this why the vendor isn't being named????

OK, now I sound like Smitty, but at least I didn't bring LPD into it.

kdhefan 7 years, 10 months ago

I have been wondering for some time why KDHE hasn't gone to cloud computing. Such a solution would be far cheaper than for a major state agency to try to maintain and operate its own servers. In addition "the cloud" would be far more secure, could offer unbreakable encryption, as meuch redundant backup as one might desire, and availability from anywhere, by any computer to the necessary parts of the agency's data, including all of the operations of the agency itself. With such a solution, no data and no software would have to reside in computer in Topeka. A wide variety of companies such as Google, IBM, Amazon, Akamai, just to name a few are offering these services. What would be saved is the cost of servers, data storage devices, and any hardware needed to connect them, the cost of space now rented to house the servers, cost of energy to run the servers, cost of energy to air condition the space so that the servers do not overheat. It is also likely that some of the IT people the agency now employs would have their jobs dramatically changed as a result of such a move, and ultimately, the state's IT professionals could be deployed to problem solve at a more micro level within the agency and professional expense overall would be decreased. If I were in charge, I would be going out with a very request for bids or proposals for supply of cloud computer for KDHE. Faster and more reliable computing would be the result, and significant cost savings, both up front and ongoing.

local_interest 7 years, 10 months ago

HIPAA and data ownership would prevent you from doing a plan like this.

Sigmund 7 years, 10 months ago

Any update on this story, are they still down??

Commenting has been disabled for this item.