Archive for Monday, August 16, 2010

Bremby: KDHE computer problems to cost more than $600,000 to solve

Kansas Department of Health and Environment Secretary Roderick Bremby on Monday discusses the computer breakdown at his agency over the past few days. Bremby said KDHE is busy restoring systems but that no data was compromised or lost.

Kansas Department of Health and Environment Secretary Roderick Bremby on Monday discusses the computer breakdown at his agency over the past few days. Bremby said KDHE is busy restoring systems but that no data was compromised or lost.

August 16, 2010, 4:27 p.m. Updated August 17, 2010, 8:01 a.m.

Advertisement

— What was supposed to be a “never event” — the crash of the Kansas Department of Health and Environment computer system — became a real event recently, causing a lot of heartburn and costing a lot of money, officials said Monday.

KDHE fixing costly hard drive problem

The Kansas Department of Health and Environment is working to fix problems that caused errors with 85 percent of its hard drives. The fix will cost about $700,000. Enlarge video

The problems started Aug. 5 when a hardware failure began cascading through the system, affecting 85 percent of KDHE’s servers and putting out of reach millions of records, from marriage certificates to immunization files.

“This outage has caused many problems for Kansans and we are very sorry for the inconvenience this has caused,” said KDHE Secretary Roderick Bremby. Bremby noted it couldn’t have come at a worse time because of the start of school and accompanying demand for birth certificates and immunization records.

Lisa Horn, spokeswoman for the Lawrence-Douglas County Health Department, said its 40 employees were affected by the outage. They were unable to access immunization records, the state’s disease surveillance system, and child care licensing records. Employees also couldn’t access appointments for WIC, a federally funded health and nutrition program for women, infants and children.

“It affected everybody,” she said.

She said the department made do by heavily using the fax machine and keeping good paper records.

“It was a really, really tough time for us to not be able to access those servers,” Horn said.

Bremby described the failure of the system and its backup checks as a “never event” that will end up costing the state and vendor “north of $600,000 to $700,000.”

The state has already spent $12,540 and 458 employee hours to work on the situation.

But Bremby said that despite the failure, no data has been compromised or lost and that the agency is “on the road to recovery.”

KDHE is now restoring 25 terabytes of data. The agency still cannot receive e-mails but Bremby said he hoped to have most systems back in operation by the end of this week.

Comments

das 5 years ago

FAIL = The Glorious lack of success..... or "never events" [my addition]

-Urban Dictionary (with help)

das 5 years ago

How about Disaster Recovery or Business Continuity plans also? I read no mention of those......perhaps it's just not in the article......right?

Louis White 5 years ago

The courts have allowed near monopolizations of redundancy systems which has outpriced a lot of clients. Take for example the UK case of Storage Computer of Nashua, NH, the developers of RAID vs. Hitachi, who stole their patent. Of course the court found in favor of big money as is always the case, and the founders of backup technology have been driven out of business.

slowplay 5 years ago

WTF... This isn't backup technology. Redundancy refers to duplication (back ups) not storage. The IT director in charge of this system should be fired.

just_another_bozo_on_this_bus 5 years ago

Nothing will kill the economy like destroying the environment.

mdrndgtl 5 years ago

Fire someone, there was zero redundancy in this case.

just_another_bozo_on_this_bus 5 years ago

Really? How do you know this?

My guess (and that's all it is) is that there was redundancy, probably the standard amount for a state agency of this type.

But that doesn't explain why this failure happened.

Cait McKnelly 5 years ago

Someone who couldn't cut it in the real IT world and took Piss Poor Pay (I just added 3 more P words) because they couldn't get a job anywhere else. You get what you pay for and the state always goes cheap.

Verdad 5 years ago

I'm not a fan of the government, but the I.T. people in the state predicted this failure, but the pepe above them vetoed the necessary improvements.

Flap Doodle 5 years ago

I blame Commie Fifth Columnists.

just_another_bozo_on_this_bus 5 years ago

Are they pleading the fifth, or drinking it?

Flap Doodle 5 years ago

You need to get out into the world more, bozoie.

"A fifth column is a group of people who clandestinely undermine a larger group such as a nation from within, to help an external enemy. See also, Democratic Party..." (slightly edited, but you get the picture)

http://en.wikipedia.org/wiki/Fifth_column

just_another_bozo_on_this_bus 5 years ago

I get it-- you were referring to a fifth column of tin foil.

Cait McKnelly 5 years ago

That's that's called in Wiki language a hack attack, Snap. The page will now be scrubbed of your graffiti.

Thinking_Out_Loud 5 years ago

Contracts are made on lowest bidder.

average 5 years ago

Of the 400 metro areas tracked by the BLS, wages for sysadmins/networking/databases in Topeka are in the lowest 25% and those for Lawrence are in the lowest 10% in the nation (the very lowest outside Puerto Rico in some categories). Not that it's an excuse, but anyone who gets very good at IT gets the hell out, at least to Kansas City.

You get what you pay for.

just_another_bozo_on_this_bus 5 years ago

The wages would depend on who does their IT, wouldn't it?

If it's in-house, the wages will be on the state salary schedule. If it's done under a contract with a private company, then the wages could be determined by the local wage levels.

average 5 years ago

You can look up every state employee's salary at the KanView site (http://www.kansas.gov/KanView/client/js/#dataTable=salary/index). The top IT person I saw listed there was making about $72k. Which isn't chicken feed in Topeka, but also isn't going to keep any senior people who have other options (that's middle-of-the-pack in KC, let alone a bigger market). But, you are right that the more serious iron may well be under an HP-EDS or IBM contract.

monkeyspunk 5 years ago

The head of IT at KDHE makes just under 100K.

Godot 5 years ago

Maybe she/his/it is Bremby. Would explain multitudes.

just_another_bozo_on_this_bus 5 years ago

Or, perhaps less of more, and a bit more of less.

riverdrifter 5 years ago

Boz, wasn't referring to you. God knows!

Godot 5 years ago

Government incompetence defined: a man declares bankruptcy, personally, and he is put in charge of an agency that has a multi-million dollar budget that pulls the strings on the economic growth of the state, and this happens. Why is anyone surprised?

notajayhawk 5 years ago

When I was a state employee (in another state), we moved into a new building, and we all had "new" computers. "New" was an extremely relative term. We were the last region in the state to get our new building, and the computers had been delivered some time back. With the state's purchasing procedures, they'd been put to bid years before. You can get one just like it now for about $80-100 from any number of 'refurbishers' on the internet - with free shipping.

I doubt the state of Kansas operates any differently.

prairierose54 5 years ago

It took 11 days to make a public statement regarding this issue. There were indications of problems before the crash. Nothing was done to stop it. Looks like no one will be held accountable for this.

He won't be there after the election. He's just riding out the big paycheck til it's over for him.

TopJayhawk 5 years ago

Just declare it unsafe like the coal plant, and do away with it. I'm surprised this guy is even still around. Figured he would go with the whiner in chief to DC.

Randy Leonard 5 years ago

This is just the tip of the iceberg at KDHE. I believe strongly in the function of government, but KDHE needs a major overhaul. The program areas seem to function well and are staffed by dedicated professionals. On the other hand the administration of this agency is totally incompetent. The computer outage is only the latest and most publicly visible sign of the problem. Many policies and practices, including IT and personnel policies, in the department are counter to employee productivity. I personally know many of the hard working professionals at KDHE and hear many tales of of almost unbelievable mismanagement. In my 30 plus years experience in the military, the private sector, and state government I feel safe in saying KDHE tops them all for incompetence in management.

slowplay 5 years ago

Snodgrass,

Doesn't it bother you that you come off as a complete idiot every time you post?

gl0ck0wn3r 5 years ago

lolz at Bremby and KDHE IT. Nice planning there guys.

Joe Ryan 5 years ago

Sounds like Bremby could use a few dollars from the tax revenue generated by the Sunflower Power plant.

Cody Ochs 5 years ago

Ah, the Monday morning quarterbacks step in. Sure, everyone knows what best practice is, but I have never worked in a place that had the resources to actually follow it. Every system I have worked with had vulnerabilities that kept me up some nights, but I was not given the resources to make everything hunky dory. Also, vendors will often sell you a poorly designed (but cheap) system in order to maximize their margin on the deal.

monkeyspunk 5 years ago

From what I have heard about the situation your last assertion there about the vendors and their promises about a system are "right on." Virtualization may have been the culprit here.

sciencegeek 5 years ago

Monday Morning Quarterbacking--exactly!

Anyone in IT, particularly for city, state or county government, is saying "there but for the grace of God". The average Joe has NO idea of what it's like trying to keep up to date with a shoestring budget. When comparing notes with peers in the private sector, they are utterly amazed at how much we do with the little we have to work with. Their staffing, training, equipment and communications are far superior to what tax-supported entities have, yet the information in government hands is critical. BUT--what's a 20% layoff here, a 10-year-old-server there? We can't even keep the building temperature within limits!

Not that KDHE's IT management doesn't have its issues but---it does no good to have adequate backups--which I think they did--,you also have to have the equipment to restore the data TO. That is the kind of redundancy that the state can't, or won't, afford.

Steve Bunch 5 years ago

I understand the new IT system will be coal powered.

devobrun 5 years ago

I don't see the big deal here. Computers behave in this manner all the time. They are always oversold. They fail at the most inopportune time. Managers and IT folks make excuses. Reality sets in.

It happens with data storage (KDHE). It happens with modeling (climate science). It happens with internet users (hackers and countermeasures). It happens with Y2K (end of all things computer or just another blip in our world?)

Computers are great when then work and disastrous when they break. The more we transfer our work to them, the more we become dependent upon them. When they fail they fail spectacularly because we rely on them too much.

I actually applaud KDHE for having paper backup and other means of restoring the system. Again, I don't see the big deal here. This kind of .....happens all the time.

just_another_bozo_on_this_bus 5 years ago

"This kind of .....happens all the time."

Yep-- even to engineers and "real" scientists running controlled experiments.

Flap Doodle 5 years ago

I think that KDHE has acted stupidly.

just_another_bozo_on_this_bus 5 years ago

Likely not-- the IT problems at KDHE probably exist throughout state government, and really fixing those problems likely entails increased funding, something that's likely not going to happen.

Sigmund 5 years ago

The problems that exist at KDHE do exist throughout the state government, it is called incompetence without any accountability. There is not enough taxpayer dollars in the world to fix stupid.

estespark 5 years ago

Perhaps it became self aware and revolted against its creators.

Jimo 5 years ago

Seems that the Legislature should have ponied up another $700k for a better storage system rather than have cut taxes.

The Legislators probably assumed that if anything went wrong (a) they'd be long gone and (b) voters would just say "those incompetent KDHE people".

Sorry, the Legislature (perhaps incompetently) oversees everything KDHE does.

Centerville 5 years ago

I know this is a tremendous lot to ask, but an enterprising reporter would have done a little cross referencing between the KDHE consultants and campaign contributions. Oh wait, that only happens when we have Republican governors. Sorry.

sciencegeek 5 years ago

Taxpayers of Kansas, take note:

--Staffing has been cut back for multiple programs that directly serve the people of the state. --State computer systems don't have enough money to be run as everyone knows they should be. -- KPERS has been underfunded by the legislature year after year, putting it on shaky financial ground.
--The state salary schedule has been so far below the private secctor that some job classes needed a 25% increase to catch up.

BUT-- --Legislators have new desks that cost approximately $1800 each. --The refurbishing of the Capitol is so far over budget that it makes a $500 toilet seat look like a steal. --Lew Perkins travels in a style reminiscent of Bernie Madoff.

SOOO--- when you re-elect the same legislators again and again, you're saying that it's all just fine with you. Think about it.

jmadison 5 years ago

$285 Million spent for the state capitol renovation. Are we getting value from our state government or are the politicians in Topeka doing as poorly as our political elites in DC?

Randall Uhrich 5 years ago

Kansas never has claimed to be at the forefront of technology. Us and Arkansas, wondering what the hell is going on around us.

Sigmund 5 years ago

"Rod Bremby, who oversees an annual budget of nearly $216 million as a state Cabinet secretary, filed this week for personal Chapter 13 bankruptcy protection along with his wife, physician April Bremby." http://www2.ljworld.com/news/2007/apr...

It is good to see that Rod is equally incompetent managing the KDHE as his personal finances. Being a political appointee he was selected for who he knows not what he knows, and his tenure is proof of that. He routinely has disregarded staff recommendations and is driven by ideology rather than rational decision making. This "Never Event" was foreseen and ignored. Breby should resign, or be faced with a full investigation and audit, or both. Or will the political class white wash this mismanagement of state resources out of fear of revealing their own incompetence?

just_another_bozo_on_this_bus 5 years ago

Clearly, no investigation is necessary, Sigmund. You apparently already have all the pertinent facts.

Sigmund 5 years ago

Clearly without an investigation, Bozo, no pertinent facts will be uncovered. Why is it you have no curiosity when it comes to Democratic appointees?

Sigmund 5 years ago

deathpenaltyliberal (anonymous) says… "Don't want to pay for a modern system, and happy to gripe about the state workers when the old systems fail."

What makes you think the system is not "modern"? The only failure that I can think of that would take 25TB of storage off-line in a "cascading" manner yet cause only a "loss" of access to data (but not data integrity) is one impacting both core switches in a modern SAN environment. Even if these are old switches, shouldn't a competent Director allocate current budget to refresh those items as needed? Isn't that Bremby's job???

Curious2010 5 years ago

There was some information that some backup data may have been corrupted. If that was the case, would that affect your analysis of what happened?

Also, why would the State have a cost, even shared with the vendor of $600,000 - $700,000? Wouldn't these devices be under maintenance? Or is this cost the estimates to add redundancy - which does not appear to have been in place - and seems to me to be an important topic.

Sigmund 5 years ago

Not really. Corrupt backup data for some systems but not others does not lead to a different root cause of this event. More likely some systems bakup's were misconfigured or had other problems which weren't discovered until they were possibly needed. This kind of equipment is designed to fail before data on disk would be corrupted. If the data on the SAN disks are not corrupt then backups will not be needed. Any corruption that occurred to "live" data during this event would not be backed up yet anyway.

As to the cost it sounds like they are replacing everything from both core switches to the edge and perhaps increasing capacity (ie 2GBS ->4,6,8GBS) to boot, and yes they really are that expensive. The good news is that new equipment of this class usually comes with a 3-5 year support contract. After that time you can purchase support yearly but typically that one year support contract costs more than purchasing new equipment to replace the old and getting the 3-5 years support for "free". The catch is it requires a lot of coordinated work from a lot of people to migrate from old to new equipment without taking every system offline. That said it happens all the time and that is why you pay a competent SAN administrator 150-200k/year.

Or you can run without any support and then pay time an materials when (not if) things go wrong which is by far the most expensive option with outages that last the longest. Me thinks this is what Bremby was doing and for more than just a couple of years.

Randy Leonard 5 years ago

Sigmund, that is exactly my understanding of the problem. The switches failed bringing down the SAN. Someone at first decided it was a software problem so they reloaded the software and it immediately failed again. From the information I have heard, their IT director has been so worried about security that the functionality and reliability of the system was practically ignored.

tolawdjk 5 years ago

Much like my current employer's IT dept.

I think they will only feel the data is secure if -no one- can get to it.

Sigmund 5 years ago

Oh, BTW, how comfortable is anyone with a National Health Service where your medical records are kept on government computers with management like Bremby who are appointed because of who they know, not what they know?

tolawdjk 5 years ago

The government is not the sole owner of incompetence. I've dealt with govt. contractors, private insurers, doctors offices, hospitals, fast food drive throughs, 5 star restaurants, schools (public and private), volunteers, and paid professionals and I can give at least on example of "incompetence" in dealing with each.

Hell, I don't even trust myself sometimes in keeping my own medical records. Pretty soon these files all start looking alike. I think we need to get back to the three L's of medicine...Leprosy, Leeches, and Lycanthropy.

Sigmund 5 years ago

Never claimed that corporations were perfect, management is incompetent at both in my experience. The difference is corporations can fail or be taken over by competent companies That will never happen at the State of Kansas, have any idea what it takes to get even a low level incompetent State of Kansas employee fired? If Mutual of Omaha fails I can easily change to Mutual of New York (or whatever). Much hard for me to move to Nebraska or NY.

Curious2010 5 years ago

From a KDHE news release. Also, KDHE presented a powerpoint that noted restoring began on 8/6 - but 8/11 they determined hardware the problem and ordered the SAN

Does it really take 6 days to determine a SAN hardware problem?

"The technical problems stemmed from a failure of the storage area network (SAN). The SAN is a central component of KDHE’s network. KDHE’s network configuration is state-of-the-art and used by many leading organizations.

Upon observing problems occurring on August 5, KDHE IT staff immediately brought in a team of vendor personnel and leading third-party consultants to assist in diagnosing the problem, rebuilding services and restoring applications. On August 11, it was discovered by IT staff and a third-party consultant brought on board, that the hardware was a factor.

A new SAN was shipped from the vendor August 12 and arrived August 13, along with vendor personnel who began working with KDHE staff on installation. Currently KDHE staff, along with the vendors, are working diligently to restore all services. However, with 150 servers and 25 terabytes of data, restoration is a time consuming and ongoing process. "

puddleglum 5 years ago

I just finished up eating a jello brand 'mousse temptations' pudding and I need to let everyone know how much it sux. do yourself a favor-don't buy this crap.

I am very beautiful, but this dessert was just horrible, even my presence couldn't save it. what was I thinking? Ray Charles could even see that this was just some kind of marketing ploy to get rid of some old, junky pudding.... deng.

CountyResident 5 years ago

I guess if KDHE would have been operating like BP ( a private company) this kind of failure would have never happened.

Sigmund 5 years ago

I too am glad that the KDHE doesn't drill for oil in the gulf. On the other hand wasn't it the Obama administration the recently gave BP a Safety Award after issuing hundreds of safety violations? Was that before or after their contributions to the Obama campaign?

At least the costs of the clean up will be born by a private company whose shareholders voluntarily invested in BP stock and the senior management has been dismissed while the costs with government incompetence gets pushed onto the shareholders and no one ever gets fired no matter how incompetent.

Sigmund 5 years ago

Of course that last line should have been, "At least the costs of the clean up will be born by a private company whose shareholders voluntarily invested in BP stock and the senior management has been dismissed while the costs with government incompetence gets pushed onto the taxpayers and no one ever gets fired no matter how incompetent."

Commenting has been disabled for this item.