Advertisement

Archive for Monday, August 16, 2010

Bremby: KDHE computer problems to cost more than $600,000 to solve

Kansas Department of Health and Environment Secretary Roderick Bremby on Monday discusses the computer breakdown at his agency over the past few days. Bremby said KDHE is busy restoring systems but that no data was compromised or lost.

Kansas Department of Health and Environment Secretary Roderick Bremby on Monday discusses the computer breakdown at his agency over the past few days. Bremby said KDHE is busy restoring systems but that no data was compromised or lost.

August 16, 2010, 4:27 p.m. Updated August 17, 2010, 8:01 a.m.

Advertisement

— What was supposed to be a “never event” — the crash of the Kansas Department of Health and Environment computer system — became a real event recently, causing a lot of heartburn and costing a lot of money, officials said Monday.

KDHE fixing costly hard drive problem

The Kansas Department of Health and Environment is working to fix problems that caused errors with 85 percent of its hard drives. The fix will cost about $700,000. Enlarge video

The problems started Aug. 5 when a hardware failure began cascading through the system, affecting 85 percent of KDHE’s servers and putting out of reach millions of records, from marriage certificates to immunization files.

“This outage has caused many problems for Kansans and we are very sorry for the inconvenience this has caused,” said KDHE Secretary Roderick Bremby. Bremby noted it couldn’t have come at a worse time because of the start of school and accompanying demand for birth certificates and immunization records.

Lisa Horn, spokeswoman for the Lawrence-Douglas County Health Department, said its 40 employees were affected by the outage. They were unable to access immunization records, the state’s disease surveillance system, and child care licensing records. Employees also couldn’t access appointments for WIC, a federally funded health and nutrition program for women, infants and children.

“It affected everybody,” she said.

She said the department made do by heavily using the fax machine and keeping good paper records.

“It was a really, really tough time for us to not be able to access those servers,” Horn said.

Bremby described the failure of the system and its backup checks as a “never event” that will end up costing the state and vendor “north of $600,000 to $700,000.”

The state has already spent $12,540 and 458 employee hours to work on the situation.

But Bremby said that despite the failure, no data has been compromised or lost and that the agency is “on the road to recovery.”

KDHE is now restoring 25 terabytes of data. The agency still cannot receive e-mails but Bremby said he hoped to have most systems back in operation by the end of this week.

Comments

Sigmund 3 years, 8 months ago

Of course that last line should have been, "At least the costs of the clean up will be born by a private company whose shareholders voluntarily invested in BP stock and the senior management has been dismissed while the costs with government incompetence gets pushed onto the taxpayers and no one ever gets fired no matter how incompetent."

0

Sigmund 3 years, 8 months ago

I too am glad that the KDHE doesn't drill for oil in the gulf. On the other hand wasn't it the Obama administration the recently gave BP a Safety Award after issuing hundreds of safety violations? Was that before or after their contributions to the Obama campaign?

At least the costs of the clean up will be born by a private company whose shareholders voluntarily invested in BP stock and the senior management has been dismissed while the costs with government incompetence gets pushed onto the shareholders and no one ever gets fired no matter how incompetent.

0

CountyResident 3 years, 8 months ago

I guess if KDHE would have been operating like BP ( a private company) this kind of failure would have never happened.

0

TrooGrit 3 years, 8 months ago

Here's what....let Bremby take a 600k pay cut for his oversight at KDHE. That would be fair. I mean, if it cost that much to fix a problem he could have avoided, it's only fair, right? Ph*ck.

0

puddleglum 3 years, 8 months ago

I just finished up eating a jello brand 'mousse temptations' pudding and I need to let everyone know how much it sux. do yourself a favor-don't buy this crap.

I am very beautiful, but this dessert was just horrible, even my presence couldn't save it. what was I thinking? Ray Charles could even see that this was just some kind of marketing ploy to get rid of some old, junky pudding.... deng.

0

TrooGrit 3 years, 8 months ago

MY management wasn't incompetent. We never had a failure of this sort in 30+ years. It has to do with expectations. If you set the bar low, you'll get that type of performance. I've seen people fired for being 'sloppy' with their work. No one ever gets fired from KDHE. I understand that the wages are low. But, if you are an I.T. professional then act it, be it. It doesn't matter that you are getting pennies on the dollar compared to your brethren in private enterprise. If you want to wear the badge, then live up to it...no matter what you are paid!

0

Sigmund 3 years, 8 months ago

Never claimed that corporations were perfect, management is incompetent at both in my experience. The difference is corporations can fail or be taken over by competent companies That will never happen at the State of Kansas, have any idea what it takes to get even a low level incompetent State of Kansas employee fired? If Mutual of Omaha fails I can easily change to Mutual of New York (or whatever). Much hard for me to move to Nebraska or NY.

0

Sigmund 3 years, 8 months ago

Oh, BTW, how comfortable is anyone with a National Health Service where your medical records are kept on government computers with management like Bremby who are appointed because of who they know, not what they know?

0

TrooGrit 3 years, 8 months ago

Or it could be, that they don't have any IT personnel worth their salt. Take your pick.

0

bear1956 3 years, 8 months ago

Sigmund, that is exactly my understanding of the problem. The switches failed bringing down the SAN. Someone at first decided it was a software problem so they reloaded the software and it immediately failed again. From the information I have heard, their IT director has been so worried about security that the functionality and reliability of the system was practically ignored.

0

Sigmund 3 years, 8 months ago

deathpenaltyliberal (anonymous) says… "Don't want to pay for a modern system, and happy to gripe about the state workers when the old systems fail."

What makes you think the system is not "modern"? The only failure that I can think of that would take 25TB of storage off-line in a "cascading" manner yet cause only a "loss" of access to data (but not data integrity) is one impacting both core switches in a modern SAN environment. Even if these are old switches, shouldn't a competent Director allocate current budget to refresh those items as needed? Isn't that Bremby's job???

0

deathpenaltyliberal 3 years, 8 months ago

All I hear here is typical two-faced conservative talk.

Don't want to pay for a modern system, and happy to gripe about the state workers when the old systems fail.

You people are mostly morons.

0

deathpenaltyliberal 3 years, 8 months ago

"Godot (anonymous) says… Government incompetence defined: a man declares bankruptcy, personally, and he is put in charge of an agency that has a multi-million dollar budget that pulls the strings on the economic growth of the state, and this happens. Why is anyone surprised?"

Are you talking about Kris Kobach and/or Mario Rubio?

0

Sigmund 3 years, 8 months ago

"Rod Bremby, who oversees an annual budget of nearly $216 million as a state Cabinet secretary, filed this week for personal Chapter 13 bankruptcy protection along with his wife, physician April Bremby." http://www2.ljworld.com/news/2007/apr...

It is good to see that Rod is equally incompetent managing the KDHE as his personal finances. Being a political appointee he was selected for who he knows not what he knows, and his tenure is proof of that. He routinely has disregarded staff recommendations and is driven by ideology rather than rational decision making. This "Never Event" was foreseen and ignored. Breby should resign, or be faced with a full investigation and audit, or both. Or will the political class white wash this mismanagement of state resources out of fear of revealing their own incompetence?

0

LJ Whirled 3 years, 8 months ago

Democrats in charge of the Statehouse = A Never AGAIN Situation

Way to kill the State, Kathy!

0

Randall Uhrich 3 years, 8 months ago

Kansas never has claimed to be at the forefront of technology. Us and Arkansas, wondering what the hell is going on around us.

0

TrooGrit 3 years, 8 months ago

sciencegeek:

I know you are trying to look at the big picture here, and your wonderful fact laden story does illustrate a state government taxed to the max to just keep up, but here are a few facts for you, the typical leftist looney poster on these boards:

  1. Legislators have always had new desks, I dunno if they actually cost 1800 or not, but that is nothing new.
  2. The refurbishing of the Capitol is a federally funded mandate, and as government does (always), it cost the taxpayers more, because of the basic ineptness of government.
  3. Lew Perkin's travel is payed by the Athletic Department, not tax dollars. So you can quit foaming at the mouth, thinking your tax dollars had anything to do with big Lew's Excellent Adventures.

Let me remind you: KPERS in around in 1985 time frame had a massive fraud, by some lunatic that bought all kinds of exotic futures with the funding. This was done by one man! He went to prison, I don't know all the details, but they have reason to be, shall we say, 'cautious'.

You aren't really very scientific at all. If you were, you'd check your facts a little closer.

0

jmadison 3 years, 8 months ago

$285 Million spent for the state capitol renovation. Are we getting value from our state government or are the politicians in Topeka doing as poorly as our political elites in DC?

0

sciencegeek 3 years, 8 months ago

Taxpayers of Kansas, take note:

--Staffing has been cut back for multiple programs that directly serve the people of the state. --State computer systems don't have enough money to be run as everyone knows they should be. -- KPERS has been underfunded by the legislature year after year, putting it on shaky financial ground.
--The state salary schedule has been so far below the private secctor that some job classes needed a 25% increase to catch up.

BUT-- --Legislators have new desks that cost approximately $1800 each. --The refurbishing of the Capitol is so far over budget that it makes a $500 toilet seat look like a steal. --Lew Perkins travels in a style reminiscent of Bernie Madoff.

SOOO--- when you re-elect the same legislators again and again, you're saying that it's all just fine with you. Think about it.

0

Centerville 3 years, 8 months ago

I know this is a tremendous lot to ask, but an enterprising reporter would have done a little cross referencing between the KDHE consultants and campaign contributions. Oh wait, that only happens when we have Republican governors. Sorry.

0

Jimo 3 years, 8 months ago

Seems that the Legislature should have ponied up another $700k for a better storage system rather than have cut taxes.

The Legislators probably assumed that if anything went wrong (a) they'd be long gone and (b) voters would just say "those incompetent KDHE people".

Sorry, the Legislature (perhaps incompetently) oversees everything KDHE does.

0

estespark 3 years, 8 months ago

Perhaps it became self aware and revolted against its creators.

0

beaujackson 3 years, 8 months ago

Will November really "fix it"?

0

Flap Doodle 3 years, 8 months ago

I think that KDHE has acted stupidly.

0

devobrun 3 years, 8 months ago

I don't see the big deal here. Computers behave in this manner all the time. They are always oversold. They fail at the most inopportune time. Managers and IT folks make excuses. Reality sets in.

It happens with data storage (KDHE). It happens with modeling (climate science). It happens with internet users (hackers and countermeasures). It happens with Y2K (end of all things computer or just another blip in our world?)

Computers are great when then work and disastrous when they break. The more we transfer our work to them, the more we become dependent upon them. When they fail they fail spectacularly because we rely on them too much.

I actually applaud KDHE for having paper backup and other means of restoring the system. Again, I don't see the big deal here. This kind of .....happens all the time.

0

Steve Bunch 3 years, 8 months ago

I understand the new IT system will be coal powered.

0

reverber 3 years, 8 months ago

Ah, the Monday morning quarterbacks step in. Sure, everyone knows what best practice is, but I have never worked in a place that had the resources to actually follow it. Every system I have worked with had vulnerabilities that kept me up some nights, but I was not given the resources to make everything hunky dory. Also, vendors will often sell you a poorly designed (but cheap) system in order to maximize their margin on the deal.

0

Joe Ryan 3 years, 8 months ago

Sounds like Bremby could use a few dollars from the tax revenue generated by the Sunflower Power plant.

0

gl0ck0wn3r 3 years, 8 months ago

lolz at Bremby and KDHE IT. Nice planning there guys.

0

Paul R Getto 3 years, 8 months ago

Regardless of the reasons, this is a good example of how vulnerable the modern world can be to computer issues. If the "evil ones" are serious about damaging our economy, they should be looking for the right boy genius to fund so they can slow or shut down computer systems around the world. That would do more damage than any IED or bombers' vest. Given the short resources in government agencies as budgets are cut, I suspect other systems are vulnerable for lots of reasons. As some have pointed out, good IT people are worth all they make when they can keep a system running.

0

bear1956 3 years, 8 months ago

This is just the tip of the iceberg at KDHE. I believe strongly in the function of government, but KDHE needs a major overhaul. The program areas seem to function well and are staffed by dedicated professionals. On the other hand the administration of this agency is totally incompetent. The computer outage is only the latest and most publicly visible sign of the problem. Many policies and practices, including IT and personnel policies, in the department are counter to employee productivity. I personally know many of the hard working professionals at KDHE and hear many tales of of almost unbelievable mismanagement. In my 30 plus years experience in the military, the private sector, and state government I feel safe in saying KDHE tops them all for incompetence in management.

0

TopJayhawk 3 years, 8 months ago

Just declare it unsafe like the coal plant, and do away with it. I'm surprised this guy is even still around. Figured he would go with the whiner in chief to DC.

0

prairierose54 3 years, 8 months ago

It took 11 days to make a public statement regarding this issue. There were indications of problems before the crash. Nothing was done to stop it. Looks like no one will be held accountable for this.

He won't be there after the election. He's just riding out the big paycheck til it's over for him.

0

notajayhawk 3 years, 8 months ago

When I was a state employee (in another state), we moved into a new building, and we all had "new" computers. "New" was an extremely relative term. We were the last region in the state to get our new building, and the computers had been delivered some time back. With the state's purchasing procedures, they'd been put to bid years before. You can get one just like it now for about $80-100 from any number of 'refurbishers' on the internet - with free shipping.

I doubt the state of Kansas operates any differently.

0

Godot 3 years, 8 months ago

Government incompetence defined: a man declares bankruptcy, personally, and he is put in charge of an agency that has a multi-million dollar budget that pulls the strings on the economic growth of the state, and this happens. Why is anyone surprised?

0

Bob Kraxner 3 years, 8 months ago

Am I the only one here who sees this for what it really is ? Its nothing more than an excuse to update their obsolete hardware...at someone elses expense, of course.

0

TrooGrit 3 years, 8 months ago

No, he's just the local village idiot. Never short of an opinion about how his mommy didn't change his diddys right, or his pot smoking thumb sucking miscreant self that never had approval, at any level. And still he projects his drivel into the universe, hoping some spaceman somewhere says, "Yeah, I was teased as a child orphan too" and will gain some sort of redemption by that act.

0

Snodgrass 3 years, 8 months ago

Who is this bozo_ on_ this_ bus poster? Is he the epitome of a Democrat or what? Cranky, Fussy, Faux Articulate and just plain lost in his virtual transformation world. Jeez

0

RidgeRunner 3 years, 8 months ago

"If you think it's expensive to hire a professional to do the job, wait until you hire an amateur." --Red Adair

0

TrooGrit 3 years, 8 months ago

I never even considered taking a job with the state of Kansas because of the abysmal pay. I received 3 times as much in the private sector, because the state of Kansas was the "little shop of horrors". When you want to pay such a low wage to a high-tech job, you show where your priorities are. Of course you are going to get 'BOZO'S' working for you. It's the universal law. If you value what I can do for you, pay me accordingly, otherwise suffer the consequences

I've had several friends and associates that did work for IT in the Kansas government. Oh, the horror stories they would tell. I thanked God every day I didn't work for that numb nuts operation that is the state of Kansas!

0

bobberboy 3 years, 8 months ago

if the state of kansas would spend some money and hire IT people with some brains this would'nt have happened. You get what you pay for - and the state of kansas is always first in line for cheaping out. And so in the end it always ends up costing them more. Dumb is as dumb does.

0

average 3 years, 8 months ago

Of the 400 metro areas tracked by the BLS, wages for sysadmins/networking/databases in Topeka are in the lowest 25% and those for Lawrence are in the lowest 10% in the nation (the very lowest outside Puerto Rico in some categories). Not that it's an excuse, but anyone who gets very good at IT gets the hell out, at least to Kansas City.

You get what you pay for.

0

toe 3 years, 8 months ago

Tax money is rarely spent well. This is another good example. These systems should have been well designed and paid for a long time ago. But, contracts are made on connections not ability.

0

Flap Doodle 3 years, 8 months ago

I blame Commie Fifth Columnists.

0

Keith Richards 3 years, 8 months ago

25 terrabyte raid rack servers cost around 25K each. Buy 4 for redudancy, 1 production, 1 mirrored test, 2 backups, at least one of which is in a secure alternate location.

the 6 P's-Prior Planning Prevents Piss Poor Peformance. Who was running the Network Admin and IT positions at the KDHE? Someone deserves to be fired over this.

0

mdrndgtl 3 years, 8 months ago

Fire someone, there was zero redundancy in this case.

0

TrooGrit 3 years, 8 months ago

After spending the better part of 30 years in the IT community, this is an epic fail! Anyone worth their salt will ensure that proper back-ups are made, such that a hardware failure becomes an insignificant event. At my place of business, we took back ups that were kept both in house, and at a remote center, every day. This way, if our data center were hit with a bomb, we'd still have data that was almost up to date. It sounds to be like they ran a very shoddy operation, and I tend to agree, Bremby needs to be out!

0

LJ Whirled 3 years, 8 months ago

Bremby needed to be doing his job, instead of playing politics with Sebelius and inventing imaginary laws to kill the economy.

Bremby = Failure. Time to resign before Gov Sam fires you.

0

ljwanonymous 3 years, 8 months ago

Based on my experience as network administrator and IT professional, sounds like a very poorly designed system. Have they never heard of redundancy?

0

das 3 years, 8 months ago

FAIL = The Glorious lack of success..... or "never events" [my addition]

-Urban Dictionary (with help)

0

Commenting has been disabled for this item.