Former President Jimmy Carter is flanked by his wife, Rosalynn, as he speaks Tuesday during “Conversations at the Carter Center” in Atlanta.

Atlanta ? Former President Jimmy Carter said Tuesday that U.S. Rep. Joe Wilson’s outburst to President Barack Obama during a speech to Congress last week was an act “based on racism” and rooted in fears of a black president.

“I think it’s based on racism,” Carter said at a town hall held at his presidential center in Atlanta. “There is an inherent feeling among many in this country that an African-American should not be president.”

The Georgia Democrat said the outburst was a part of a disturbing trend directed at the president that has included demonstrators equating Obama to Nazi leaders.

“Those kind of things are not just casual outcomes of a sincere debate on whether we should have a national program on health care,” he said. “It’s deeper than that.”

Wilson, a South Carolina Republican, was formally rebuked Tuesday in a House vote for shouting “You lie!” during Obama’s speech to Congress last Wednesday.

The shout came after the president commented that illegal aliens would be ineligible for federal subsidies to buy health insurance. Republicans expressed their disbelief with sounds of disapproval, punctuated by Wilson’s outburst.

Tuesday’s rebuke was a rare resolution of disapproval pushed through by Democrats who insisted that Wilson had violated basic rules of decorum and civility. Republicans characterized the measure as a witch hunt and Wilson, who had already apologized to Obama, insisted he owed the House no apology.

Wilson’s spokesman was not immediately available for comment, but his eldest son defended his father.

“There is not a racist bone in my dad’s body,” said Alan Wilson, an Iraq veteran who is running for state attorney general. “He doesn’t even laugh at distasteful jokes. I won’t comment on former President Carter, because I don’t know President Carter. But I know my dad, and it’s just not in him.

“It’s unfortunate people make that jump. People can disagree — and appropriately disagree — on issues of substance, but when they make the jump to race it’s absolutely ludicrous. My brothers and I were raised by our parents to respect everyone regardless of background or race.”