Republican hopeful Newt Gingrich, under pressure from his party’s establishment, pulled a Spanish-language ad in which he had accused his rival Gov. Mitt Romney of being “anti-immigrant.” But was the ad really unfair?

The question will not go away, and will haunt Republicans for the remainder of the race if Romney wins the Republican nomination. President Barack Obama’s campaign will surely make the most of it.

Gingrich yanked the ad, which claimed that Romney is “the most anti-immigration candidate,” after conservative Hispanic Sen. Marco Rubio complained that it was “inaccurate” and “inflammatory.” Gingrich said he was withdrawing that ad out of respect for the Florida Senator, but did not retract from the its content in later interviews and public debates leading to Tuesday’s Florida Republican primary.

Hours later, in Thursday’s CNN debate, Romney responded that the ad was “simply inexcusable,” and defined himself as a “pro-legal immigration” candidate.

So who is right? The fact is that both Romney and Gingrich have used a hard-line rhetoric against immigrants in the Iowa and South Carolina primaries, and have softened their rhetoric somewhat in recent days as their campaigns shifted to Florida, where 13 percent of voters are Hispanic. But pro-immigration advocates say Romney has taken the most extreme positions on immigration.

Consider:

l On deportations: Romney said at the Dec. 10, 2011, debate in Iowa that the estimated 11 million undocumented people living in the country should be given a “transition period” to “settle their affairs and then return home.” He later described it as a “self-deportation” plan.

Gingrich responded that he would allow people who have lived here for 25 years and have strong ties to the community to apply for U.S. residency, although not for citizenship. That’s more realistic, and more “humane,” he said.

l On Arizona, Alabama and South Carolina’s harsh anti-immigration laws, which allow local police to ask people for their immigration papers under certain circumstances, both Romney and Gingrich have embraced them. But Romney on Jan. 11 got the endorsement of Kansas Secretary of State Kris Kobach, the legal architect of Arizona-styled anti-immigration laws, and called him “a true leader on securing our borders.”

l On undocumented college students: Romney and Gingrich have opposed the Dream Act — a bill that would give legal status to soldiers and college students who were brought to this country as children by no fault of their own — and said they would veto it. Both now say they would make an exception with those who serve in the military.

Most pro-immigration advocates, while disliking both Republican front-runners, say they dislike Romney the most.

“Romney’s self-deportation strategy is an ugly plan aimed at harassing undocumented immigrants until they flee the country,” says Frank Sharry, head of the America’s Voice advocacy group. “It leads to the institutionalized abuse of hardworking immigrants, and to widespread discrimination against Latinos”

My opinion: Romney, and to a lesser degree Gingrich, are badly hurting the Republican Party’s chances in November by alienating Hispanic voters. They come across as pandering to xenophobes by playing up the anti-immigration theme even though illegal immigration has slowed dramatically since the 2008 U.S. recession.

And Romney’s argument that he is not “anti-immigration,” but “pro-legal immigration,” is deceiving. There is no realistic way of achieving “self-deportation” of 11 million people without turning America into a police state.

And there will be no realistic way of stopping the flow of undocumented immigrants unless the U.S. visa system is expanded to accommodate people who are being hired by U.S. employers to fill jobs that Americans won’t do. It now takes at least 18 years for a Mexican migrant, and up to 70 years — yes, you read right — for a highly skilled Indian immigrant to get a U.S. visa, immigration advocacy groups say.

I know, every time I write this I get swamped with angry tweets from furious conservatives who call me a liberal “open-borders” advocate. They should not listen to me, but to former Florida Gov. Jeb Bush, who cannot be accused of being a Commie. In a Jan. 25 interview with CNN, Bush regretted “the tone” of Republicans’ rhetoric on immigration, adding that “I don’t think a party can aspire to be the majority party if it’s the old white guy party.”

He is right. I would have only added that, especially in Romney’s case, what’s offensive to most Hispanics is not just his tone, but his anti-immigrant message. Unless he changes it, Gingrich’s ad cannot be called unfair.