Kabkabiya, Sudan ? Former President Jimmy Carter confronted Sudanese security services on a visit to Darfur on Wednesday, shouting “You don’t have the power to stop me!” at some who blocked him from meeting refugees of the conflict.

The 83-year-old Carter, in Darfur as part of a delegation of respected international figures known as “The Elders,” wanted to visit a refugee camp. But the U.N. mission in Sudan deemed that too dangerous.

Instead, Carter agreed to fly to the World Food Program compound in the North Darfur town of Kabkabiya, where he was supposed to meet with ethnic African refugees, many of whom were chased from their homes by militias and the Arab-dominated government’s forces.

But none of the refugees showed up and Carter decided to walk into the town – a volatile stronghold of the pro-government janjaweed militia – to meet refugees too frightened to attend the meeting at the compound.

He was able to make it to a school where he met with one tribal representative and was preparing to go farther into town when Sudanese security officers stopped him.

“You can’t go,” the local chief of the feared Sudanese secret police, who only gave his first name as Omar, ordered Carter. “It’s not on the program!”

“We’re going to anyway!” an angry Carter retorted as a small crowd began to gather around. “You don’t have the power to stop me.”

However, U.N. officials told Carter’s entourage the powerful Sudanese state police could bar his way.

“We’ve got to move, or someone is going to get shot,” warned one of the U.N. staff accompanying the delegation.

Carter’s traveling companions, billionaire businessman Richard Branson and Graca Machel, the wife of former South African President Nelson Mandela, tried to ease his frustration and his Secret Service detail urged him to get into a car and leave.

“I’ll tell President Bashir about this,” Carter said, referring to Sudanese President Omar al-Bashir.

Omar, the security chief, said Carter had already breached security once by walking to the school and would not be allowed to breach it again.

“We are in the security field. We’re not that flexible,” he said after the confrontation ended.

In an interview with The Associated Press later in the day, Carter played down the encounter, saying the security chief was only doing his job.

“But it’s true that I’m not accustomed to people telling me I can’t walk down the street and meet people,” he told the AP after returning to a United Nations compound in El Fasher, the capital of North Darfur state.