The U.S. Secret Service says 11 of its agents have been put on leave as it investigates their conduct in Colombia ahead of President Barack Obama's arrival there for the Summit of the Americas.

Five members of the U.S. military, posted to the Secret Service, have also been pulled from duty in Cartagena, Colombia, where Obama arrived Friday.

The Secret Service agents have returned to the United States, while the military personnel are still in Colombia but confined to quarters.

All are being investigated for an incident that occurred overnight Wednesday to Thursday and involved a woman who demanded money after having a "sexual encounter" with at least one agent, according to the Washington Post.

At least one agent sent ahead of President Barack Obama's trip to Cartagena, Colombia is being accused of being with prostitutes. (Dolores Ochoa/Associated Press)

A disturbance ensued at Cartagena's Hotel Caribe, where the Secret Service staff had been staying while they undertook security preparations for Obama's visit. Several members of the White House staff and press corps were also staying at the hotel.

Colombian police were called, and authorities alerted the U.S. Embassy.

Prostitution allegations

A hotel employee, speaking on condition of anonymity for fear of losing his job, said the agents arrived at the beachfront hotel about a week ago. The employee described the agents as drinking heavily during their stay. Representative Peter King, chairman of the U.S. House's homeland security committee, said after he was briefed on the investigation on Saturday that "close to" all 11 of the Secret Service agents had brought women back to their rooms.

Jon Adler, president of the U.S. Federal Law Enforcement Officers Association, which represents federal law enforcement officers, including the Secret Service, said he had heard that there were allegations of prostitution but had no specific knowledge of any wrongdoing.

White House spokesman Jay Carney said that Obama retains confidence in the Secret Service and that the incidents under investigation had no impact on presidential security.

"These personnel changes will not affect the comprehensive security plan that has been prepared in advance of the president's trip," Secret Service spokesman Ed Donovan said.

Still, the allegations were an embarrassment for the president and his delegation while guests of the Colombian government. And the incident threatened to torpedo White House efforts to keep the president's trip focused squarely on the economy and boosting U.S. trade ties with fast-growing Latin America.

Leaders from 34 Western Hemisphere countries, including Canadian Prime Minister Stephen Harper, are in Colombia for the summit.

With files from The Associated Press