As prostitution allegations engulf more of the agents tasked with his security, U.S. President Barack Obama addressed the scandal for the first time Sunday, saying that "of course I'll be angry" if the reports about the Secret Service are proven true.

Obama said the agents represent the United States and are supposed to conduct themselves with the highest levels of dignity anywhere in the world. "Obviously, what's been reported doesn't match up to those standards," Obama said at a news conference wrapping up his appearance at the Summit of the Americas in Colombia.

U.S. President Barack Obama, left, had to address the scandal engulfing the Secret Service during a joint news conference with Colombian President Juan Manuel Santos on Sunday. (Kevin Lamarque /Reuters)

The president never directly mentioned the specific accusations, confirmed by media outlets, that Secret Service agents were partying with sex workers in Colombia last week, before the president arrived there.

The Secret Service sent 11 agents home and placed them on misconduct leave as it reviews what happened. Five members of the U.S. military, seconded to the Secret Service, have also been pulled from duty in Cartagena, Colombia, where Obama arrived Friday.

"I expect that investigation to be thorough, and I expect it to be rigorous," Obama said. "If it turns out that some of the allegations that have been made in the press are confirmed, then of course I'll be angry. ... We are representing the people of the United States, and when we travel to another country, I expect us to observe the highest standards."

Sex work is legal in parts of Colombia, but the unseemly topic still dogged Obama to the end there, as he fought to keep a focus on America's trade relations with partners throughout the Americas. When Obama took a question about the topic, he was standing next to Colombian President Juan Manuel Santos.

Obama tried to put the controversy in context by lauding the agents who protect his life.

"These men and women perform extraordinary service on a day-to-day basis protecting me, my family, U.S. officials," Obama said. "They do very hard work under stressful circumstances and almost invariably do an outstanding job. So I'm very grateful."

He said he would reserve judgment until the investigation is done.

Blackmail concerns

A leading Republican member of the U.S. House said Sunday that the Secret Service agents' alleged behaviour probably wasn't an isolated incident.

California Representative Darrell Issa, chairman of the House's oversight and government reform committee, said he wasn't certain whether Congress would hold hearings on the misconduct. But legislators will be looking "over the shoulder" of the Secret Service, he said, to make sure that the agency's method for training and screening agents isn't endangering American VIPs.

"Things like this don't happen once if they didn't happen before," Issa said.

Issa told TV network CBS's Face the Nation that legislators are still trying to confirm the precise number of U.S. personnel involved in the scandal. He said the number could be higher than initially thought.

Of primary concern, he said, was that agents who behave badly are vulnerable to blackmail and therefore risk the security of the president and others under their protection.

Leaders from 34 Western Hemisphere countries, including Canadian Prime Minister Stephen Harper, completed their weekend summit on Sunday in Colombia. The meetings finished without a closing declaration because of American and Canadian refusal to allow Cuba to attend the next Summit of the Americas, in Panama in 2015. The summit also saw Latin American countries that favour the possible decriminalization of drugs butt up against Canada and the U.S., which have ruled out such a move.

With files from CBC News