CARTAGENA, Colombia — An embarrassing scandal involving prostitutes and Secret Service agents deepened Saturday as 11 agents were placed on leave, and the agency designed to protect President Barack Obama had to offer regret for the mess overshadowing his diplomatic mission to Latin America.

The controversy also expanded to the U.S. military, which announced five service members staying at the same hotel as the agents in Colombia may have been involved in misconduct as well. They were confined to their quarters in Colombia and ordered not to have contact with others.

All the alleged activities took place before Obama arrived Friday in this Colombian port city for meetings with 33 other regional leaders.

Put together, the allegations were an embarrassment for an American president on foreign soil and threatened to upend White House efforts to keep his trip focused squarely on boosting economic ties with fast-growing Latin America. Obama was holding two days of meetings at the Summit of the Americas with leaders from across the vast region before heading back to Washington Sunday night.

The Secret Service did not disclose the nature of the misconduct. The Associated Press confirmed on Friday that it involved prostitutes.

The White House said Obama had been briefed about the incidents but would not comment on his reaction.

“The president does have full confidence in the United States Secret Service,” presidential spokesman Jay Carney said when asked.

Carney insisted the matter was more a distraction for the media than Obama. But Secret Service assistant director Paul Morrissey said in a statement: “We regret any distraction from the Summit of the Americas this situation has caused.”

Rep. Peter King, chairman of the House Homeland Security Committee, told the AP after he was briefed on the investigation on Saturday that “close to” all 11 of the agents involved had brought women back to their rooms at a hotel separate from where Obama is now staying.

The New York Republican said the women were “presumed to be prostitutes” but investigators were interviewing the agents.

The lawmaker also offered new details about the controversy.

King said he was told that anyone visiting the hotel overnight was required to leave identification at the front desk and leave the hotel by 7 a.m. When a woman failed to do so, it raised questions among hotel staff and police, who investigated. They found the woman with the agent in the hotel room and a dispute arose over whether the agent should have paid her.

King said he was told that the agent did eventually pay the woman.

The incident was reported to the U.S. embassy, prompting further investigation, King said

The 11 employees in question were special agents and Uniformed Division Officers. None were assigned to directly protect Obama. All were sent home and replaced, Morrissey said, given “the nature of the allegations” and a zero tolerance policy on personal misconduct.

The Secret Service says the incidents have had no bearing on its ability to provide security for Obama’s stay in Colombia.

Another jolt came Saturday when the U.S. Southern Command said five service members assigned to support the Secret Service violated their curfew and may have been involved in inappropriate conduct. Carney said it was part of the same incident involving the Secret Service.

As for the apparent misconduct by the military members, Gen. Douglas Fraser, commander of U.S. Southern Command, said he was “disappointed by the entire incident” and said the behavior was “not in keeping with the professional standards expected of members of the United States military.”

Col. Scott Malcom, chief of public affairs for Southern Command, said of the five service members: “The only misconduct I can confirm is that they were violating the curfew established. He said he had seen the news reports about the Secret Service agents involved in alleged prostitution at the hotel but could not confirm whether the service members also were involved.

The military is investigating.

The Secret Service agents at the center of the allegations had stayed at Cartagena’s five-star Hotel Caribe. Several members of the White House staff and press corps subsequently stayed at the hotel.

King credited the Secret Service director for acting quickly to remove the agents in question and replace them before Obama’s arrival.

A hotel employee, speaking on the condition of anonymity for fear of losing his job, said the agents arrived at the beachfront hotel about a week ago and said the agents left the hotel Thursday, a day before Obama and other regional leaders arrived for the weekend summit.

Three waiters interviewed by the AP at the hotel described the agents as drinking heavily during their stay.

On Friday, the hotel began filling up with the delegations of some of the more than 30 countries whose leaders are convening for the weekend Summit of the Americas.

The hotel’s public relations director, Ana Beatriz Angel, refused to comment on the incident, which she said “concerns only and exclusively the U.S. government.”

On the steamy streets of Cartagena, a resort city with a teeming prostitution trade, there was condemnation for the Secret Service agents for what residents saw as abusing their station and dishonoring their country.

Edwin Yepes, a souvenir vendor, said “they are supposed to come here and set an example. We are an inferior culture, and so it’s better if they don’t come than if they damage our image of them.”