The top U.S. envoy to Pakistan and Afghanistan Richard Holbrooke, right, shakes hands with a displaced man during his visit at the Chota Lahore refugee camp, at Swabi, in northwest Pakistan on Thursday. (Mohammad Sajjad/Associated Press)

A top U.S. envoy went tent by tent to comfort refugees in Pakistan's northwest Thursday, telling them his country can offer them aid, but not security.

"It's up to the Pakistan army to give you security. That is not our job," said Richard Holbrooke in Mardan.

The Pakistani military's month-long operation in the Swat Valley and surrounding areas, which the U.S. supports, has displaced up to three million people.

About 200,000 of them are in tent camps, run by the Red Cross and the United Nations, with the vast majority of evacuees staying with friends and relatives.

The U.S. has already pledged $110 million in aid to the refugees, and Obama is pushing to send another $200 million.

Holbrooke asked refugees for their stories about the Taliban, though many refugees preferred to talk about their need for electric fans to cope with the sweltering heat and better food.

Though the U.S. has given a lot of support, "it is up to the United Nations and the Pakistan government to carry out the programs. We are not in the camps," Holbrooke replied.

Meanwhile, in a message purportedly from Osama bin Laden broadcast on Al-Jazeera Television on Wednesday, he blamed the U.S. for the refugee crisis.

Holbrooke rejected the claim at a press conference Wednesday in Islamabad.

"The idea that anyone is responsible for the refugee crisis other than al-Qaeda, and the Taliban and the other people that have caused such tragedy in Pakistan is ludicrous."

With files from The Associated Press