When developing "evidence-based" guidelines, the World Health Organization routinely forgets one key ingredient: evidence. That's the verdict from a study published in the Lancet online Tuesday.

The medical journal's criticism of WHO will shock many in the global health community, as one of WHO's main jobs is to produce guidelines on everything from fighting the spread of bird flu and malaria control to enacting anti-tobacco legislation. 

'This is a pretty seismic event. It undermines the very purpose of WHO.'—Lancet editor Dr. Richard Horton 

"This is a pretty seismic event," Lancet editor Dr. Richard Horton, who was not involved in the research for the article. "It undermines the very purpose of WHO."

The study was conducted by Dr. Andrew Oxman and Dr. Atle Fretheim, of the Norwegian Knowledge Centre for Health Services, and Dr. John Lavis at McMaster University in Hamilton. They interviewed senior WHO officials and analyzed various guidelines to determine how they were produced. What they found was a distinctly non-transparent process.

"It's difficult to judge how much confidence you can have in WHO guidelines if you're not told how they were developed," Oxman said. "In that case, you're left with blind trust."

WHO issues about 200 sets of recommendations every year, acting as a public health arbiter to the global community by sifting through competing scientific theories and studies to put forth the best policies.

'Going to get our act together'

WHO's director of research policy Dr. Tikki Pang said that some of his WHO colleagues were shocked by the Lancet's study, but he acknowledged the criticism had merit, and explained that time pressures and a lack of both information and money sometimes compromised WHO work.

"We know our credibility is at stake," Pang said, "and we are now going to get our act together."

WHO officials also noted that, in many cases, data from developing countries is patchy at best, and in an outbreak, information changes as the crisis unfolds.

To address the problem, they said, WHO is trying to develop new ways to collect information in poor regions, and has proposed establishing a committee to oversee the issuance of all health guidelines.

The Lancet study - conducted in 2003-04 through analyzing WHO guidelines and questioning WHO officials ? also found that the officials themselves were concerned about the agency's methods.

One unnamed WHO director was quoted in the study as saying: "I would have liked to have had more evidence to base recommendations on." Another said: "We never had the evidence base well-documented."

Pang said that, while some guidelines might be suspect and based on just a few expert opinions, others were developed under rigorous study and so were more reliable. For example, WHO's recent advice on treating bird flu patients was developed under tight scrutiny.

Oxman also noted that WHO had its own quality-control process. When its 1999 guidelines for treating high blood pressure were criticized for, among other things, recommending expensive drugs over cheaper options without proven benefit, the agency issued its "guidelines for writing guidelines," which led to a revision of its advice on hypertension.

"People are well-intended at WHO," Oxman said. "The problem is that good intentions and plausible theories aren't sufficient."

It remains to be seen how the organization's 193 member countries will react to the Lancet study, released just before WHO's governing body, the World Health Assembly,  meets next week at the UN.