Commentary: "From Spin City to Fat City"

Daniel Henninger's May 6 Wall Street Journal column (link above for WSJ subscribers) discusses the peculiar saga of how the science of obesity was drowned in politics -- and by the scientists themselves. He writes:

This is a case study in how public policy gets formulated in a highly advanced, highly educated and not least, highly neurotic society.

It is also an example of how peer review is an insufficient tool for discerning the quality of scientific information that will be used to set public policy.

FEDERAL PEER REVIEW

Regulatory Checkbook has followed the recent history of obesity politics in the context of the Office of Management Budget’s recent efforts to improve the quality and quantity of scientific peer review performed by federal agencies. OMB’'s authority rests in the Information Quality Act, which authorized it to issue government-wide guidelines requiring federal agencies to meet certain minimum standards for all information they disseminate, and rather stringent standards for information deemed “influential.”

OMB issued a proposed Bulletin on Peer Review in August 2003. Peer review is the standard tool used by scholars to rank research proposals for funding and allocate scarce pages in scholarly journals. Dr. John D. Graham, Administrator of OMB’s Office of Information and Regulatory Affairs, is himself a former professor and widely known to believe that peer review is an effective tool for discerning quality. Thus it was not a surprise that he would propose this Bulletin to improve the quality and effectiveness of so-called “pre-dissemination review” procedures which OMB’s Information Quality Guidelines had required every agency to adopt.

Many agencies had only pretended to adopt effective pre-dissemination procedures. Often they merely asserted that their existing internal systemswere adequate to ensure that inaccurate or biased information was not disseminated in the first place. Were that so, of course, Congress would not have enacted the law and OMB’s Information Quality Guidelines would have been superfluous.

Regulatory Checkbook provided public comments on the proposed Bulletin, as did 186 others — an extraordinary amount of interest in an obscure guidance document proposed by an agency that is equally obscure outside the Beltway. In our comments we discussed a number of legal, philosophical and technical issues raised by the proposed Bulletin. In general, we found the proposed language extremely helpful – especially OMB’s desire that scientific peer review stick to scientific issues and refrain from trying to resolve policy disputes under the guise of “science”.

We were concerned, however, that the proposed language permitted agencies too much discretion and control over their peer review programs, which to be effective needed to be genuinely external and independent, and rigorous in the intensity of effort:

OMB's language seems inadequate, however, to deal with agencies and institutions that have active peer review programs but routinely ask reviewers to address policy matters or impose policy-driven constraints on scientific review. An obvious and commonplace example is an agency request that peer reviewers opine as to whether the agency’s interpretation of the science is reasonable given a litany of so-called science policy defaults. As indicated below, one way to assist these agencies in making their peer review practices more compliant with the policy-neutrality criterion is to modify the Bulletin such that persuasive evidence of noncompliance is sufficient grounds for challenging the adequacyof pre-dissemination peer review. (p. 2)

Under pressure from opponents, OMB relented and re-proposed a substantially weaker version in April 2004 and belatedly agreed to accept another round of public comments. This time, 57
more comments were submitted including Regulatory Checkbook’s comments. As expected, those who thought the original draft was too demanding and burdensome liked the weaker revised version; those like Regulatory Checkbook who thought the original version ought to have been more stringent were deeply disappointed; and those who opposed the entire enterprise continued to vigorously do so. OMB issued a final Bulletin in December 2004 that closely tracked the weak April 2004 version.

THE CASE OF OBESITY

Because of our concern that weak peer review requirements would not achieve OMB’s legitimate objectives, Regulatory Checkbook has been on the lookout for examples that would illustrate our point, if not completely prove it. In November 2004 a spate of news accounts appeared suggesting that peer review had failed, and failed rather badly, with respect to the scientific issue of the extent to which obesity resulted in reduced life expectancy. Most disconcertingly,this failure occurred within the most highly respected federal agency:

We summarized our concerns in a letter to OMB dated December 16, 2004 (copies available on request).

The recent incident in question involves certain studies disseminated bythe Centers for Disease Control (CDC) concerning the relative mortality risks posed by obesity. These studies are clearly influential, as they are intended to reallocate the funding, research and public health priorities of the federal government and to significantly alter the behavior of individuals, households, firms and other institutions. Based on publicly available information and news reports, which of course are necessarily incomplete and may not be entirely accurate, it appears that CDC’s extensive pre-dissemination review procedures failed even though vigorous peer review is a core feature of these procedures and its commitment to peer review is perhaps the strongest among all agenciesof the federal government.

In addition to CDC’s failure, OMB’s reliance on peer review by scholarly journals proved to fail as well. One of the world’s most highly respected medical journals failed to perform the quality assurance role that Dr. Grahamexpected of it. We wrote:

This incident also seriously undermines OMB's long-standing deference to the presumptive objectivity of scientific information published in peer reviewed journals. The critical scientific paper supporting policy and programmatic positions of both CDC and its parent Department of Health and Human Services was peer reviewed and published by the Journal of the American Medical Association (JAMA), one of the top medical journals in the world. Many peer reviewed journals have missions that are ideological or otherwise contrary to the statutory information quality standard of objectivity such that affected persons could be able to rebut, with little difficulty, the presumption that peer review by these journals was justified. However, there are a few scientific and professional journals whose peer review procedures would have been expected to easily fulfill OMB’s expectations as an effective non-governmental arbiter of objectivity such that affected persons would have an extremely difficult job mounting a successful rebuttal. JAMA is clearly one such journal, but it appears that its peer reviewers and editors were unable to detect the errors discovered and apparently acknowledged by CDC. Indeed, the very mechanism by which these errors were detected and publicized suggests that peer review can inhibit or prevent, rather than enhance, the achievement of a high level of informationquality in federal information dissemination. (p. 3)

* Mokdad AH, Marks JS, Stroup DF, Gerberding JL (2004). "Actual Causes of Death in the United States, 2000," 291:JAMA 1238-1245.

Details of the obesity controversy are found in Regulatory Checkbook's letter:

• CDC researchers reviewed existing epidemiological studies and mortality data reported to CDC for the year 2000, and concluded that 400,000 fatalities per year were attributable to obesity, an increase of one-third since 1990and less than 10 percent below the number attributable to tobacco.
• CDC scientists suggesting that obesity is an “epidemic” public health problem; after it was published by JAMA, CDC promoted the study and offeredspecific policy conclusions based on it.
• CDC Director Julie Gerberding was a co-author of this study and Department of Health and Human Services Secretary Tommy Thompson was a strong advocatefor policies combating obesity.
• the study (but allied with CDC's anti-tobacco camp) gave the story to a Wall Street Journal reporter. They implied that CDC's peer review procedures were skewed because of Dr. Gerberding's prominence, or because of a strong, high-level policy-level commitment to elevate the relative importance of obesity as apublic health hazard.
  
• According to press accounts in November 2004, CDC and its researcher-authors admitted to having overstated the number of annual fatalities from obesity by at least 80,000 cases. Instead of a 33% increase in deaths from obesity between 1990 and 2000, the ten-year increase now appears to be no greaterthan 7% before increases in the U.S. population were taken into account.
  
• Scientists both within and outside CDC said that the methods the authors used to estimate deaths from obesity were upwardly biased when compared to the methods used to estimate deaths from tobacco, did not account for genetic factors, and thus implied that deaths from overweight and obesity are behavioral. Other scientists said they were “puzzled” by the authors’ statistical methods.
  

On May 6 Science (access by subscription) published a news article titled "A Heavyweight Battle over CDC?'s Obesity Forecasts." The article discusses the 2004 paper that started the controversy (400,000 premature deaths per year); the 2004 article authors?' formal revised estimate in response to the controversy (365,000), less than the 80,000 overestimate hinted at in November 2004; and a new estimate by CDC epidemiologist Katherine Flegal and colleagues from the National Cancer Institute (112,000) published April 20 by JAMA. The importance of this new estimate is highlighted in an editorial by David Mark accompanying the Flegal et al. paper:

The study by Flegal et al is likely to generate interest because it provides an estimate for deaths attributable to obesity that appears to strongly contradict prior estimates published in JAMA. Flegal et al estimate that there were about 112 000 obesity-attributable deaths in the United States in 2000, far lower than the 414 000 estimated by Mokdad et al for the same year and the 280 000 estimated by Allison et al. for 1991. The magnitude of the differences cries out for explanation of the reasons behind these differences. Some might wonder: If well-intentioned efforts to calculate this number can result in such widelyvarying estimates, is it worth trying to do at all?

One possible explanation, which Dr. Mark does not appear to have even contemplated, is that the demands of public policy advocacy on behalf of obesity prevention interfere with scientific objectivity in the estimation of the magnitude of the problem. If Flegal and colleagues are right, no obesity epidemic exists.

The absence of an epidemic could be politically problematic. A number of high-profile individuals and institutions, in addition to CDC's Dr. Gerberding and DHHS' Secretary Thompson, have hitched themselves to the fact that an obesity epidemic exists. For example, California Governor Schwarzenegger is promoting an Obesity Initiative that followed the Governor's Summit on the Prevention of Obesity. As the website promoting the Summit states:

Governor Schwarzenegger understands that obesity has become a nationwide epidemic and that California has not been spared.

The obesity case offers important lessons for those who believe peer review is The Solution to the problem of ensuring that scientific information disseminated by federal agencies meets established standards for utility, integrityand objectivity:

• However rigorous CDC's or DHHS' internal peer review procedures might be in general, they could not be reasonably be expected to be effective in any case where the authors are prominent or the science an integral part ofthe agency's public policy advocacy. The problem is vastly greater for other federal agencies lacking CDC's stellar reputation for rigorous peer review.


• Conventional methods of peer review failed to ensure that OMB's quality standards were met. The usual model for peer review is to appoint an ostensibly unbiased panel of distinguished scientists. That is presumably what both CDC and JAMA did, but to no avail. Independent experts who volunteer their services have limited interest in detecting error, yet that is what OMB demands peer review to accomplish. In this case, it was a highly motivated and scientifically competent scientists employed by an internal agency competitor (CDC's Office on Smoking and Health) who originally detected the error and forced both CDC and the authors to come clean. Even now, both CDC and Mokdad and colleagueshave offered only the barest of admissions.


• Relying on peer review unwittingly retards advancements in information quality -- a result exactly opposite of OMB's intent. OMB's Peer Review Bulletin confers an undeserved presumption of objectivity to scientific information that has been internally peer reviewed and published in a scholarly journal, especially a journal such as JAMA. Once this presumption of objectivity hasbeen obtained it is virtually impossible to rebut.

We will post again on the specific issues raised by the news article at a laterdate. For now, the problem remains that current peer review procedures, particularlythe fatally flawed version adopted as federal policy by OMB, are wholly inadequateto the task. Once science has become a weapon in the pursuit of public policyagendas, it is intensely vulnerable to various forms of corruption sometimesobvious but usually subtle. Conventional peer review methods practiced in academiamay be helpful, but they cannot be relied upon to discern objective scientificresearch from that which is driven by political or policy objectives.