Reality is a hard nut to crack – it doesn’t give up its secrets easily. When scientists researching the Earth’s climate realized that humanity was the major driving force behind observed global heating, many people didn’t believe it. Some couldn’t accept that our collective activity was overwhelming natural forces like changes in solar irradiance and Milankovich cycles. Some wanted to wait until more and better data was available and a scientific consensus formed. Others simply didn’t care, either rejecting the science in an anti-intellectual paroxysm or concluding that global heating would never affect them or their families. But there were more than a few people who saw the science, understood it, and then set about deliberately undermining the science for political, religious, or financial reasons.

On November 8, 2007, the anti-global heating tactic of questioning the scientific consensus behind global heating got a fresh piece of “evidence”: DemandDebate.com issued a press release about a survey that purports to undermine the scientific consensus on global heating. DemandDebate.com claims to be “more worried about the intellectual climate” and has started up an educational video series called “We Debate, You Decide” that is supposed to provide actual debate on global heating. DemandDebate.com’s survey, however, is pure propaganda.

According to the professional organization American Association for Public Opinion Research (AAPOR), there are a few basic guidelines to ensure the accuracy of any public opinion poll: having specific goals, maximizing the response rate to the survey, taking great care in wording questions correctly for the survey’s population, and holding respondent’s identities as privileged, among others. In all of these areas, DemandDebate.com’s survey methodology fails.

When designing a survey, the designer is supposed to choose objectives that are “specific, clear-cut and unambiguous.” The goal of such surveys is to acquire statistically valid information about the population being surveyed, not to produce predetermined results. The stated goal of DemandDebate.com survey was to determine the level of consensus among the scientists involved in the drafting of the United Nations Intergovernmental Panel on Climate Change (IPCC) “Climate Change 2007: The Physical Basis” document. However, the discussion of the results do not present an unbaised appraisal of the survey data. Instead, the discussion is entirely biased against global heating:

But if there’s no agreement on what the target climate should be, what precisely is the point of taking action on global warming? What is the climatic goal at which we are aiming?

…So was there no climate change before mankind? And if there was natural climate change before man, why not now also?

…But when you ask questions that are off the usual script, the supposed consensus seems to readily fall apart.

And let’s not forget that many climate experts no longer participate in the IPCC process because they perceive it to be biased….

These statements do not just present the survey data, they provide readers the “correct” interpretation of that data, propose alternatives to the actual questions, imply nefarious motives to the scientists who answered the survey questions, and add information external to the survey sample itself in order to support the survey’s supposed conclusions. These actions introduce an obvious bias to both the results and the questions themselves, suggesting that this survey was not intended to be a valid statistical survey as claimed, but rather the equivalent of a political “push poll” for global heating.

Quality surveys attempt to maximize the rate of response to the survey, because more responses mean greater accuracy. However, in this case, DemandDebate.com did nothing to maximize the number of scientists who responded – in fact, DemandDebate.com intentionally ignored about 45% of the scientists involved in the IPCC and focused on the U.S.-based scientists only. If the intent had truly been to test the level of consensus among global heating scientists, the survey would have been sent to all 620 or so IPCC scientists, not just the 345 scientists who are based in the United States. This introduces a selection bias into the answers to the survey. Furthermore, the fact that only 54 of the U.S. scientists responded suggests that the surveyed population detected bias in the survey and simply refused to answer the questions posed. In fact, several of the scientists who refused to answer this survey blogged about it, and their refusal to participate, at RealClimate.org: The “Have you stopped beating your wife yet (yes/no)” questionnaire. The end result is that less than 10% of the possible survey respondents actually responded. As such, the results of the poll are strongly suspect due to self-selection bias (ie a non-representative sample) and qualify as unscientific and thus essentially meaningless SLOP (Self-selection Opinion Poll).

Which brings me to the questions themselves. According to S&R’s resident marketing expert, Dr. Slammy, there are at least two questions (of the six total) that are biased and specifically designed to produce results that can be spun by DemandDebate.com and Mr. Milloy for their own political ends. Lets look at questions 5 and 6 (results removed from questions):

Question #5. The climatic impacts of a mean global temperature that is 1-degree Celsius warmer than today are:
[ ] Undesirable.
[ ] Desirable.
[ ] Desirable for some and undesirable for others.
[ ] Too difficult to assess.
[ ] No opinion.

Question #6. The ideal global climate is…
[ ] Warmer than the present.
[ ] Cooler than the present.
[ ] Occurring today.
[ ] There is no such thing as an “ideal” global climate.
[ ] No opinion.

The selections for question #5 are specifically designed to use scientific nuance and vagueness to get an answer that can be spun the way the survey designer wants. “Desirable for some and undesirable for others” is undoubtedly the “correct” answer, but it’s the correct answer for nearly every temperature increase or decrease of any magnitude – some areas will always get “better” and some will get “worse.” But better or worse, desirable or undesirable for whom, or for what? People? Other species? Coastline erosion? Sea level rise? And the temperature rise, 1 degree C, was carefully chosen to produce the most easily manipulated results. After all, The Pew Center on Global Climate Change said that low amounts of temperature rise would probably benefit the United States, but that higher temperatures (like those in the 3-5 degree C range that the IPCC actually predicts by 2100) will be bad for the U.S. These levels of nuance are get lost because the question is so poorly designed. At least, poorly designed for producing valid statistical data.

Similarly, question #6 is intended to get a predetermined answer, namely “there is no such thing as an ideal climate.” Not only is this the only scientifically viable answer – and thus the answer that most good scientists will give – but the answer is designed to prejudice survey readers against scientists for whom the word “ideal” is nearly meaningless. For example, “ideal” for people (the likely interpretation for most survey readers) may be different from ideal for lions, tigers, and bears. In addition, there may not actually be an “ideal” climate for human beings – something that is unknown and may be fundamentally unknowable. However, because the scientifically valid answer is the one that looks the worst for the scientific consensus, this question is both great propaganda and a no-win for any scientist who responded to this survey. And in fact, both the press release and the actual survey used this question as a jumping-off point for a biased, explicitly political message, namely “But if there’s no agreement on what the target climate should be, what precisely is the point of taking action on global warming? What is the climatic goal at which we are aiming?” So, have you stopped beating your wife yet?

For a detailed look at how good survey questions should look, I recommend The AAPOR’s question wording and SurveyPro’s survey design tips.

However, perhaps the greatest methodological error, if we can call attempted character assassination an “error,” is the fact that Mr. Milloy failed to hold the identity of the respondents private. In fact, in his Fox News commentary on the survey’s results, Mr. Milloy threw out these gems:

One National Oceanic and Atmospheric Administration scientist responded simply by dropping an f-bomb-laced insult into an e-mail.

This particular response and any institutional intolerance for climate skepticism, so I am informed, is being investigated by NOAA chief Vice Admiral (Ret.) Conrad Lautenbacher….

…[It] just so happens that [IPCC scientist John] Christy’s survey responses were within the 50 percent who didn’t think that a 1-degree Celsius rise in global temperature was uniformly undesirable and the 86 percent who didn’t think there was any such thing as an ideal climate.

I’m not going to support expletive-laden insults from NOAA scientists, but the fact that Mr. Milloy reported the incident for an official investigation instead of holding the information confidential (as expected of professional public opinion researchers) illustrates that his goal throughout the entire survey process was to manipulate the U.S.-based IPCC scientists into appearing to be less unified on global heating than they actually are. And his quoting the exact answer of a scientist is also professionally unethical for a public opinion researcher (again according to AAPOR’s best practices regarding confidentiality).

Mr. Milloy was contacted for this article, but did not respond to a request for an interview.

Even without the severe problems in the methodology, there are significant issues with the data itself. As I pointed out above, there are approximately 620 scientists who worked on the IPCC Physical Basis for Climate Change document, either as principal writers or reviewers. However, only the 345 scientists in the United States were surveyed, and of them only 54 actually took the survey and returned it. And yet, given that this survey purports to be a serious, statistically-valid survey, there is no mention of the confidence level, nor is there any information about the margin of error for this survey. Given the strong selection biases in the sample, we actually cannot draw any statistically valid conclusions from this survey. But if we could, and we applied the industry standard 95% confidence level to this survey, we find that 54 respondents produces a margin of error of +/- 12.75% (found using this online calculator). Given that the target margin of error for meaningful political polls is less than 3%, this is a huge margin of error, and it effectively erases the differences between the questions when the differences are less than approximately 13%.

Let’s look at all the questions again to determine what the statistically meaningful (separated from other answers by greater than 13%) answers are.

For question #1 (What best describes the reason(s) for climate change?), the only statistically meaningful answer becomes “Human activity drives climate change, but natural variability is also important”, at 63% of respondents. All the other answers are within 13% of each other and so are statistically meaningless when applied to the larger, 620 member population of scientists.

The only answer to question #2 (Which best describes the role of manmade CO₂ emissions in climate change?) that is statistically meaningful is “Manmade CO₂ emissions drive climate change, but other natural and human-related factors are also important,” at 70% of respondents.

The only statistically valid answer for the full scientific population for question #3 (Which best describes the impact on global climate of controlling manmade CO₂ emissions?) is “Limiting manmade CO₂ emissions would have a strong impact,” at 72%.

Question #4 (Current mean global temperature is:) is the only question that has two statistically valid answers, namely “Unprecedentedly warm and getting warmer” and “Within natural variability but moving to unprecedentedly warmer levels,” at 56% and 31% respectively.

Question #5 has two answers that are statistically differentiated from the others but not from each other. In answer to The climatic impacts of a mean global temperature that is 1-degree Celsius warmer than today are, only “Undesirable” and “Desirable for some and undesirable for others”, at 48% and 39% respectively, are differentiated from the other answers, although they are within the margin of error of each other and so cannot be directly applied to all the IPCC scientists.

Question #6 (The ideal global climate is…) has only a single statistically valid answer: “There is no such thing as an “ideal” global climate,” at 61%.

In terms of the entire population, this means we can make only the broadest of statistical claims. And yet Mr. Milloy reports that his survey data illustrates a complete lack of consensus among scientists, when in fact it does nothing of the sort. In fact, were we actually make this survey statistically valid, the only conclusions we could draw for the the entire population (all 620 IPCC scientists) is that between 50 and 75% of scientists believe that human activity is driving global heating and that natural forces also matter, that between 57 and 83% feel that anthropogenic CO₂ is the driving force behind global heating, that limiting anthropogenic CO₂ emissions would strongly impact global heating, that between 74 and 100% believe that global mean temperatures are or will be soon at unprecedented levels, and the same number (74-100%) believe that the impacts will be somewhat or wholly bad. And again, this is applying scientifically valid conclusions to a biased, thoroughly unscientific survey.

Even if you accept Mr. Milloy’s data and assume the results are perfectly accurate (which, as I’ve illustrated above, they aren’t), he still intentionally misrepresents them in order to make his point. We can use his very own data to completely reverse his conclusions.

1. Only 4% of respondents deny the role of human activity in global warming.
2. 0% of their respondents denied the role of manmade CO₂ or human-related factors in global warming.
3. 0% of the respondents believed that limiting manmade CO₂ would have no impact on global warming.
4. Only 4% of respondents believed that mean global temperature is within natural variability and stable.
5. Only 4% of respondents believe that a 1 degree rise in mean global temperature would be desirable.
6. Only 2% of respondents believe that a warmer mean global temperature would be desirable.

There’s a reason that there’s jokes about lies, damn lies, and then statistics, and that joke applies to real statistics, never mind fake statistics like these.

Ultimately, this so-called survey is nothing more than a vehicle for Mr. Milloy and DemandDebate.com to try and discredit the very real (and statistically valid) consensus that does actually exist among climate scientists. It is nothing more than an attempt to use pseudo-science to create propaganda that can be used by Mr. Milloy and his conservative allies to imply significant disagreement where there is none.

Tomorrow: Part 2 – A closer look at the background of Steven J. Milloy, executive director of DemandDebate.com
Wednesday: A look at Steven J. Milloy’s current affiliates and backers