Richard Tol Versus Richard Tol On The 97% Scientific Consensus

97% piechartLast year Cook et al. released a paper in which they analysed the scientific consensus on anthropogenic global warming based via the peer-reviewed scientific literature.

What they did in that study was examine 11,944 abstracts from 1991 to 2011 that included the words “global climate change” or “global warming” in their abstract. What they found after analysing these abstracts is that among those that expressed a position on global warming, 97% endorsed the consensus position that humans are causing global warming. They also contacted 8,547 authors to ask if they could rate their own papers and received 1,200 responses. The results for this again found that 97% of the selected papers stated that humans are causing global warming.

For anyone aware about similar research this was not a surprising result as in 2004 Oreskes did a similar literature search – although it included ‘only’ 928 abstracts – which already found this scientific consensus. A 2009 survey of Earth scientists found that among climate scientists actively publishing climate research, 97% agreed that humans were significantly raising global temperature. A 2011 analysis of scientists’ public statements about climate change found that among those who had published peer-reviewed climate research, 97% accepted anthropogenic global warming. When you take a look at how this consensus evolved from 1996 to 2009 you see a steady increase in the agreement among scientists (Bray 2010).

The 97 percent science consensus

The 97% consensus was found in Doran & Zimmermann 2009, Anderegg et al 2011 and Cook et al 2013.

This remarkable agreement exists because a scientific consensus is reached on the weight and amount of research that is available in the literature. It’s also this scientific evidence that led to the scientific consensus on for example evolution, plate tectonics, the big bang, germ theory, and so on. Such a consensus only arises through meticulous study and hard work by scientists.

This is also what makes it such a powerful tool to communicate science to the public. There’s a huge gap between what the experts say based on the scientific literature and the public awareness of this.

The consensus gap

Which is also the reason the Cook et al. study is so relentlessly attacked by science deniers and pseudo-sceptics. It’s the only tactic they really have as they can’t base their case on scientific research, they just don’t have the supporting evidence to show that they are right. Most of the time they can only allude to nefarious going ons that prevent such evidence from getting into the literature. But that ignores that proving a well established scientific idea wrong advances the career and reputation of a scientist more than providing ground breaking supporting evidence.

The lack of evidence science deniers and pseudo-sceptics have shows via the methods they use for attacking the cook et al. study. Often they claim that not all needed data for verifying the results are available and/or that the study is deeply flawed. But both aren’t true.

How the abstracts were rated and how results were analysed are available in the paper. All the data needed to verify the results can be downloaded from the very same page this paper is hosted on (it’s linked under supplementary data). It contains everything you need to see if problems in the used methodology exist or that for example abstracts were incorrectly rated. Any serious problems in the Cook et al. paper will be detected by using that data and trying to replicate the results. But I already mentioned that to my knowledge science deniers and pseudo-sceptics aren’t doing that.

Even if they found problems in the abstract ratings in the Cook et al. paper it doesn’t invalidate the consensus they found. Simply because they also asked the authors of the selected abstracts to rate their own papers. The same consensus in the scientific literature was found via this method and confirmed the results the Cook et al. team found via the abstract ratings. But I rarely see any of the ‘critics’ even acknowledge this, when they do it’s often because they are dismissing it.

The growth of the scientific consensus on human-caused global warming in the peer-reviewed literature from 1991 to 2011, from Cook et al. (2013)

The growth of the scientific consensus on human-caused global warming in the peer-reviewed literature from 1991 to 2011, from Cook et al. (2013)

I expect this kind of behaviour from science deniers and pseudo-sceptics. I didn’t expect it from an economist like Richard Tol. Since the release of the Cook et al. paper Tol has made several statements about this paper being flawed. And has since been working on proving that this initial statement is correct. No matter how unwarranted or irrelevant his criticisms are towards the results found in the Cook et al. paper.

One of the more repeated ones is that data is hidden so he can’t check everything. But many, including me, have pointed out to Tol that he has everything he needs to check if the results are valid. And that so far there isn’t any real indication in the Cook et al. paper that it’s fundamentally flawed. As indicated by the agreement between the abstract ratings and the ratings done by the authors of the papers.

He also often speculates about a bias towards rating abstracts as endorsing the scientific consensus on anthropogenic global warming when they didn’t. However, one of the biggest hints that this didn’t happen are again the author ratings as they show that the raters were conservative with their ratings. Meaning that the ratings for the abstracts tended to er towards being neutral or rejecting that humans were the cause of global warming.

Difference between abstract ratings and author ratings

Another criticism Tol often talks about is fatigue influencing how abstracts were rated. Which is a real phenomenon in surveys, but the Cook et al. paper isn’t a survey. It’s not a paper based on people filling in a questionnaire, it’s about categorizing abstracts of papers. A tedious job, but that’s the reason they worked on this for months. They took their time to do this right.

One of the methods that was used is that every abstract was rated by at least two people. This was done as a person is fallible, they can make mistakes. So having at least two people do this reduces any issues that might arrise from that. And this is what the Cook et al. paper says about that particular process:

Initially, 27% of category ratings and 33% of endorsement ratings disagreed. Raters were then allowed to compare and justify or update their rating through the web system, while maintaining anonymity. Following this, 11% of category ratings and 16% of endorsement ratings disagreed; these were then resolved by a third party.

That’s a lot of work to make sure that they got their ratings right. As previously mentioned this shows through in the agreement between the abstract ratings and the author ratings, and in the detail that the abstract ratings tended to er towards being conservative. So any issues found in the ratings done on the abstracts will be minor and it won’t negate in any way the agreement found through the author ratings.

The reason all this so puzzles me is that Tol has said that “It is well-known that most papers and most authors in the climate literature support the hypothesis of anthropogenic climate change. It does not matter whether the exact number is 90% or 99.9%.” And that there’s no doubt in his mind “that the literature on climate change overwhelmingly supports the hypothesis that climate change is caused by humans. I have very little reason to doubt that the consensus is indeed correct.”

Yet he has said on twitter, and several other venues, that the found scientific consensus of “97% is a load of nonsense“. I know he says this because he thinks there’s a slight error in the found consensus percentage, but statements like this make it look like he’s saying that the entire paper is fundamentally flawed. It also leaves out the detail that he agrees with the general findings of the Cook et al. paper.

So why go after the paper in this way when he has all the data needed to verify the results? Fortunately he has explained why he’s doing it like this:

I have three choices:
a. shut up
b. destructive comment
c. constructive comment

a. is wrong
c. is not an option. I don’t have the resources to redo what they did, and I think it is silly to search a large number of papers that are off-topic; there are a number of excellent surveys of the relevant literature already, so there is no point in me replicating that.

that leaves b

When I read this explanation last year it left me dumbfounded, because option c is the one that you choose. Also rating all the abstracts again doesn’t have to be necessary, with a large enough sampling any big issues with the abstract ratings would be noticed. Everything Tol needs for a constructive criticism is already available to him.

In this case, considering he isn’t choosing option c, I think that option a was the correct one.

Collin Maessen is the founder and editor of Real Skeptic and a proponent of scientific skepticism. For his content he uses the most up to date and best research as possible. Where necessary consulting or collaborating with scientists.