In the set-up post for Saturday's Media3 programme, I noted that it was important to recognise that the controversial ecologist Dr Mike Joy ought not be immune from criticism -- but his critics should be able to do what he does, and cite evidence for what they say.
Mike Dickison has done just that in a critique of Evaluating the Relative Environmental Impact of Countries, which appeared in the open-access journal PLoS One and was cited in the International Herald Tribune/New York Times story that kicked off the current fuss about New Zealand's "100% Pure" environmental status. Joy and others have also cited it demonstrate that, as Joy put it in a column for interest.co.nz:
We are now much much closer to the bottom than the top of global comparisons on environmental performance; this is contrary to the belief expressed by Prime Minister Key.
But are we? Dickison's assessment of the accuracy of our "18th worst" ranking in the PLoS study:
Not very. One of the variables dominates all the others, but is a little unfair. One of our rankings is just a blunder. A couple of variables are calculated wrongly, one database looks pretty dodgy, and the whole issue of sustainable fishing when you don’t have a coastline skews all the results.
Ironically, the lack of rigour Dickison finds in the PLoS study is mirrored (and then some) in another study, one that also attempted to provide a social overlay for environmental data: the 2010 Yale Environmental Performance Index, whose ranking of New Zealand as second only to Iceland for water quality was repeatedly cited by the Prime Minister last year in support of our environmental performance.
I interviewed Joy about his disagreements with the Prime Minister when he first came to national attention in May last year. It's an interesting watch 18 months later:
At the time, we recorded an Extra segment in which Joy went into some detail on the failings of the Yale index. But I can't readily point you to that, because it was never uploaded to YouTube and TVNZ, in its wisdom, has deleted all TVNZ 6 and 7 content from its on-demand service.
But the quotes from Joy in this Dominion Post story from the same period give a reasonable idea of his criticisms:
But Dr Joy poured cold water on the study yesterday, claiming the data it used was "totally flawed". Firstly, out of the 130 countries ranked, only half actually had any water quality data available for the comparison, while the rest had estimates.
The data was also an unfair representation of nations as a whole. "The 11 sites they got from Australia were all from one state. You can hardly represent the water quality of the whole country by examining one tiny portion of it."
He said half of the 80 measurements used to calculate New Zealand's water quality were from pristine "control sites" known to be free from the impacts of agricultural and industrial pollutants.
Mr Joy said he called the Yale study's authors when he heard it being quoted by the Government and was "shocked" by the research it was based on.
As Joy predicted at the time, New Zealand slipped down rankings in the 2012 Yale index. Peter Griffins's recent SciBlogs post says this:
In the rankings, New Zealand comes in in 14th place out of 132 countries – between Iceland and Albania. Not a bad result. But Yale also issues an EPI Trend ranking – looking at how we’ve progressed over the past decade. Using this indicator we are ranked 50th, between Armenia and Slovenia.
Intriguingly, New Zealand's overall 2012 ranking is actually one place up on 2010 -- but if you compare our 2010 and 2012 country profiles, you'll see that our score (high scores are good in this instance) on "water resources (ecosystem effects)", which includes water quality, has gone from 94.98 to 40.3. We're ranked 43rd in the world there.
I can't tell you have much of that shift is a consequence of the Yale team getting more substantial data from Joy and others. But the indicator series for the 2012 index still looks like there's a lot of arbitrary data in there anyway.
[NB: It's messier than that. Nathaniel Wilson notes in comments that the indices have changed but the numbers haven't. He has another useful comment on the limits of global rankings here.]
Mike Dickison concludes his post thus:
New Zealand’s environmental impact might well put us in the bottom half of the world league table, but we actually don’t know if it does or not, and this study is not much help. Our country has real environmental issues—and is not by a long shot “100% pure”, absolutely or relatively, despite what the PM says—but claiming we are 18th-worst in the world seems just flat-out incorrect. And if you continue to claim it after knowing how dodgy the data are, you’ve crossed a line from science into advocacy.
On the face of it, his revised analysis of the PLoS study puts New Zealand 39th-worst, which is still not incompatible with Joy placing us closer to the bottom than the top of international rankings -- but you'd need to reassess every country's data the way he has New Zealand's to reliably say so. He also can't reconcile our relatively okay performance on water pollution with the damning 2004 NIWA study of lowland rivers cited by Joy and others, which depicts an emerging catastrophe.
Joy, for his part, will have ruffled a few feathers with this comment on that line between science and advocacy in Saturday's Media3 show:
I'm not prepared to catalogue the decline of species in New Zealand and make a career out of that. I made a decision quite a while ago that writing journal articles that other nerds like myself -- half a dozen, probably, around the world -- read is not going to change anything. My belief is that if the public realises we have a problem, that's the only way we'll get to change. If nobody knows then nothing changes. It just keeps on getting worse.
It's really the same reason that social scientists develop environmental indexes, even at the peril of fudging the hard science: there must be some way of getting the hard science into the social and political domain. Both the indexes covered here -- the one heralded by the Prime Minister and the one cited by Joy -- seem to have significant shortcomings.
But perhaps, as Joy readily agreed when I put it to him on the show, the issue isn't arguing about our relative standing in comparison with the rest of the world, but the absolute fact that New Zealand has significant environmental issues, they appear to be getting worse over time -- and we need to do something about that.