Today the Guardian reported on a new study that claims a large sale of legal ivory in 2008 actually led to an increase in illegal elephant poaching. Basically in 2008 China and Japan were allowed to pay for a large stockpile of legally-obtained ivory, in the hopes that this would crash the market and drive ivory traders out of business. Instead, the study claims, the sale led to a big increase in poaching – approximately a 66% increase in elephants killed, according to the study. This is interesting because it appears to put a big dent in a common libertarian idea for preserving endangered species – that allowing a regulated trade in them would lead to their preservation. It is also one of those cute findings that puts a hole in the standard just-so story of “Economics 101” that everything is driven by supply and demand. We all know that in reality there are many factors which moderate the effect of supply and demand on crucial markets, and on the surface this study appears to suggest a quite contradictory supply and demand relationship in illegal poaching markets, in which increasing supply boosts poaching. But is it true?
The Guardian report links to the original study, which is held at the National Bureau of Economic Research behind a paywall, but which I managed to get a copy of through my work. I thought I would check the statistical methods and see if the study really did support this conclusion. My judgment is that this study is quite poor, and that the data doesn’t support that conclusion at all, due primarily to three causes:
- A poor choice of measure for illegal poaching that doesn’t clearly measure illegal poaching
- The wrong choice of statistical method to analyze this measure
- The wrong experimental design
I will go through each of these reasons in turn. Where equations are needed, I have used screenshots from the original paper because I’m terrible at writing equations in html. Let’s get started.
The PIKE is a terrible measure of illegal poaching
The study is based around analysis of a data set of “legal” and “illegal” carcasses observed at search sites in 40 countries. Basically a “legal” carcass is an elephant that died on its own, while an illegal one is one that was shot and looted. Apparently poachers don’t bother to clean up the corpse, they just cut off the ivory and run, so it’s easy to see when an elephant has been poached. However, because no one knows the full details of elephant populations, the authors study an outcome variable called the PIKE, which is defined as the ratio of illegal carcasses to total carcasses. In their words (screenshot):
They say that this enables them to remove the unknown population from the outcome by “normalizing” it out in top and bottom of the ratio. They justify this with a little proof that I am not convinced by, since the proof assumes that probability of discovering carcasses is independent of the number of carcasses, and that legal mortality and illegal mortality are not related in any way. But even if it factors out population, this PIKE measure doesn’t tell you anything about illegal poaching. Consider the following hypothetical scenario, for example:
Imagine a population of elephants in which all the older elephants have been killed by poachers, so only the pre-adult elephants remain. Every time an elephant becomes mature enough to have decent tusks a poacher kills it and the corpse is found. Further, suppose that the population is not subject to predation or other causes of legal mortality – it is young, and the environment is in good shape so there are large stocks of easier prey animals for lions to target. This population is at high risk of collapse due to adults being killed as they mature; indeed, let’s suppose no babies are born because adults are poached as soon as they reach sexual maturity. Thus every time an elephant is killed, the population drops by one towards its inevitable crash.
In this case, at every time point the PIKE would be 1, because there are no legal carcasses. The PIKE will remain 1 until there are no elephants left to die, at which point it will jump to infinity. It doesn’t tell us anything about the impending population collapse.
Consider now a situation where there are a great many more legal deaths than illegal deaths. Denoting illegal carcasses by y and legal carcasses by x, we have y/(y+x) where y<<x. In this case we can approximate the PIKE by y/x, and if e.g. the number of illegal carcasses suddenly doubles we will see an approximate doubling in the PIKE. But suppose y is approximately the same as x. Then we have that the PIKE is approximately 1/2. Now suppose that the number of illegal carcasses doubles; then the PIKE increases to 2/3, i.e. it nowhere near doubles. If the number of illegal carcasses again doubles, it increases to 4/5. But if all deaths drop to 0 it then increases to infinity … So the magnitude of the increase in PIKE is not a direct reflection of the size of the change in poaching, and in at least one case even the direction is not meaningful. That is not a well-designed measure of poaching. It is also scale free, which in this case is a bad thing because it means we cannot tell whether a value of 1 indicates a single illegal carcass or 10 illegal carcasses. Similarly we don’t know if a value of 1/2 corresponds to 1 or a million illegal carcasses; only that however many there are, they are half of the total.
The authors say that this variable is constrained between 0 and 1, but this is not strictly true; it actually has an additional non-zero probability mass at infinity. This strange distribution of the variable has implications for model choice, which leads us to the second problem with their data.
All the models in this study were poorly chosen
The authors choose to model the PIKE using an ordinary least squares (OLS) model with fixed effects for country and a (separate) fixed effect for each year. An OLS model is only valid if the residuals of the model are normally distributed, which is a very strong assumption to make about a variable that has lots of values of 0 or 1. The authors claim their residuals are normally distributed, but only by pooling them across years – when you look at residuals within individual years you can see that many years have much more normally distributed residuals. They also don’t show us the crucial plot of residuals against predicted values, which is where you get a real idea of whether the residuals are well-behaved.
An additional consequence of using an OLS model is that it is possible to predict values of the PIKE that are unphysical – values bigger than 1 or less than 0 – and indeed the authors report this in 5.6% of their data points. This is indicative of another problem – the PIKE shows a non-linear response to increased illegal kills (see my example from 1/2 to 2/3 to 4/5 above), so that for a fixed number of legal kills each additional illegal kill has a diminishing effect on the value of PIKE, but a linear OLS model assumes that the PIKE changes by a uniform amount across its range. Given that the goal here is to identify increases in the PIKE over time, this runs the risk of the model over- or under-estimating the true effect of the 2008 ivory sale, because it is not properly modeling the response of the PIKE score.
The authors try to test this by fitting a new model that regresses ln(illegal carcasses+1) against a function that includes ln(legal carcasses+1) like so:
This introduces a new set of problems. The “+1” has been added to both variables here because there are many zero-valued observations, and ln(0) doesn’t exist. But if there are lots of zero-valued observations, adding one to them is introducing a big bias – it’s effectively saying there was an illegal carcass where previously there wasn’t one. This distorts low numbers and changes the patterns in the data. The authors claim, furthermore, that “The coefficient on legal carcasses φ will be equal to unity if the ratio of illegal carcasses to legal carcasses is fixed”, but this is both nonsensical and obscures the fact that this model is no longer testing PIKE. It’s nonsensical because that is not how we interpret φ. If φ=1, then we can rewrite their equation (8) so that the left hand side becomes the natural logarithm of (illegal carcasses+1)/(legal carcasses+1). Then we are fitting a linear model of a new variable that is not the PIKE. We are not, however, assuming the ratio of illegal carcasses to legal carcasses is fixed. If φ is not 1, we are modeling the natural logarithm of (illegal carcasses+1)/(legal carcasses+1)^φ. The ratio here is still fixed, but the denominator has been raised to the power φ. What does “fixed” even mean in such a context, and why would we want to model this particular strange construction?
The authors do, finally, propose one sensible model, which is similar to equation (8) (they say) but uses a Poisson distribution for the illegal carcasses, and still fits the same right hand side. This is better but it still distorts the relationship between illegal and legal carcasses by adding a 1 to all the legal (but not the illegal) carcasses. It also doesn’t properly account for elephant populations, which is really what the legal carcasses serve as a proxy for. There is a much better way to use the legal carcass data and this is not it.
Finally there are two other big problems with the model: It uses fixed rather than random effects for country and site, which reduces its power, and also it doesn’t include any covariates. The authors instead chose to model these covariates separately and look for similar spikes in specific possible predictors of ivory usage, such as Chinese affluence. The problem with this is that you might not see a strong spike in any single covariate, but multiple covariates could move together at the same time to cause a jump in poaching. It’s better to include them in the model and report adjusted poaching numbers.
The wrong experimental design
An expert cited in the original article noted this interesting fact:
The Cites spokesman also noted that there had never been a one-off sale of rhino horn: “However, the spike in the number of rhinos poached for horn largely mirrors what has been seen with ivory. The illegal killing of rhino for its horn in South Africa alone increased from 13 in 2007 to close to 1,200 last year.”
This suggests that there has been an upsurge in illegal poaching across Africa that is independent of the ivory sale, and could reflect changing economic conditions in Africa (though it could also reflect different markets for ivory and rhino horn). It’s possible to test this using a difference-in-difference approach, in which rhino poaching data is also modeled, but is treated as not having been exposed to an intervention. The correct model specification then enables the analyst to use the rhino data to estimate a general cross-species increase in poaching; the elephant data identifies an additional, elephant-specific increase that could be said to be due to the ivory sale. The authors chose not to do this, which means that they haven’t rigorously ruled out a common change in poaching practice across Africa. If the CITES spokesman’s point is correct, then I think it likely that we would conclude the opposite to what this study found: that compared to rhinos, elephant poaching did not increase nearly as much, and in fact the ivory sale protected them from the kind of increased poaching observed with rhinos.
Indeed, it’s possible that there were poachers flooding into the market at around that time for other reasons (probably connected to development and increasing demand in Asia), but after the ivory sale most of them switched to killing rhinos. That would suggest the sale was successful, provided you aren’t judging that success from the standpoint of a rhino.
A better model: Bayesian population estimation followed by Poisson regression
It’s possible to build a better model using this data, by putting the legal carcass data to proper use and then using a correctly-specified Poisson regression model on the illegal carcass data. To see how different the results might then look, consider Figure 1, taken from the Appendix of the paper, which shows the actual numbers of illegal carcasses in each year.
Figure 1: Distribution of illegal elephant kills, 2002 – 2013 (year is above its corresponding histogram)
Does it look to you like the number of elephants killed has increased? It certainly doesn’t to me. Note that between 20 and 50% of observed data are 0 kills in all years except 2002 (which the authors say was the start year of the data, and exclude from their analysis). Can you strongly conclude any change from these figures? I haven’t shown the legal kill data but it is broadly similar in scale. Certainly, if there is any upward step in illegal kills in 2008, it could potentially be explained simply by changes in populations of elephants – if even a small change in elephant density leads to an extra 1 or 2 extra kills per site per year, it would lead to distributions like those in Figure 1. To me it seems likely that the single biggest determinant of elephant kills will be the number of elephants and the number of poachers. If we assume the number of poachers (or the pace of their activity) changed after 2008, then surely we need to consider what happened to the population of elephants overall in 2008. If it declined, then poachers might catch the same number as 2007; if it increased, they would catch more.
The best way to analyze this data is to directly adjust for the population of elephants. We can use the legal kill data to do this, assuming that it is mostly reflective of elephant population dynamics. It’s not easy, but if from published sources one can obtain some estimate of the mortality rate of wild elephants (or their life expectancy), a Bayesian model could be built to estimate total population of elephants from carcasses. This would give a credible interval for the population that could then be used as what is called an offset in a Poisson regression model that simply modeled counts of illegal kills directly against time. The advantage of this is that it uses all 0 count events, because a Poisson model allows for zeros, but it adjusts for the estimated population. I think the whole thing could be done in a single modeling process, but if not one could obtain first a distribution of the elephant population, then use this to simulate many different possible regression model coefficients for the effect of the ivory sale. In this model, the effect of the ivory sale would simply represent a direct estimate of the relative increase in mortality of elephants due to poaching.
Then, to complete the process, one would add in the rhino data and use a difference-in-difference approach to estimate the additional effect of the ivory sale on elephant mortality compared to rhinos. In this case one would find that the sale was protective for elephants, but potentially catastrophic for rhinos.
Conclusion
Based on looking at this data and my critical review of the model, I cannot conclude that the ivory sale led to an increase in poaching. I think CITES should continue to consider ivory sales as a tool to reduce elephant poaching, though with caution and further ongoing evaluation. In addition, based on the cited unnamed CITES spokesman, evidence from rhino culling at the time suggests the sale may even have been protective of elephants during a period of increased poaching; if so, a further big sale might actually crush the business, although there would be little benefit to this if it simply drove poachers to kill more rhinos.
With regard to the poor model design here, it shows a lot of what I have come to expect from economics research: poor definition of an outcome variable that seems intuitive but is mathematically useless (in health economics, the incremental cost effectiveness ratio shows a similar set of problems); over-reliance on OLS models when they are clearly inappropriate; poor model specification and covariate adjustment; and unwillingness to use Poisson or survival models when they are clearly most suited to the data.
I think there is lots of evidence that legal markets don’t necessary protect animals from over-exploitation (exhibit A, the fishing industry), but it is also obviously possible that economic levers of supply and demand could be used to kill an illegal industry. I suspect that more effective, sustainable solutions to the poaching problem will involve proper enforcement of sales bans in China and Japan, development in the regions where poaching happens, and better monitoring and implementation of anti-poaching measures. If market-crushing strategies like the 2008 ivory sale are going to be deployed, development is needed to offer affected communities an opportunity to move into other industries. But I certainly don’t think on the evidence presented here that such market-crushing strategies would have the exact opposite of the intended effect, and I hope this poor quality, non-peer-reviewed article in the NBER doesn’t discourage CITES from deploying a potentially effective strategy to stop an industry that is destroying a majestic and beautiful wild animal.
Compromise and Conceit 
August 11, 2016 at 2:12 pm
It is a common misperception that the two CITES ‘one-off sales’ were intended to crash the market. This is a red herring planted by anti-trade campaigners. No single sale of any commodity can crash a market, for the simple reason that no one controlling a commodity would do such an idiotic thing, even if it were possible.
The main flaw with the study in my mind is that the authors did not test any alternative hypotheses. ‘Effect of sale’, the independent variable, was defined as the year 2008. Any event in 2008 substituted for ‘Effect of sale’ would have resulted in the same causal inference.
The study is a political tool intended for use at the CITES CoP17, it is not a serious scientific study.
August 11, 2016 at 2:30 pm
Thanks for your comment Daniel. I notice web hits on this post have gone up a lot this week, I guess in anticipation of CoP17.
First I’d like to say that I didn’t have any conceptions or misconceptions about the CITES one-off sales – I was just reporting what the Guardian and the authors stated as possible benefits of the sale. I agree that the study is flawed for not testing an alternative hypothesis, although this is often difficult with natural experiments of this kind. My proposed alternative (a difference-in-difference design with rhinos as the control) is one partial solution to that problem. But I don’t think any meaningful results can be extracted from the measure of poaching they used – first they need good estimates of the elephant population in the affected area.
Do you think the CITES CoP17 will lead to any policy changes about ivory poaching?
August 13, 2016 at 4:04 pm
Your post was recently disseminated by someone who belatedly found it. Many of us have been discussing this paper and its shortcomings. The problem with CITES coming up with any beneficial policy changes that might reduce poaching is that there is too much misinformation being produced by anti-trade NGOs, this paper being just one example.
A colleague who is much better placed than I to understand your arguments was in agreement with your post. He would like to send you a PowerPoint, is that possible?
August 13, 2016 at 7:24 pm
I don’t think you should accuse them of propaganda (at least it would be better if you refrained on my blog). I think there are legitimate reasons to be suspicious of legal trade in ivory, and in the absence of good research everyone has to make judgments based on predictions and theories. As you might see from other recent posts on this blog, I don’t think highly of the quality of research published by the NBER, and it’s perfectly possible that they’ve simply deployed poor quality research in pursuit of clarifying this issue (of course I don’t know anything about the different factions involved in this debate, so I have no idea how dishonest anyone is being).
I’m happy to look at a powerpoint, although I can’t guarantee I’ll be interested in providing feedback. I’ll send you an email.
September 2, 2016 at 3:09 am
The pro-trade lobby (leaving aside its apparent inability to understand wildlife markets) is mainly characterised by extreme paranoia. Nowhere is this more obvious then in the ramblings of Daniel Stiles. The study you refer to came out of left field, was not orchestrated by NGOs and, according to the authors, ended with a conclusion that did not tally with their previous stance on legal trade in ivory. Whatever you may think of their paper, thank goodness for researchers who are prepared to say “I was wrong”. It’s a rare thing. Having just read a piece by a South African journalist, where he repeats the same tired old phrases about “animal rights” NGOs (do any of those actually go to CITES?) influencing Kenya’s stance on policy and, in fact, going so far as to accuse them of threatening the Kenyan government with tourism boycotts if they change their position, I now read that somehow NGOs are responsible for “disseminating” this study. As someone who loves to see his own name in print, Daniel Stiles should know that researchers are quite capable of disseminating their own work.
September 2, 2016 at 3:14 am
Oh, and i did not mean to imply that “faustusnotes” is among the paranoid brigade! On the contrary s/he appears eminently rational.
September 2, 2016 at 10:44 am
Thank you for commenting Michael. As I said above, I prefer not to engage in speculation about motivations on my blog – please don’t. And accusations that I’m rational probably don’t reflect well on you – have you seen the other stuff on this blog!?
If you have any commentary on the paper or my critique of it, I’d like to hear it.
October 16, 2016 at 11:08 am
Everything IS driven by supply and demand. That doesn’t mean that we have all the variables that are needed to make predictions. The idea, for example, that as demand increases prices will rise, is predicated on the concept of Ceteris Paribus. In this example, and many others, not everything remains equal. In this case, if it’s to be believed, demand for ivory increased based of the removal of the stigma associated with buying ivory. This is not an indictment of the standard “Economics 101” model. It is an affirmation of it. The fact that these people forgot to factor in the effects of, in essence, removing a law (an artificial barrier to trade) and thereby increasing demand has nothing to say about the voracity of the laws of supply and demand. The increased demand from people for ivory caused a corresponding increase in supply from poachers, just like the theory would predict. They also forgot to factor in the need for poachers to produce more ivory to make up the cost of the drop in price.
These stopgap measures are no test of an actual trade in ivory, elephants, or any other endangered species (or private property rights where none exist). Poachers aren’t going to suddenly stop what they’re doing simply because profits are dropping. That’s not how people go about doing business.
This all assumes that these people know what they’re talking about, however.