Hey, just what you see, pal!
With the sad events recently in America, and president Obama’s newfound (apparent) determination to do something about assault weapons, a lot of attention has focused on Australia’s National Firearms Agreement (NFA), which outlawed certain categories of guns and included a program to buy back guns already in circulation (commonly called the “gun buyback program”). Several studies have been conducted in the aftermath of the agreement, and these seem to have found that it was associated with a reduction in suicides and possibly homicides, with some evidence that the widely expected substitution in suicide methods didn’t offset the gains. The suicide gains appear to be quite large. I think the definitive paper is probably that by Simon Chapman and colleagues, which can be read free online (I think) through pubmed and gives some impressive charts showing declines in gun-related mortality. It is obvious from those charts though that homicide rates were declining anyway, so the results on homicide are weak. Leigh and Neill published a state-by-state analysis of similar trends, with details on the magnitude of the buyback program in each state, and also found tentative evidence in support of the law’s effects on homicides and suicides. However, there is a side debate about whether the law had an effect on mass killings, these events having been the spur for the law in the first place and also the primary focus of any gun laws that Obama will introduce.
The figures for mass killings are fairly impressive: Australia had 12 between 1980 and 1996, but has had zero since. New Zealand had 3 between 1980 and 1996, and had 1 since (in 1997). The USA has a continuing rate of mass killings, and these comparisons are suggestive of an effect of the law. Here I will argue that we need to wait another 10 years before we can make any statement about the effectiveness of the laws in this regard, and probably another 20 years before we can rule out the possibility that the sudden cessation of mass killings was due to other reasons. I will be basing a lot of this analysis on a dubious paper by McPhedran and Baker in the Justice Policy Journal, which contains the data on mass killings for the two countries.
Methods
Data on all mass shootings in New Zealand (NZ) and Australia were collected from the McPhedran and Baker paper for the period 1980-2012. In this paper a mass shooting was defined as an event where at least 4 people were killed by gunshots, not including the perpetrator. McPhedran and Baker give the number of mass shootings and the total death toll for each country. The study period was divided into pre-NFA (1980-1996) and post-NFA (1997-2012) periods. For the pre-NFA period the number of mass shootings was turned into a rate per unit of NZ population per year, on the assumption that although populations changed over time during the study period, the ratio of Australia’s to NZ’s population remained roughly static at approximately five times. Practically, this means that the number of mass shootings in NZ was divided by the length of the pre-NFA period (16 years) to get an annual rate of shootings, while the the number of mass shootings in Australia was divided by five times the length of the period to get its rate. This enables a sloppy, back-of-the-envelope calculation without fiddling with tricky population changes[1].
Once the rates had been calculated for the pre-NFA period, the probability that the observed number of shootings would be seen in the post-NFA period was calculated, on the assumption that mass shootings follow a poisson distribution and that the rate of shootings was not affected by the NFA. If the probability of observing the given number of shootings in Australia in the post-NFA period was less than 5%, we conclude that the NFA was effective; if the probability of observing the given number of shootings in NZ in the post-NFA period was less than 5%, we conclude that some other cultural change was occurring to drive down rates of mass shootings.
Finally, if the probability of observing the given number of shootings was greater than 5%, the additional number of years required to get the probability below 5% was calculated for both countries. Dubious conclusions were drawn on the basis of this[2].
Results
There were 12 shootings in Australia in the pre-NFA period, with 96 deaths, and 0 in the post-NFA period. There were 3 shootings in the pre-NFA period in NZ, with 24 deaths, and 1 in the post-NFA period with 5 deaths. Using the bodgy standardization method, we would expect to observe 15 mass shootings causing 120 deaths in Australia in the pre-NFA period if it had the same mass shooting rate as NZ, which suggests that shootings were less prevalent in Australia before 1996.
The rate of mass shootings per unit of NZ population per year in the pre-NFA period were 0.19 in NZ and 0.15 in Australia. Under the assumption that mass shootings are Poisson distributed, if these rates were to continue in the 15 year post-NFA period, there would be a 22% probability of observing only 1 or 0 mass shootings in NZ, and an 11% probability of observing 0 mass shootings in Australia. That is, we cannot say that the large reduction in the number of shootings post-NFA was statistically significant in Australia.
In fact, in order to conclude that the NFA had a significant effect on mass shootings, we would need to observe no mass shootings in Australia between 1997 and 2017. Conversely, if there are no mass shootings in NZ between now and 2023, we can conclude that a cultural change occurred in NZ that led to a reduction in mass shootings. So really, we have to wait until 2023 before we can draw any statistical conclusions about the effect of the NFA on the rate of mass shootings, unless we are willing to argue that New Zealanders are so radically different to Australians that they are not an appropriate control group for such a comparison.
Conclusion
It is not possible to conclude at this stage that the National Firearms Agreement reduced the rate of mass shootings, and we will need a much larger period of study before we can. Note that were we to approach this problem in 2017, when we had sufficient post-NFA data, we would also be best advised to extend the pre-NFA study period to 20 years, and this would likely reduce the rate of shootings in Australia (I think the 90s were a high water mark for this), which would again mean that we would likely again find no effect. It’s a bit of a hare and tortoise thing, this.
This study has limitations. Its primary limitation is that it is extremely bodgy. Secondary limitations include its having bought into the dubious framing of the McPhedran and Baker paper, its lack of attention to mortality, and the fact that it used a Poisson rather than a negative binomial distributional assumption – I think the results would be even less conclusive if a negative binomial distribution were used but I don’t now that much about those distributions, so can’t comment.
As a final note, it’s my impression that the NFA is well liked in Australia, easy to implement, cheap, and apparently effective in reducing suicides and homicides, possibly meaning it was more effective than originally intended. This law was implemented very quickly by a new and quite inexperienced prime minister, in a pressure cooker public environment and often against the interests of a constituency that was naturally inclined to vote for him. I think this is a testament to the policy-making skills of Australia’s bureaucracy at the time, and also to Howard’s political skill in that period. This exactly the kind of situation in which bad legislation will be drawn up, but he seems to have done a good job. I think it would be really fascinating if he and a few of his advisors and bureaucrats of that era were to get together and write a perspective piece for a major medical journal, describing the pressures they faced and how they managed to produce the legislation given the public and policy challenges they faced. I think it would get into the BMJ – they should do it! Whether the legislation was implemented as a cynical popularity boost by Howard, or for genuine public policy purposes, the results have been surprising and I think it would be interesting to see how such a beneficial law can be developed and implemented in such an environment.
It’s also worth remembering that the previous government under Bob Hawke faced a similar challenge with HIV/AIDS in 1984, and responded similarly effectively. This suggests that in health and law enforcement, Australia’s policy development framework is robust, effective and well liked by the population independent of the political leanings of the party in power. Let’s hope it stays that way long beyond the post-NFA period!
—
fn1: opinions are easy to have, and one shouldn’t have to work too hard to arrive at them.
fn2: For those tempted to try this at home, this is definitely not how the experimental method works. It’s a thought[3] experiment!
fn3: Obviously I’m using this term quite loosely.
Compromise and Conceit 
December 31, 2012 at 11:52 am
What you are saying is that there is no way we can statistically prove that Australian gun laws dimished the chance of a mass killing. Fair enough – you are a statistician. But there are other plausible kinds of “proof”. When you observe the tendency of trends in Australia to mimic those in the US, and the increased media exposure given the frequent killings there, the social narrative trend would seem to have been towards more mass killings in Australia. The gun laws broke that narrative trend, and substituted one where a gun fixation is increasingly seen as a perversion (note eg that the Shooters Party does not talk about gun manliness or guns as liberty but guns as instruments for the control of feral animals). Quite what I would look for to confirm that trend I don’t know, but it would not be mass killings.
December 31, 2012 at 4:49 pm
That’s pretty much what I’m saying, though I do think if we wait long enough we can see a statistical effect – currently we can say with 90% confidence that rates of mass shootings declined in Australia, for example.
I think in these kinds of situations we need to seek the other plausible kinds of proof you mention, and I agree with you that it does appear Australia’s attitude towards guns is completely different, and the NFA plays a role in that. The same applies to Bob Hawke’s response to HIV/AIDs, I think: not only is it hard to prove that any given Australian policy prevented the spread of HIV here, but similar policies tried elsewhere haven’t been successful (e.g. in Africa). Nonetheless, I think everyone is fairly confident that it was Australia’s policy response that worked in that case, too, based on evidence other than the statistical.
The problem in America appears to be that the sides are so strongly drawn that it is impossible to debate the other plausible kinds of proof honestly. Though probably that problem would apply even if there was a single killer statistical proof, as well…
January 1, 2013 at 12:35 pm
Based on the climate science, budget and other discussions, not even a direct edict from God Almighty would be enough to end the argument.
May 23, 2013 at 12:07 am
The problem, of course, is that gun control is a bit like climate-change denial. You’ll simply never get through to people whose conviction is religious rather than scientific.
Also, as a statistician, you might want to look at the suicide and homicide by gun in Australia, as there is no evidence that the gun ban effected either. As a percentage of all homicides and all suicides, those using guns declined at the same rate after the ban that they had been declining before the ban, and then plateaued over a decade later.
As a brief example, in 1988, 24% of suicides were by gun. In 1996, that had fallen to 16%, and in 2004, it was 8%. Currently, it varies between 6% and 7%. Even during 97 and 98 when all suicides were on the rise, this general trend in gun suicides held. In fact, examined year-to-year, there is no observable relationship between changes in total suicides and gun suicides such as would eliminate substitution. And, given the other historical evidence, the substitution seems more likely to be due to cultural influences than legal ones.
In the case of homicides, the decline began in 1968 and accelerated to a rate in the mid 80s that it maintained until the early 2000s, at which point it also plateaued. As with suicides, there is no comparison between total homicides and the percentage committed with guns with a year-by-year comparison to indicate any kind of relationship that could attribute declines in total homicides to declines in gun homicides.
It is certainly possible that the existing trends would not have continued without the legislation, but given the existing trends, it is impossible to state with any certainty whether the gun ban had any causal effect whatsoever.
May 26, 2013 at 5:24 pm
Brkibbey, thanks for commenting. I think you might have missed the first paragraph of this post, though, when you say “there is no evidence that the gun ban effected either” suicide or homicide by gun. From Leigh and Neill’s abstract:
This is pretty well-established, statistically.
Incidentally, presenting your analysis of suicide rates in terms of cause fractions is dangerous, since cause fractions are affected by the total suicide rate. But out of interest, where are you getting your figures from?
Your final statement doesn’t agree with the literature I cited in the first paragraph. Given this, I think your opening statement about religious conviction is a little rich, and you need to present a more coherent argument with more data before you can make that claim. Perhaps you could start by addressing the Leigh and Neill paper?
(And you might want to note that I have laid out clear conditions under which the NFA can be said to have reduced mass killings. Do you dispute this argument, and what will this mean for your own convictions if the conditions are met?)