Last week’s Journal of the American Medical Association had an excellent article by Chapman et al giving a robust analysis of the effect of the change in Australia’s gun laws that happened in 1996. These laws (the National Firearms Agreement) were enacted very rapidly after a major mass shooting (the Port Arthur massacre) in which 35 people died. Their major components were banning certain kinds of weapon, and introducing a gun buyback scheme to enable gun owners to hand in their guns and be compensated, provided they did so within an amnesty period. Wikipedia describes the law changes in a short paragraph that shows how wide reaching they were:

The law, which was originally enforced by then-Prime Minister of Australia John Howard, included a number of provisions. For example, it established a temporary firearm buyback program for firearms that where once legal now made illegal, that according to the Council on Foreign Relations bought over 650,000 firearms. This program, which cost $230 million, was paid for by an increase in the country’s taxes. The law also created a national firearm registry, a 28-day waiting period for firearm sales, and tightened firearm licensing rules. The law also required anyone wishing to possess or use a firearm with some exceptions, be over the age of 12. Owners must be at least 18 years of age, have secure storage for their firearms and provide a “genuine reason” for doing so.

The laws have been partially evaluated a few times, were the subject of an excellent John Oliver piece, and have been controversial amongst pro-gun activists for some time, with much debate about whether or not they worked. One big problem with analyzing their impact is that the rate of firearm homicides was already in decline when the laws were enacted, and at the same time the rate of non-firearm suicides began to decline in a sharp turnaround from past trends. This has given a lot of room for people concerned about the laws to argue they had no impact.Chapman et al’s article provides a thorough analysis of all the available data on the laws. The analysis uses nationally-available death and population data from 1979 – 2013, so it can analyze two 17 year periods of data to look for changes in rates. It uses the correct analytical method to handle the low numbers of counts (negative binomial regression), and the models are constructed carefully to enable comparison not just of the changes in deaths that occurred at the time the laws were introduced, but to calculate changes in trends at this point in time, and to test if these trends occurred by chance. They conducted the analyses separately for firearm- and non-firearm suicides and homicides, total homicide deaths and gun homicide deaths with mass shooting-related deaths removed. Their key findings were:

    The rate of decline of firearm homicides accelerated, though this acceleration was not statistically significantThe rate of decline of firearm suicides accelerated, and this change was statistically significantThe increase in non-firearm suicides changed to a decrease, and this change was statistically significant

They conclude that there was no evidence of substitution of suicide methods due to the change in laws. Overall their findings seem to be robust, but actually there is a small flaw of interpretation and modeling in this paper that makes it, in my opinion, a missed opportunity to give a definitive answer to the question of the true effect of these laws.

Several limitations with the paper

The big problem with this paper is its failure to directly compare changes in different rates of death. They fitted separate models for the four kinds of death, when in fact they could have fitted a single model for all four kinds of death, plus time and interactions between the four kinds of death with each other, time and the laws. This model would have been slightly nasty to interpret, but would have the benefit of enabling the reader to identify any additional effect of the law on firearm homicides vs. non-firearm homicides, and firearm suicides vs. non-firearm suicides. Statistically significant terms for these parts would imply that the law had a bigger effect on firearm-related deaths than non-firearm-related deaths. This would also have the advantage of giving the model larger numbers of counts, thus reducing confidence intervals. My suspicion, just looking at the data presented in the paper, is that if this more complex model had been fit the authors would have found that the change in laws affected homicide and non-homicide deaths, and suicide and non-suicide deaths. This probably wouldn’t be as interesting a finding, but it would have been more robust.

The second big problem with the paper is that it doesn’t include a control group. I have previously written a post on this, in which I suggested using New Zealand data as a control group, since NZ is very similar to Australia but didn’t enact gun laws at that time. In that post I found that we would probably need to wait until 2023 to make a definitive conclusion on whether the gun laws prevented mass shootings. I didn’t touch so much on the homicide/suicide analyses but the same rules would apply. By using a control group we can rule out any possible cultural changes that may have happened more broadly at that time.

It’s also worth noting that the study doesn’t adjust for age. As Australia ages we expect to see the rate of homicides decline, since older people don’t shoot each other as much as the crazy young’uns, and this adjustment didn’t happen in the study. Given the conclusion about firearm homicides is primarily one based on trends, and a slowly aging society should see the effect of age through changes in trends, this was a missed opportunity. Similarly, suicide tends to happen in age groups where homicides don’t (above the 30s) and an aging society might be at higher risk of suicide, so adjusting for age might find an even bigger effect of the laws. I think it’s possible that a combination of aging society plus increasing proportions of non-white migrants[1] might explain the sudden cessation in mass shootings, especially if you treat mass shooting as an infectious disease, that is less likely to break out as the period of time between outbreaks increases.

Finally, the study doesn’t appear to have actually analyzed statistically the decline in numbers of mass shootings. Is this because the result was non-significant? It’s a strange omission…

Conclusion

This study provides better evidence than previous studies of the effect of the national firearms agreement on firearms-related deaths in Australia, but it is not conclusive. There is still a possibility that the decline in firearm homicides was non-significant, and that the effect on firearm suicides was coincidental. In the absence of a control group, and without constructing a full interaction model testing differences in trends between suicide methods, it is not possible to definitively conclude that the observed effects were due to the national firearm laws. Also, in the absence of a statistical test of the effect on mass shootings, we also cannot conclude that the national firearms agreement reduced these shootings. Nonetheless, the study provides strong evidence that the laws achieved their intended purpose. A more thorough analysis with proper interaction terms might answer this question definitively, but sadly didn’t happen in this particular paper.

fn1: This is probably a slightly controversial position but I have a suspicion – purely theorizing – that mass shootings start off as an in-group thing, they’re something that the majority population do to themselves. This appears to have historically been the case in the USA, with most shooters being white, but somehow in the last 10 years the disease broke out of this group and into non-white minorities, first Asian and then black Americans. I suspect this is unusual, and requires a long period of regular exposure to shootings by the in-group before it happens. This isn’t meant to say that any particular racial group is more prone to mass shootings than any other, just that it starts in the mainstream group and, while it remains a very rare event, remains there. So as the proportion of the population that this group fills declines, the rate of mass shootings also declines, leading to less and less social contagion both within the in-group and between the in-group and others. The exception to this is the USA, where the easy availability of guns means that there is no brake on the continued high rate of events, and eventually the infection spreads out of its main host[2].

fn2: In case it isn’t clear, I think that mass shootings should be seen as a kind of infectious process, spread by media hype, and have suggested changes in media laws to prevent this.