… What would statistical analysis look like?

Last night my partner, for whom computers always seem to fail, made a comment about having to “tap the secret window and turn three times widdershins” in order to get her facebook app to work. She is one of those people for whom computers hold slightly mystical properties, because they never seem to work properly. It’s quite hilarious at times because they are guaranteed to work the moment she calls me to see the problem, creating that common feeling some people have that computers aren’t really scientific objects and/or that some people exude some magical property of technological disruption.

When you stand back and think about what computers can do, they are a kind of magical phenomenon. Before their invention, modern mathematical calculations had to be performed by teams of – mostly young, female – assistants called “calculators,” who amongst other things played an important role in scientific support during world war 2. Now tasks that were beyond the skill of even those teams of dedicated young women are conducted in a microsecond with a single click. Plus our computers can be used to get oracular information (predicting the weather, or changes in our leaders), to scry on people from afar, and even to make important military and investment decisions, as well as cursing strangers with death from the skies (drone warfare).  They daily work a kind of magic.

My partner’s comment, then, had me imagining what it would be like if computers’ semi-magical powers were actually implemented through magic, and those who used computers had to actually invoke some kind of little magic to control the greater magic behind the screen. Perhaps the spell would require some verbal, somatic and material components, and instead of having a keyboard one would have a little altar or magical bench. Then, instead of accessing a remote website to learn tomorrow’s weather, one is instead using the magic box on the magical bench to contact a distant infernal entity, which gives you its oracle. The distant magic is great, but your workstation is simply a minor magic that opens a conduit to the greater power.

So to find a train time you simply need to invoke some boggart living under the ground, who is a train enthusiast: perhaps you sprinkle some iron filings inside a pentacle on your workbench, and it opens a link to the distant boggart, who grudgingly answers your question. The weather means contacting a sylph, so perhaps you have to stand naked in front of your computer screen and read her a love poem: in exchange she will give you predictions of the weather for the next three days with perfect accuracy – unless your reading of the poem was poor, in which case you will get no warning about the need for an umbrella.

But then, what would a statistical analysis look like? For this I have to call on a greater entity, one capable of processing huge amounts of information in a short time. Imagine I’m at work, analyzing four years of inpatient data from the NHS: that’s about 64 million records that I want to apply a multi-level poisson regression model to. Thanks to the great intellect of Sophia Rabe-Hesketh, this will take a couple of hours on a decent modern workstation (if it has the RAM to load the records). But it involves a truly stupendous level of calculation, and the invocation of a power capable of conducting such calculations would surely require some more powerful magics. So there I would be, chalking out a magic circle on the floor of my office while chanting an ancient and rhythmic calming chant; then I light some incense, step inside the circle, and place some drops of my blood on a specially-prepared silver plate. Next to it I place a research plan, written perhaps on the flayed skin of a heretic who himself had great mathematical ability, and this will call up the demon that processes my data; but this demon will only produce results according to the exact wording of my fiendish research plan, so any ambiguities or confusion in the methods section will waste the magic. Once the demon has been apprized of the plan, it sets to work, and I have to sit in the circle, slowly reciting the value of pi. If I have chosen an oracle who is to weak, or I have misspecified the research plan, then perhaps I will run out of digits of pi before the oracle finishes their  work – this is kind of like when GLLAMM fails to converge, or I only get the generalized linear model approximation to the true values. But if I have chosen my oracle well and recite the value of pi at just the right speed, I will be handed the results.

Of course, I could run a pre-ritual with a minor imp, to learn the value of pi to a greater number of decimal places, because perhaps this is cheaper than providing the necessary propitiations to the fastest and most powerful oracles. Thus research planning will involve trade-offs not between sample size and power, as they do now, but between magical reagents and accuracy – I can choose cheaper reagents, but will get an oracle who provides results with less precision. My grants will not be funded in money for better computers, but in scraps of skin, bundles of rare herbs, and piles of powdered silver and specially-prepared chalk. Perhaps there will be the odd quest to find rare ingredients (far more interesting than a conference trip!) or funding to send adventurers after them in my stead.

Anyone passing my office while I’m doing this work would see this strange sight, of me sitting cross-legged in the circle, dripping blood onto that silver plate and reciting numbers, while strange colours swirl on the screen on my workdesk, and fragrant incense drifts around the room. Perhaps the results will not be printed out, but will be just delivered into my head, so no one will know when I’m finished whether I actually ever really contacted the oracle, or just shuffled around muttering mumbo-jumbo and made up a conclusion.

So really, maybe not so different from the way things are done now, in the end.