Plain old conceit

In preparation for a post on the political origins of anti-vaccination ideology, I want to make a point about the way that the ordinary public interact with scientists. My last post on anti-vax and Republicans has been linked to by a climate change blog, and on that blog one of the commenters is making big claims about what the public should do to understand climate science. In particular he or she says:

‘Trust the science’ is a very ambiguous statement. People should follow the scientific method, but I think what you mean is closer to ‘people should blindly believe what authority figures in science tell them’. This can get very dangerous and appeal to authority is not part of the scientific method. Rather the scientific method involves questioning authority and skepticism.

I think this is an incredibly unreasonable and unrealistic depiction of how people should interact with scientists, and is an advanced form of epistemological nihilism. I want to give a specific example of why, though I’m sure there are many others.

My father left school at 15 to take up a trade as a typesetter, my mother left school at 13 to work in a cake shop and her father left school in Spain at 15 to fight fascism (he subsequently became a forester in England after the war and devoted his spare time to raising a family and learning English). Not only did all my forebears leave school before they got a chance to receive an advanced science education, but they went to school before computers were common, when quantum mechanics was still in its learning stage (e.g. before Bell’s Inequality) and in my grandfather’s case before the invention of the microwave, the guided missile, or a man on the moon. The idea that my parents and my grandfather can “follow the scientific method” – indeed, that they even know what it is – is ludicrous, as is the idea that they have any kind of skill or capacity to question authority where science is concerned. Furthermore, the idea that they should work 9 hour days of physical labour and then come home and devote their time to learning about these things in order to understand policy about important issues like global warming is both unreasonable and, frankly, insulting. If you can’t explain this shit to these people in a way they understand, don’t get uppity that they aren’t willing to put the time into learning your shit properly. They’re busy, and their reasons for being busy are just as valid as yours. I would go further and say that most scientists don’t have a clue about how typesetting works (that’s why LateX was invented!) and would get quite miffed if they got an email from their publisher saying “we can’t be bothered fixing up the typesetting in your paper so that it can be legible in print. You should have learnt this stuff. Your paper is going to look like shit.” Life is too big to learn everything, and this is why we have specialization. Expecting everyone to engage with your trivial little skillset[1] is called “arrogance” in the real world.

So, I don’t think this is how ordinary people should interact with science. In a functioning society, ordinary people should be able to assume that scientists are working for the good of all, that government funds are expended on science in a way that is somehow subject to reasonable oversight and judgement, that experts translate this stuff into public policy, and that ordinary people with no science background can trust that their political representatives are handling the science in a way that is open, rational and coherent. With proper governance structures they can have mild confidence that science is being done ethically and to certain basic standards of intellectual rigor, and with a professional and well-run media they can have some confidence that the popularization of science doesn’t also debase and ruin it. In this sense, while “appeal to authority” is not part of the scientific method it is very much part of how we the public interact with and make decisions about the implications of the scientific method for policy. If 97% of climate scientists say the earth is warming through human influence, then the public should be able to be confident that this means mitigation needs to be debated. If doctors say vaccination for mumps, measles and rubella is needed at a certain age, we should be able to go along with it because we trust that those doctors came to that position through a transparent, ethical process of scientific inquiry, and this position only entered practical health policy through a well-governed and robust process of policy development. We should not have to follow the chain of logic, statistics and biological science that led to this decision in order to support it.

This process by which people actually engage with science in practice is the reason that ideologically-determined views of science are both unavoidable and necessary. Left-wing or working class people will trust that the science minister they voted in from the labour party interprets science into policy in a way that is both a) in the interests of all of society and b) in their class interests; similarly right-wing or rich people will assume that the science minister they voted in will interpret science into policy in a way that represents their love of the free market and eating puppies while kittens cry. This is both inevitable and right (except for the kittens crying). The breakdown of political responses to climate change and the acrimonious debate surrounding it has not occurred because ordinary people failed to follow the scientific method (or did, and found it wanting); it has occurred because a certain part of the political scene in the rich west has abandoned its responsibilities to its electors and started lying to them about a fundamentally important issue. This is a failure of governance and ethics on the part of our leaders, not a failure of ordinary people to engage in scientific critique.

Provided their political representatives hold their best interests at heart, people can be as ignorant as sin about science, and still see good scientific outcomes without even knowing that there is such a thing as “the scientific method.” Conversely, when our political representatives go feral on us and refuse to act on science that is compelling and urgent, it doesn’t matter how educated and engaged ordinary people are – we’re toast. This is why the global warming issue is not being addressed, and in my opinion (as I hope to show in my next post) there is absolutely no lesson to be learnt about vaccination policy from the clusterfuck that is global warming. And in any of these scientific debates, the expectation that ordinary people should engage with science rather than judging it through reference to their authority figures is simultaneously arrogant, unrealistic and ignorant.

fn1: “Skillset” is one of my most hated words, and I use it here to be deliberately insulting

This week’s issue of the New England Journal of Medicine has a perspective piece by a doctor from Kentucky, describing the changes wrought by Obamacare in its first year. The doctor, Michael Stillman, is writing from a clinic serving a relatively poor area with limited health access: 60% of the doctor’s patients had no health insurance in the year before Obamacare’s introduction. It’s a short but quite powerful piece, and worth reading if one wants to get a sense of the transformative effect of Obamacare for the working poor. Dr. Stillman writes:

Last year, I encountered a patient with widely metastatic colon cancer whose diagnosis had been delayed because of lack of health insurance. He had clearly become ill at the wrong moment in our commonwealth’s history. Before Kentucky Governor Steve Beshear decided to implement the Affordable Care Act (ACA) and accept federal funding for Medicaid expansion, the 60% of my clinic patients and 650,000 Kentuckians who lacked health insurance received disjointed and disastrous care. They could be seen in subsidized facilities and be charged for their visits on a sliding scale, but they were asked to pay in advance for most diagnostic tests and consultations. Many of them avoided routine and preventive care — and worried that a medical emergency would leave them bankrupt.

This describes a pretty terrible situation in the world’s richest nation, and something that could hardly be imagined in any other developed nation. Dr. Stillman’s first sentence in this paragraph has a reference to another perspective he wrote last year, Dead Man Walking, in which he describes the case of this patient with colon cancer in more detail:

We met Tommy Davis in our hospital’s clinic for indigent persons in March 2013 (the name and date have been changed to protect the patient’s privacy). He and his wife had been chronically uninsured despite working full-time jobs and were now facing disastrous consequences.

The week before this appointment, Mr. Davis had come to our emergency department with abdominal pain and obstipation. His examination, laboratory tests, and CT scan had cost him $10,000 (his entire life savings), and at evening’s end he’d been sent home with a diagnosis of metastatic colon cancer.

The year before, he’d had similar symptoms and visited a primary care physician, who had taken a cursory history, told Mr. Davis he’d need insurance to be adequately evaluated, and billed him $200 for the appointment. Since Mr. Davis was poor and ineligible for Kentucky Medicaid, however, he’d simply used enemas until he was unable to defecate. By the time of his emergency department evaluation, he had a fully obstructed colon and widespread disease and chose to forgo treatment.

This is not the fate that should be allotted to the working poor. Obamacare will change this situation for a large number of Americans: in Mr. Davis’s case, Obamacare made him eligible for Medicaid, so had Obamacare been passed just a little earlier he might have been able to diagnose and treat his cancer earlier; or at the very least, would not have used up all his life savings on a mere diagnosis. The author reports that in Kentucky Obamacare has expanded access to 430,000 previously-uninsured Americans, and the same process is being repeated across the country. The Obama administration itself is forecasting a total of 9 million people will gain health insurance through the ACA, and I earlier reported on the first assessments of its impact from the Commonwealth Fund. Although most of the rest of the world agrees that Obamacare is flawed and doesn’t go far enough towards universal health coverage, this is still a huge achievement and any politician who opposes it, or any pundit like Michael Cannon who deploys disingenuous arguments to destroy it on ideological grounds should be seen as the wrecker and low-life that they are. Responsible politicians of any political stripe should be focusing on improving it, not destroying it, but sadly this is not the way the American political system seems to work.

Dr. Stillman finishes his article with this little observation:

I was once uncomfortable discussing politics with my patients, but now I routinely ask them if they are registered to vote and remind them that certain candidates do not support the legislation from which they have so palpably benefitted.

This little sentence should have Republicans and other opponents of Obamacare worried. If they can’t destroy it in the Supreme Court soon, they’re going to have to hoodwink 9 million Americans at the next election. The fact that they would even consider such a reckless and destructive policy is a depressing indictment of the cruelty and shortsightedness of the modern Republican party. I hope their intransigence on this issue destroys them at the next election, and Obamacare survives to be improved and consolidated, rather than dismantled and discredited. For where will the working poor be without it?

Over the past few years I’ve looked at a lot of the probabilistic and statistical aspects of specific game designs, from the Japanese game Double Cross 3 to Pathfinder, including comparing different systems and providing some general notes on dice pools. I’ve also played various amounts of World of Darkness, Iron Kingdoms, D&D, Warhammer 2 and 3, and some Japanese systems, that all have quite diverse systems. Given this experience and the analytical background, it seems reasonable to start drawing it all together to ponder what make for some good basic principles of RPG system design. I don’t mean here the ineffable substance of a good RPG, rather I mean the kind of basic mechanical details that can make or break a system for long term play, regardless of its world-building, background and design. For example, I think Shadowrun might be broken in its basic form, to the extent that people who try playing it for any length of time get exasperated, and this might explain why every gaming company that handles Shadowrun seems to go bust.

So, here is a brief list of what I think might be some important principles to use in the development of games. Of course they’re all just my opinion, which comes with the usual disclaimers. Have at ’em in comments if you think any are egregiously bad!

  • Dice pools are fun: everyone likes rolling handfuls of dice, and the weighty feeling of a big hand of dice before a big attack really makes you feel viscerally there, in comparison to a single d20
  • Big or complex dice pools suck: Big dice pools can really slow down the construction and counting parts of rolling a skill check, but on top of this they are basically constructing a binomial distribution, and with more than a couple of trials (dice) in a binomial distribution, it’s extremely hard to get very low numbers of successes. So large and complex dice pools need to be limited, or reserved for super-special attacks
  • Attacks should use a single roll: Having opposed skill checks in combat means doubling the number of rolls, and really slows things down. Having cast around through a lot of different systems, I have to say that the saving throw mechanism of D&D is really effective, because it reduces the attack to one roll and it makes the PC the agent of their own demise or survival when someone attacks them. On the other hand, rolling to hit and then rolling to damage seems terribly inefficient
  • Where possible, the PC should be the agent of the check: that is, if there is a choice in the rules where the GM could roll to affect the PC, or the PC could roll to avoid being affected by the GM, the latter choice is better. See my note above on saving throws.
  • Efficiency of resolution is important: the less rolls, counts and general faffs, the better.
  • Probability distributions should be intuitively understandable: or at least, explainable in the rules – and estimates of the effect of changes to the dice system (bonuses, extra dice, etc.) should be explained so GMs can understand how to handle challenges
  • Skill should affect defense: so many games (D&D and World of Darkness as immediate examples) don’t incorporate the PC’s skills into defense at all, or much. In both games, armour and attributes are the entire determinant of your defense. This is just silly. Attributes alone should not determine how well you survive.
  • Attributes should never be double-counted: In Warhammer 3, Toughness determines your hit points and acts as soak in combat; in D&D strength determines your chance to hit and is then added again to your damage. In both cases this means that your attribute is being given twice the weight in a crucial challenge. This should be avoided.
  • Fatigue and resource-management add risk and fun: Fighting and running and being blown up are exhausting, and so is casting spells; a mechanism for incorporating this into how your PCs decide what to do next is important. Most games have this (even D&D’s spells-per-day mechanism is basically a fatigue mechanism, if a somewhat blunt one), and I would argue that where possible adding elements of randomness to this mechanism really makes the player’s task interesting. But …
  • Resource-management should not be time-consuming: this is a big problem of Warhammer 3, which combined fatigue management with cool-downs and power points. Too much!
  • The PCs should have a game-breaker: we’re heroes after all. Edge, Fate, Feat points, Fortune … many games have this property, and it’s really useful both as a circuit-breaker for times when the GM completely miscalculates adversaries, and as ways for players to escape from disastrous scenarios, and to add heroism to the game
  • Skills should be broad, simple and accessible: The path of Maximum Skill Diversity laid out in Pathfinder is not a good path. The simplification and generalization of skills laid out in Warhammer 3 is the way to go.
  • Wizards should have utility magic: the 13th Age/D&D 4th Edition idea of reducing magic to just another kind of weapon is really a fun-killer. The AD&D list of millions of useless spells that you one day find yourself really needing is a much more fun and enjoyable way of being a wizard. It’s telling that D&D 5th Edition has resurrected this.
  • Character classes and levels are fun: I don’t know why, they just are. Anyone who claims they didn’t like the beautifully drawn and elaborate career section of Warhammer 2 is lying. Sure, diversity should be possible within careers but there should be distinction between careers and clarity in their separate roles (something that, for example, doesn’t seem to actually be a strong point of Iron Kingdoms despite its huge range of careers). At higher levels characters should really rock in the main roles of their class
  • Bards suck: they just do. Social skills should be important in games, but elevating them to a central class trait really should be reserved for very specialized game settings. Bards suck in Rolemaster, they suck in D&D, they suck in 13th Age and they suck in Iron Kingdoms. Don’t play a bard.
  • Magic should be powerful: John Micksen, my current World of Darkness Mage, is awesome, but mainly because he is cleverly combining 4 ranks in life magic and 3 ranks in fate magic with some serious physical prowess and a +5 magic sword (Excalibur, in fact!) to get his 21 dice of awesome. Most of the spells in the Mage book suck, and if you made the mistake of playing a mage who specializes in Prime and Spirit… well, basically you’re doomed, and everyone is going to think you’re a loser. Mages should be powerful and their powers – which in every system seem to come with risk for no apparent justifiable reason – should be something that others are afraid of. You’ll never meet a World of Darkness group who yell “get the mage first!” What’s the point of that?
  • Death spirals are important: PCs should be aware that the longer they are in a battle, the more risky it gets for them. They should be afraid of every wound, and should be willing to consider withdrawal from combat rather than continuing, before the TPK. Death spirals are an excellent way to achieve this combination of caution and ultra-violence. Getting hit hurts, and players should be subjected to a mechanism that reminds them of that.

I don’t know if any game can live up to all these principles, though it’s possible a simplified version of Shadowrun might cut it, and some aspects of the simplified Warhammer 3 I used recently came close (though ultimately that system remains irretrievably broken). Is there any system that meets all of these principles?

Will Self has declared war on George Orwell, anointing him the “Supreme Mediocrity” in a mediocre essay distinguished only by its needless use of the word “lucubrations.” For those of you Americans out there who know of Orwell but have never heard of Will Self (I wonder how that came about?), Self is a novelist who is something of a darling of the British “lovie-liberal” inner city late-sipping champagne socialist set (and Guardian readers, where those two don’t overlap), who is famous for a pretentious writing style that uses too many fancy words. If you doubt the quality of my judgment, try anything from this extract from his new book:

Claude experiments, turning his whole head because his eye sockets …are filled with gritty sand, – he sees the sea green to aquamarine to cobalt blue to silver blue to silvery to silver white then vanish completely as …I push my head up her skirt… Mm–mm, finest ear-protectors a fellow can get – flesh-filled nylons fitted snug to the head and dried with talc… The kid is maybe forty feet down now, yet his dancing plummeting body can still be clearly seen

Unadulterated bullshit, or quality literature? You be the judge.

Will Self, of course, is emblematic of a vanguard of … how can we put this nicely? … dickheads who have managed between them to kill much of the joy of the English novel over the past couple of decades. Over that time I’ve met a few people – some literature majors – who have told me they’ve basically given up on reading novels, because for example “I couldn’t give a toss about another rich white person’s shallow imaginary world,” or because “it’s all showmanship and self-importance, there’s no joy in it” or just because “Oh my god what a pack of tossers the literary world has become.” The Man-Booker prizelist is an example of this: supposedly composed of the best writers of English in the Commonwealth, it is actually a shortlist of writers to avoid if you want to read a good book, and for all the reasons those who eschew the modern novel have given me. The Man-Booker prize winners are a small clique of stuck-up novelists who write to impress each other, rather than to extend the joys of the English language. That’s fine, soggy sao is a fine public school tradition and if that’s what gets you off then by all means, do your worst … but must you demand a prize for it? Will Self, of course, has been shortlisted for the Man-Booker, and if he keeps randomly resampling his thesaurus, eventually he’ll win. He’s a pretentious writer who knows how to use long words and wide vocabulary to hide a lack of ideas or talent.

In short, the antithesis of everything George Orwell stood for. With a new book out, so of course it’s a prime time to lay the boot into one of his dead idols.

It’s telling that Self opens the essay with a quote from Chesterton, an author who would very much fall into Orwell’s camp where opinions about pretentious writing are concerned. He then lays out a rather petty theory of why British people laud mediocrities, against all the evidence that the real heroes of the British are not mediocre at all (Churchill and Thatcher, mediocre? “Individuals who unite great expertise and very little originality – let alone personality”?). This is an example of the kind of theorizing a vapid writer-at-large can pull out of thin air, gloss with a few fancy words and toss about without regard for truth or sense, but it is certainly no kind of comment on Britain or how the British construct their idols. So from this lead-in to the conclusion that Orwell is a mediocrity, we have a logical failure built on a failed premise. He then manages to find two paragraphs of his entire essay to actually discuss what might be wrong with Orwell’s work, though again here he bandies his opinion about without any evidence or logic. No support for the claim of “obvious didacticism” and no discussion of what makes an “unadorned Anglo-Saxon style,” or indeed what might be wrong with such a thing. Make no mistake, this is how famous people troll – with fact-free assertions they can get away with because they’re given free rein over the BBC’s essay pages.

But it’s from this piss-poor effort at criticizing Orwell’s actual written work that Self then goes on to make his biggest fail in this essay. Citing Orwell’s famous essay on straight English, Self spends several paragraphs showing exactly the extent to which he failed to understand that essay. He says first of all that it is wrong, and then explains why: because language grows and mutates, and is defined by how people use it and why they use it, and attempts to dictate centrally how language should be used are morally wrong, whether enforced by George Orwell or the Ministry of Truth.

The thing is, Orwell’s essay has nothing to say about how language mutates or is reformed, he doesn’t give a toss about “African American Vernacular English” (oh Will, why did you choose such a pathetically PC-baiting example?!) or how much English consumes other languages and reconstitutes them to feed its voracious needs. His famous essay is about the dangers of using whatever language is at your disposal in disingenuous or deceptive ways, to hide what you mean rather than to say it. Furthermore, the primary value of the essay is not in what it says to authors and fiction writers, but to all those other users of English who are clearly beneath Self’s gaze: scientists, bureaucrats, business people, journalists, politicians and the like. Of course Will Self doesn’t have to actually work, except inasmuch as he is overpaid for occasionally dressing up shallow and simple ideas so that they look interesting. But for the rest of us – those of us for whom English is important to express to others the content of our real jobs – the ways in which English can be abused to hide meaning are very important and require a lot of skill to grasp.

For example, if you’re going to be paid a crapton of money to talk about some dude watching a kid drown while imagining going down on his nanny, it frankly doesn’t matter how much florid language you use, you’re just trying to make a dumb non-sequitur look interesting. But if you’re writing a textbook on Bayesian statistics and you have to explain the interpretation of a Bayesian credible interval compared to a frequentist confidence interval, you already have a lot of jargon to juggle and you need to think very carefully about how to calibrate your prose so that it is comprehensible between all the technical language. If you have to polish your language down to a couple of thousand words expressing a huge research task, you need to be very careful about when and how you embellish your language. And conversely, if you want to reassure the American public that you aren’t water-boarding innocent people when in fact you are, but you don’t want to be caught lying, you need to very carefully use language deceitfully in order to get away with it. Orwell’s concern is not with whether someone lies in the Queen’s English or Ebonics; his concern is with the deceitful use of language to lie or obfuscate, or the accidental embellishment of language in a way that makes it incomprehensible.

That Will Self didn’t understand this fundamental point of the essay he presents as evidence for the mediocrity of Orwell’s writing style would be hilarious if it weren’t so pathetic. And roping in a poorly-understood version of Chomsky to help in the task of getting it wrong is really not a good look either. Nor is mistaking the clear elucidation of a terrible dystopia as “didacticism” just because you think that making a thing clearly understood is the same as talking down to someone like an authoritarian teacher. This says more about Self’s insecurities than it does about Orwell’s writing style.

Self finishes his ignorant little rant by talking about how poorly he views “those mediocrities who slavishly worship at the shrine of St George.” For a lot of people, Orwell’s essay on how to use English has been a guiding light in a world of business jargon, weasel words, dissembling and deceit; many of us work in jobs where English is important for expressing the actual content of our work, rather than as a way of dressing up our shallow, imbecilic imagination for a sycophantic crowd of fellow-travellers. For us, work like Orwell’s essay on politics and the English language is a clear guide on how to improve our writing so that we can express complex ideas clearly and accessibly; and also a workbook on how to identify when people are lying to us through creative use of language. To Will Self, however, we are “mediocrities.”Faced with a choice between the “Supreme Mediocrity” and an advanced thesaurus-user who can’t even win the Man-Booker prize, I think I’ll stick with Orwell for future writing advice. And I suspect that 50 years after Self’s death, the majority of the English-speaking world will be making the same decision as me.

I have recently been spending a lot of time around (white) foreigners in Japan, much more than usual, and a recent experience with a particular pair of them has helped me to understand a kind of increasing discomfort I have with the way westerners interact and the way they perceive Asia. I guess being largely removed from the brute force of western interactive style, I notice it more when it is laid before me in its full glory. I want to report on some of my feelings about this here.

I have been travelling – not by choice – with a German man and his wife, in their late 20s. The woman is shy and the man won’t shut up, but not just in an annoyingly tiring way. He speaks over and for his wife in a terribly obvious way, even about things asked directly of her about her. But worse still, if you try to short-circuit this behavior by addressing her directly, she looks to her husband and waits for him to speak. For all the presence she has in the group, she might as well just be veiled and sat behind him. His behavior is really depressing and irritating and her quiescence – I would like to guess it is due to exhaustion but I am worried from her manner that it may actually be a preference – is saddening. Furthermore, this man feels himself an expert on Japan having read some stuff before his (first) visit, and needs to prove himself knowledgable on all things. He is also very convinced of the importance of expressing his own opinion, and me and my friend being not averse to bloviating ourselves, the time we spend together is a constant frustrating round of opinion-exchange and fact-boasting. I can’t help getting sucked into this but I get so tired of it. Of course being a westerner myself I’m really at home in and good at this type of conversation, but 7 years out of the loop means I now look down on myself engaging in it and don’t reflect well on myself.

I believe this style of fact-boasting interaction is just a form of empty chest-beating, it doesn’t change anything about the world or each others’ opinions – it isn’t communication in any sense that the word is really understood. A phenomenon I notice a lot around foreigners and that I think is related to this is their complete inability to take in other cultures (most especially Asian cultures) without judging them. In sum this behavior has me thinking: I really hate Western so-called “enlightenment.” This guy has mentioned once or twice Japan’s reputation for sexism, in the usual tone and context as to imply that we Westerners are so enlightened and over that shit (his country even has a female Chancellor, for god’s sake!) but all of his behavior stinks of sexism, classic unconscious sexism of the patriarchal ignorance kind. He’s not alone in this: I can’t count the number of times western men and women have spoken to my partner and I as a couple but looked only at me when talking; the number of western women who completely blank my partner and only engage with men, or worse still friend me but not her after a meeting; the western men who speak on one side of their mouths about how sexist Japan is while bragging out of the other side of their mouths about how sexually and domestically submissive their Japanese woman is; the number of foreign men here who have no use for Japanese men and never engage with them; the number of foreign men whose Japanese wives are “not like other Japanese women” because they aren’t “weak”; the number of foreigners who judge anyone who is not loud and opinionated to be stupid.

I hate the way westerners have to judge everything around them as if everything around them is waiting for their shallow, half-considered opinions to affirm its validity. I hate the way every half-baked racist trope that they conjure up to misinform them about the value of behavior they witness but don’t understand is reinterpreted as proof of their own superiority. I hate the way this happens as they blow their noses; stand in the way of patient, polite Japanese people; put their dirty feet places they shouldn’t; speak too loud; and generally explode into every space they are currently judging like a cloud of massive, usually fat and underdressed obnoxiousity.

I hate the way every time anything untoward or unexpected happens white westerners immediately start wondering if it might be because they’re foreign and experiencing racism. Invariably they aren’t, but it reveals so much about their implicit recognition of how hard it is to be foreign in a white country. I hate how they project every seedy element of their own culture onto others and then when the confirmation bias kicks in, interpret it as proof that their own culture is superior. Spare me this self-aggrandizing cultural blindness!

I’m sure many of the feminists are familiar with and quite irritated by this kind of hypocrisy. Over here I see it leavened with racism and colonialism, and I am thoroughly and heartily sick of it. Westerners aren’t enlightened or any more advanced than much of the rest of the world, we’re just full of our own self-righteous opinionated rubbish. And so much of what is constantly passing for opinion and argument in our much-vaunted “individualist” culture is just ill-informed, hypocritical cant with a healthy side serve of colonialism. I really, really wish that as a culture we could just shut up and listen for five minutes of our incredibly short, dirty and plague-ridden history. I would like to parley this into some kind of set of rules for westerners visiting Japan – not rules about how to treat the locals, but about how to understand them and how to take away from Japan something more than just “ooh, cute shrines and everyone’s polite but oh hasn’t everything I already ‘thought’ about Japan just been confirmed in spades!?”

I can’t though, because this is just a rant.

On Monday I was required to monitor at the Tokyo University undergraduate entrance exams. I shepherded 60 terrified 17 year olds through a 2.5 hour Japanese language test and then a 100 minute maths test. These tests were part of a two day examination process for those want to enter the humanities faculty of Tokyo University. About the Japanese test I can say nothing, but the maths test interested me, and can be found online (in Japanese) at the Mainichi Shinbun newspaper. In order, based on my feeble attempts at translating the exam, the four questions were:

  • A straightforward but nasty calculation of the properties of a line intersecting with a cubic function, including elucidation of all minima and maxima of the products of the lengths of two line segments
  • A geometry question with two proofs
  • A constrained linear programming problem
  • A simple Markov model with a slight twist

The students had 100 minutes, and to their credit quite a few of the students managed all four, though a lot also stumbled and didn’t get past two. I would say that for a well-trained student with good maths skills, these four questions can all be done inside their allotted 25 minutes, but it’s a pretty risky process – even a small error at the start, or misconception of how to do the problem, and you have basically lost the whole question because you only have time to attack the problem once. And these problems are probably about the same level of difficulty as the questions on a standard year 12 maths exam in Australia – where usually we would have three hours.

But these questions were for the Humanities Faculty of this university. If you want to study Japanese literature at Tokyo University, you first have to get through that 100 minutes of high level mathematics. It says something, I think, about the attitude of Japanese people towards mathematics, and towards education in general, that they would even set a mathematics test for access to a Humanities Faculty; and it says even more about the national aptitude for maths that the students could tackle this exam.

At about the same time as these exams were being held, the Guardian and the Sydney Morning Herald released articles slamming the mathematical and science abilities of the average student in the UK and Australia, respectively. The Guardian reported on a new study that found English star students were two years behind their Asian counterparts in mathematics, with 16 year old English students at the same level as 14 year old Chinese. The study also found that

The research also found England’s most able youngsters make less progress generally than those of similar abilities across the 12 other countries studied. The other countries studied were Singapore, Hong Kong, Taiwan, Japan, Australia, Slovenia, Norway, Scotland, the US, Italy, Lithuania and Russia.

Meanwhile, the Sydney Morning Herald reported on a new study showing that the proportion of students doing mathematics is falling fast, with apparently only 19% of students studying maths, science or technology in their final year of school, and a rapid fall in mathematics enrollments amongst girls especially. The corresponding figure in Japan in 2002 was 64%.

So is this a problem, why is it common to the English speaking world and viewed so differently in Asia, and what can be done about it? Obviously as a statistician I think this is woeful[1], and it certainly is my personal opinion that understanding mathematics is a good thing, but is it bad for a society as a whole to neglect mathematics education? I don’t know if that’s objectively verifiable. So let’s skip that question, assume for now that improving the number of people taking mathematics is good, and just jump onto the question of why it is unpopular in Australia, and why the British are so bad at it.

First, I would like to dispute the possible explanation provided in the Guardian article by “the researchers”:

In east Asian cultures education has historically been highly valued. This can be seen not only in teachers’ high salaries, but also in the heavy investment of families in private tutoring services

While it may be true that “social and cultural factors” affect maths achievement, the idea that Asians are better at maths because they value education more highly is a very weak one. If this were the case, would it not also be the case that Japanese would universally be better at foreign languages than the British or Australians? Japanese get a long exposure to English teaching but are generally woeful at it, despite all the money they sink into private tutoring services. No, there’s something else going on here, something about the Asian approach to maths and the way it is taught that is important.

It is certainly the case that private tutoring services need to be considered in the mix. When comparing a 16 year old English student to a 14 year old Japanese student, for example, you are comparing someone who does a 9 – 5 study day with very long winter and summer holidays against someone who does an 8 – 8 study day with two-week holidays, and who gets 2-on-1 or small group tutoring in key subjects for up to 3 hours a day, and on weekends. This process starts at age 10 and really ramps up at about age 15-16, just when the linked article finds the biggest gap between English and Asian students. It’s also the kind of process that benefits the “brightest” students most, and would explain the gap very nicely.

It may be that if the UK wants to compete with the sleeping giants of Asia on basic educational outcomes, it’s just going to have to face up to a simple fact: British students need to study harder. A lot harder.

There are some more nebulous cultural factors that come into play, however, and I am going to go out on a limb here and name a few factors in Japanese society (the part of Asia I am familiar with) that I believe make Japanese so much better at maths than their western counterparts.

  • It isn’t about native talent: A pet hate of mine about western approaches to mathematics is the idea that some people are talented at it, and most people aren’t. I don’t think this is true at all, and I think it’s not something that Japanese believe very strongly. The reality is that getting good at maths is a long, hard slog that involves a huge amount of repetition of basic skills (things like completing the square, substitution, differentiation, interpreting graphs, sign diagrams, etc.) – just like learning a language. Sure, solving maths problems requires creativity and intuition, but these are only of any value if you know the tools you can apply them to, and are familiar enough with those tools to recognize when and how to use them. Mathematics – and especially high school mathematics – is a process of drilling, drilling, drilling, and I think that Japanese recognize this. In Japan the default assumption is that if you pay attention at school and do your homework, you will be good at maths. Sure, they recognize that advanced maths requires extra commitment and talent, but there is a fundamental assumption here that the broad body of maths (up to and including differentiation, integration, limits, and basic probability theory) are things that anyone can learn.
  • The teacher is important: the flip side of the idea that education is important is an increased stress on the value of the teacher, and their role as a guide. The role of the guide is also viewed very differently if they are teaching something that they believe anyone can do, compared to if they are teaching a subject that everyone believes is impossible for most mortals to comprehend. Find me a westerner under the age of 30 who is “terrible at maths” and I will show you someone who was humiliated by an arrogant maths teacher at a crucial time in high school, usually around when they were 14. I was in the bottom class in mathematics when I was 14, expecting to drop out as soon as possible, until a good teacher put some time into teaching me, and I found that I really loved it. In Japan, teachers can be bullies and they can be cold and hard, but I would also argue that they have a much greater burden of personal care and responsibility placed on them compared to western teachers, and the failure of their students is treated more like a professional failure (rather than due to the student’s personal talents) than it is in the west. I think this is especially important with mathematics, because when you don’t get it it really hurts – like a kind of itching in the back of your brain – and the failures pile up rapidly. Just a single year between 12 and 14 in which you give up on maths is enough to make all the subsequent years ever more challenging, meaning the damage and the attendant confidence failures compound.
  • Being nerdy is cool: In Japan, it’s okay to be a nerd, and being good at mathematics is admired and respected. It’s virtually unheard of to find someone here who looks down on a man who can do maths, or thinks that it is beyond the female brain, or thinks that being interested in mathematics is weird. Furthermore, the nerd world in Japan is much more gender neutral than in the west, so there’s nothing unusual about girls doing maths. Good mathematics skill – up to and including being able to rearrange equations or solve systems of equations, for example – is not seen as a weird foible, but as an admirable sign that you are a rounded human being.
  • There is a social expectation of mathematical skill: In addition to nerdiness being much more acceptable, the range of mathematical abilities that qualify you as a nerd in Japan is much more esoteric and advanced than in the west. There is a general expectation that ordinary people can solve maths problems, that they understand the basic language of mathematics so that even if they can’t solve a problem they know roughly what it is and where it sits in the pantheon. Parents assume that their kids will learn mathematics, and don’t dismiss it as the too hard subject that only the special or the weird get ahead in. Whereas in Australia having a kid who is good at maths is unusual, in Japan it is unusual (and embarrassing!) to have a kid who is not good at maths.

I think these properties add up to a society in which mathematical achievement is encouraged and widespread. I think that Australia and the UK need to change some cultural factors so that the intellectual and educational landscape is closer to that in Asia if they want to keep up on mathematics and technology achievement – especially since China’s education system is maturing, and other Asian nations like Vietnam, Singapore and India are getting wealthier, with all the educational gains that implies. So what should Australia do?

  • Ditch the nerd-baiting: there’s something really wrong with the way the English-speaking world treats people who do nerdy things. I’m sure it’s mellowed a lot since I was a kid but it’s still there, the kind of ugly-four-eyes assumption about anyone who is interested in anything that isn’t sport or fashion. Until this weird attitude dissipates – and until the nerd world becomes more gender-balanced, to boot – it’s going to be hard to encourage the kind of cultural changes needed to make maths achievement standard across the board
  • Less intuition and initiative, more drills: I think it’s very sweet that maths teachers want to encourage their charges to think about the broader world of maths, about creative problem-solving, about applying maths to the real world, etc. But I think those are natural talents all humans possess, that cannot be unlocked without a robust background in the basic skills that make mathematics work. So leave the creativity for people who need it, and stuff kids’ heads full of “useless” rote learning of techniques and drills. It’s boring, but it’s essential to the bigger stuff. If you aren’t able to immediately see when and how to complete a square, then any problem which requires this basic technique is going to be beyond you, no matter how intuitive you are. Maths, possibly more than any other discipline, is built from the ground up, tiny block by tiny block, and all those blocks are essential. So ram them down every kid’s throat, and make every kid think that knowing the quadratic formula is not a test of some kind of obscure talent, but a basic expectation of every 12 year old
  • Force mathematics at higher school levels: When I finished school our balance of subjects had to include at least one science/technology subject, but it didn’t have to include maths. This is wrong, and part of the reason that so many students in Japan do mathematics is that you can’t get into a good university if you take this approach: every one of the better universities includes mathematics in its entrance exam. My personal belief is that completion of higher school certificates should require one foreign language, mathematics, and English. That leaves two other subjects to choose from, and guarantees that you have to do some kind of mathematics to the end of school. Not only will this very quickly lead to a society where entire generations of people are generally familiar with mathematics, it will also put a real focus on the quality of teaching at the earlier years, since any student who is doing badly in years 8 – 10 is going to fail their higher school certificate. [Probably this suggestion for a national curriculum is completely unreasonable, but at the very least students could be forced to do mathematics up until year 11, for example].
  • Make school more robust: The Japanese school system is about to shift to a “tougher” system that will include Saturday morning classes, because the previous system was considered “relaxed” compared to earlier years. This is, frankly, ridiculous, but so is the attitude towards education of most of the English-speaking world. Summer holidays are way too long and relaxed, there is a real lack of extension classes and tutoring, and expectations are altogether too low. Education isn’t valued enough, and until this changes anyone who wants their child to do better is going to be swimming against a strong current. Educational achievement is partly supported through the shared goals of a whole society, not just through the targets of individual families, and the expectations we hold for education are primarily set through the school system. So toughen it up – not in the sense of making teachers scarier or bringing back outdated “three Rs” educational styles, but by increasing the amount of time students spend at school, setting tougher standards for graduation and university entrance, making schools compete with each other (as Japanese schools partly do) and forcing parents to take greater responsibility for and involvement in their children’s education. This change isn’t specific to mathematics, but it would certainly help.

I don’t think there’s anything special about Asian students, or about Asian culture, that we can’t adopt. Asians’ mathematics achievements aren’t some kind of native or racial talent. It’s just a collection of attitudes towards education, mathematics and nerdiness that we can adopt if we want. Obviously there will be (potentially challenging) institutional changes required as well, and many people may judge it not worth the effort, but I personally think a world where everyone is good at mathematics is a better world, and we should be aiming for it. With these cultural changes maybe one day everyone will know the obvious thrill of being able to complete a challenging mathematics exam … and enjoying it!

fn1: Though obviously, the less people doing maths, the longer I will remain competitive in the marketplace …

Do any of these dickheads look virtual to you?

A couple of days ago Australia’s prime minister (PM) Julia Gillard found herself in the unprecedented position of having to host an extended news conference to hose down allegations of corruption from 17 years ago. A slightly abridged version of the news conference is available here, and it’s a barn-storming performance by Ms. Gillard that shows some of her finer qualities. My reader(s) from countries with a more timid political climate might like to feast their eyes on it as an example of how politicians should handle the idiots from the press.

This post isn’t about the press conference or Gillard’s finer qualities, but rather about the issue she was asked to address late in the press conference about how to handle what she (scornfully) describes as the “Americanization” of debate with its “lunar right, tea party fringe” and the role of new media in promoting a vitriolic atmosphere of political debate. The press conference itself is supposedly an example of how the internet has changed politics, since many of the allegations about corruption that Gillard has been forced to address have only been kept alive on the internet, by a couple of (apparently) misogynist and lunatic websites. A common question that consumes a lot of the (admittedly limited) processing power of the average journalist’s brain is whether the rise of internet communications and “new” media is corrupting political debate, and what journalists can and should do about it: near the end of the press conference Gillard summarizes this question more eloquently than any journalist could, and calmly points out to them what they should be doing about it – she doesn’t point out that her need to hold the press conference at all is an implicit proof that journalism has failed to rise to the task.

At the same time the Observer is running another of the seemingly endless run of journalistic pleadings about whether the blogosphere is responsible for the modern atmosphere of political hysteria. It cites some of the now famous research that claims the blogosphere fragments rather than facilitates political debate, and calls on some (imo) fairly trite stereotypes of the internet generation as self-serving and individualistic. But is blogging, and internet debate more widely, really the cause of this modern hysteria? Can journalists really stand above the fray and pretend to be offering a better, more reasoned or more “balanced” form of public debate? Or is this all smoke-and-mirrors aimed at hiding journalism’s corruption, and subsequent loss of control of the space of cultural and political discourse?

Returning to Gillard’s press conference, I can’t say I’m convinced that the problems she faces would just go away if a couple of vile and sexist websites were to disappear, and judging by her tone when she deals with a journalist called “Sid” from the Australian newspaper, she doesn’t think so either. Although the allegations she was confronting have been floating around for years, they were largely unknown to the wider public until 2007 – when the Australian published a defamatory version of them. And then 2010 – when the Australian published a defamatory version of them. And then last weekend, when the Australian published a defamatory version of them. Are we seeing a pattern here? On every occasion that it has chosen to move these allegations from the fringe blogosphere into public debate, the Australian has had to apologize and publish a retraction, and in 2010 it sacked the journalist who wrote the story. This newspaper – Australia’s only national newspaper – is on record as having declared a plan to destroy Australia’s environmental party, The Greens; its editor in 2003, Paul Kelly, traveled the country openly drumming up support for the Iraq war and calling anyone who opposed it cowards. The Australian is owned by News International – on the same weekend as the Observer was blathering about standards on the blogosphere, News International’s the Sun was publishing pictures of Prince Harry’s naked arse, presumably in the interests of free speech. This is the same News International that probably used illegal means to obtain and then broadcast a recording of Prince Charles telling his girlfriend he wished he was her tampon; the same News International that hacked a dead girl’s cellphone and deleted some messages, giving her parents false hope that she was still alive. The same News International – an American company, incidentally, run by an Australian – that probably also hacked the phone of the UK prime minister, and the families of a couple of soldiers who died in Iraq.

So is Gillard’s problem really with the blogosphere alone? As she observes in the press conference, in a world with more and more information people will tend to put more weight on the opinions of trustworthy mainstream sources; but when these mainstream sources simply regurgitate the opinions of “the nutjobs and misogynists on the internet” then the issue becomes bigger than just the opinions of some lonely wanker with a PC – the bigger issue is the judgment and respectability of the employees of a newspaper company that thinks hacking dead girls’ cellphones is a justifiable act. The reality is that journalists are cheap and easily bought, and they were running down the respectability of their own profession long before the internet made it possible for lonely misogynists to pile on.

But looking a little wider, beyond the issue of what journalists choose to confer legitimacy on, is the increasing nastiness of public debate really the fault of arseholes on the internet at all? The picture at the top of this page is from a rally against Australia’s carbon price. The poster at the back refers to Ms. Gillard, and is suggesting in quite a vile and sexist way that she is the sexual toy of the leader of Australia’s environmentalist party, Bob Brown. The man standing under that banner at the front, with the microphone, telling the demonstrators he agrees with them, is Tony Abbott, the leader of Australia’s opposition liberal party and Australia’s potential future PM. The woman next to him is Bronwyn Bishop, a senior and respected politician from that same party. One might call it merely an error of judgment, but it’s hard to say that they’re doing much to keep debate above board and polite when they choose these kinds of banners as their backdrop. Where are the bloggers in this picture? It wasn’t a lonely wierdo on the internet who called a feminist activist in America a “slut” for wanting contraception to be covered by health insurance – that was Rush Limbaugh, a major media figure. It was a Republican who decided to invoke the 10th Century fiction of “legitimate rape” in defending his anti-choice views; it was a politician, not a blogger, who put rifle cross hairs over pictures of American democrats (or was it their offices?); and there are more than a few birthers in the Republican party (indeed, in congress).

So is the problem really with the blogosphere and the increasing fragmentation of political debate on the internet, or is that a symptom of a wider unhinging, that is being driven by powerful forces in politics and the media? Indeed, even though he’s completely wrong and definitely not honest or well-meaning, there’s not really anything wrong with what Anthony Watts does, in principle, in his little denialist fantasy land. There’s also lots of debate and engagement between the two sides of the AGW “debate” on the internet – if anything, the question is whether there should be less, not more, given how wrong and mendacious the denialists can be. And the role of the media here, too, is questionable since the average journalist’s understanding of the concept of “balance” doesn’t extend past “giving a nutjob a voice on national tv.” The notion that balance is best obtained through calm and rational presentation of facts and getting it right doesn’t seem to have stuck with modern journalists, who constantly trot the likes of “Lord” Monkton out to defend the indefensible – and in fact as the science gets more settled and the denialist population shrinks to a smaller and crazier rump, the journalist notion of “balance” just leads to crazier and crazier people being put on national tv to represent the “opposing view.” Again, is this the blogosphere’s fault? Sure some of those bloggers love to feed the fires, but everyone is craving the legitimacy of the mainstream public eye, and it’s journalists who offer that legitimacy, not blogs with too many colours and 30% of the words in block letters. If AGW was a fiction conjured up by powerful voices in the mainstream, then Watts’s work would be honourable rather than misguided, and he would be justified in both using harsh language, and allowing insulting and rude language on the part of his commenters. And even though some of the stuff he does there – particularly the shenanigans with publishing private correspondence that just happens to be embarrassing – is scummy and low and something he should be ashamed of, that kind of stuff is par for the course with national media and has been for a very long time.

I guess there’s a fine line between being an arsehole and being a hero – a lot of politicians seem not to like the Watergate journalists, or Assange, and I guess from their perspective the work of these people is more than just an inconvenience. But there’s more than enough arseholes in politics and the media, and they’ve been around long enough and doing dirty enough work, that one hardly needs to look to the internet for the cause of the increasingly strident and aggressive nature of modern political debate. The Palins and Limbaughs and Abbotts and Murdochs of the world have pretty much cornered the market on being nasty in public, and given how often journalists offer them the fig-leaf of legitimacy through unquestioning regurgitation of their crap, acceptance of the “legitimate questions” they raise, or straight-out editorializing in their favour, I think it’s fair to say that when journalists start pointing the finger at new media they’re either trying to shift the blame, or warn each other that their time is up. They certainly aren’t trying to improve the quality of public debate, because they and their political masters managed to debase that years ago.

It’s come to my attention recently that I’ve never actually written a comments policy, or stated what my privacy policy is on this blog. Even though very few people visit this tiny part of the internet, it may surprise both of you to discover that I have actually had to enact an element of my privacy policy before, and this makes me think that I probably ought to specify what it is for those of you who comment here. So here’s my privacy policy in (more than) three easy dot points:

  1. I don’t demand real names or identification of any kind, though you’re welcome to use your real name and of course I encourage interesting screen names. Furthermore, I don’t demand that you provide a valid email address (one of my friends posts here with a pathetically obvious fake email address, that he should be ashamed of)
  2. I will never reveal your identity publicly on the blog or privately, and no matter how much I disagree with your loathsome political views (let’s face it, you’re all worms) I won’t be cashing them in to your boss/wife/mistress/dog, because that kind of behavior is complete shit. If some neo-nazi gobshite turns up on my blog and gives the email address “[insert obviously real name]”, that gobshite can rest assured that I won’t be telling the cops that they’re a nazi gobshite (though admittedly, giving that email address would be profoundly stupid – but if you weren’t profoundly stupid, you wouldn’t be a neo-nazi gobshite, would you?]
  3. I’m a curious and kind of voyeuristic guy, so I might do a whois inquiry or google you or something, but I won’t reveal that information to others on the blog or privately. Furthermore, I discourage commenters from referring to people using details that haven’t been revealed online. If you know that commenter neonazigobshite is a woman because you shagged her at the last End Apathy! gig (you poor bastard!), that doesn’t mean you should reveal her gender here if she is trying to keep it ambiguous (some chicks do that). I’m also not going to get pissed with anyone for obvious stuff-ups in this regard (let’s face it, you guys are pretty stupid)
  4. I see search terms that link to my blog, and if I find that someone is trying to identify you on google, or they have your identity and are trying to find out something about e.g. your political opinions or your online activity, I will do my best to tell you (I did this for someone once before)
  5. If you are looking for a job/partner/friend/drinking buddy and you are worried about them googling you and finding out that you wrote something stupid on my blog (because, let’s face it, you did), then contact me and I will do my best to delete it and/or everything you ever wrote (your choice). I’m pretty lazy, though, so if you wrote a lot your chances of scoring a clean record are pretty low, plus there’s the wayback machine (but anyone who is small-minded enough to google you is probably not clued up to that stuff). Also note in this regard that if I discover someone hunting you out on google and I’m worried that what you’ve said here might endanger your job chances, I may unilaterally delete you from the internet. You can thank me later!
  6. None of these considerations apply if you reveal that you have committed serious crimes, especially involving children. I’m not a priest!

Of course, a lot of this stuff is irrelevant because if you say anything really horrible I’ll just apply my comments policy, which is essentially: you’re free to say whatever you like, but I’m going to delete anything really nasty. I think if you review the comment threads on my Tolkien and Nazism or Tolkien and Fascism stuff you’ll see I’m pretty relaxed in my definition of “nasty.” I’ve never deleted a comment here before, but I’m sure one day I’ll have to.

I think it should be fairly obvious that this comments policy means I’m in favour of internet anonymity. I’m aware that this anonymity encourages rudeness but I have also seen a few situations where people online got into a lot of real life trouble – including losing jobs – because of the shitty behavior of other commenters and/or blog owners, and I don’t think that encourages decent debate (plus it’s really not very nice is it?). I don’t hide my identity, but I don’t flaunt it either, and that’s because I like to use this blog for exploratory or speculative thinking about topics related to my work, that I can’t do at work. I don’t want people thinking that what I say here represents either my fully-formed professional ideas (which are generally quite conservative) or my employer’s opinion of the same topics, and I also don’t want to make any claims to authority in what I present here. I think it’s better if people view me in terms of my nickname and treat my comments here with all the seriousness the nickname deserves (i.e. none). Also, a lot of what I talk about here blends sci-fi and politics and real life, and it’s easy for people googling someone to confuse that kind of speculative thinking with real opinions. I don’t want that to happen to me or you, so I’m happy to preserve your anonymity.

I know a lot of people on the internet think that anonymity is not one of its better qualities, but I disagree. I think debate can be enhanced by genuine anonymity, because people can say what they think without fear of work/loved ones/dogs discovering that they’re secretly an idiot. In general it hasn’t degraded the quality of debate here, there’s very little rudeness here and everything seems to be working out, but I certainly think that some of my commenters would be less inclined to, shall we say, engage in robust disagreement with me (you wankers) if they had to put their real names on the comments. So I’m going to preserve your anonymity. Although if the CIA come calling I’ll sell you down the river in a second – no one’s waterboarding me!

Oh, and on that note, I’m sure there are hundreds of people out there with great things to contribute who’ve been scared to on account of not knowing my privacy policy. So please, lurkers, delurk now! You are guaranteed anonymity! But not freedom from ridicule…

Gruumsh not think R help much like poetry. Gruumsh need use R to crush human foe. Gruumsh not like read help, but sometimes have to. Here help for round function, Gruumsh quote verbatim:

Note that for rounding off a 5, the IEC 60559 standard is expected to be used, ‘go to the even digit’. Therefore round(0.5) is 0 and round(-1.5) is -2. However, this is dependent on OS services and on representation error (since e.g. 0.15 is not represented exactly, the rounding rule applies to the represented number and not to the printed number, and so round(0.15, 1) could be either 0.1 or 0.2).

Gruumsh not trained statistician, but Gruumsh think this is big pile of steaming Ogre shit. Gruumsh check with Stata oracle. Stata rounds 0.5 to 1, not 0. Stata sensible god of numbers. Gruumsh not mathematician, but when Gruumsh round 0.15, Gruumsh expect 0.2. Gruumsh want to smash idiot that made IEC 60559 standard. Stupid jobsworth die slow nasty death under Gruumsh-club.


Today, doing a little task in R, I had cause to look up the following “warning” that appeared after compiling a script:

Warning message:
In readLines(file) : incomplete final line found

I couldn’t figure out what this warning meant, because the script ran fine, so I did a web search and I came across this exemplary example of why working with R really sucks: the help files are completely useless, the warning messages are cryptic and meaningless, the inbuilt editor is broken, there is no standardization of externally-developed editors, and the people who provide help online are some of the rudest people you will ever meet in computer science. This simple warning shows it all at once. I’ve complained about the dangers of R’s cryptic and meaningless warning messages again, but this example should really serve to show how they also cry wolf in a really unhelpful way.

The linked page is a message board of some kind (I think a reproduction of the “official” R boards on a another site) where a person called Xiaobo.Gu has posted up a request for help in decoding the above warning message. The request is polite enough though not voluminous, asking “Can you help with this?” but the first response (from someone with 7328 posts on this board!) consists entirely of the following:

Help with what? You got a warning. And it had information that should
tell you how to edit the file if the warning bothers you.

What is the point of a reply this rude and dismissive? This person actually took the time to reply to a post, in order simply to say “I won’t help you.” On a message board explicitly intended to help resolve problems with R. In addition to being rude it’s arrogant: there is no information abou thow to edit the file, just a pointer to the final line. We will shortly see the cause of the error, and it should be clear that no one in their right mind would consider the warning to have provided “information” of any form.

The next reply admonishes the original poster for failing to follow the posting rules (though doesn’t say how they were breached – so is essentially another contentless reply!) and then includes a little sneering aside about the way Windows encodes ASCII text that makes me think the developers of R have an elitist refusal to engage with Windows’s flaws. It then reveals that the warning is harmless and only appears in R version 2.14.0 (unpatched).

Why bother putting such a warning into a program? Whose idea was it to put a harmless warning in a single version of R, and why and how can a warning be a warning and also be harmless? Either something risky is going on, or it’s not. If it’s not, don’t waste my time with red text.

Finally another person comes along to sneeringly answer the question and provide actual information:

A warning message such as this could not be clearer.
It means that the last line of the file does not end with a <newline> sequence ==> the final line of the file is incomplete.

In an editor go to the end of that line and press <Enter> or <Return>
And save.

Alternatively configure your editor to always terminate the last line of a file with  a <newline> sequence.

This is a sparkling gem of passive-aggressive “help.” I can see a simple way in which the warning could be “clearer:” It could say “you did not press enter or return.” Then, it would be clearer. As it is, there is no information about what is missing in the final line: it just says it is “incomplete.” How can anyone claim that a warning such as this could not be clearer?

But then, just to top it off, this commenter has suggested that the poster configure their editor to “always terminate the last line of a file with a <newline> sequence.” This might seem to be reasonable advice, except that I get this warning in every script I write and I am using the built-in editor! This means that some muppet at C-RAN shipped a version of R with an editor configured to write scripts in such a way that they would trigger a warning. By default. Then, the very first patch they released got rid of the warning. wtf!? Is this what passes for quality control at C-RAN?

This is why wherever possible I use Stata for my work. I need software I can trust to produce the same results every time I run it, that isn’t going to waste my time with meaningless warnings and threats in glaring red, that isn’t configured to do things wrong by default, and that performs all calculations correctly. In order to trust that my stats software will perform all calculations correctly, I really need to know that the designers have some degree of basic quality control. When I see stuff like this – simple programmatic failings in things like the default settings of the script editor – I find it really hard to believe that the correct attention has been paid to, say, the way that the program performs adaptive Gaussian quadrature.

I also expect that the people who design this stuff will be polite when answering questions. I don’t need some passive-aggressive guy on the internet telling me off for failing to understand an extremely vague warning message that is only troubling me because C-RAN don’t have adequate quality control. The replies on that thread should have been polite requests for more information followed by an apology and a promise to fix this problem – or, if these people aren’t directly involved in C-RAN (and we know one of them is … one of R’s designers is on that thread) then a suggestion about how to alert the developers to the problem. Sneering and bullying – no thanks. I don’t get that when I contact Mathworks for help with Matlab, no matter how stupid my request.

This is why when I teach my students about stats packages I tell them a) you can’t trust R and b) it has a nasty community. I teach them its value for automation and experimental stats, and warn them away from using it for anything that has to be published in serious journals.

I think R is just another example of how dangerous it is to run your business on open source software, though I’m sure there are times when it’s safe. And I think it would be fascinating to see a detailed textual analysis comparing the message boards of an open source community (linux, R, latex) with a proprietary product like Stata, because in my experience there’s a world of difference between the two communities. Why  that difference exists would not only be a fascinating anthropological study, but would no doubt be of relevance to the scientific study of neckbeard behavior, because I have a strong suspicion that neckbeards are the dominant species in the open source world. Will an anthropologist somewhere take on the task?


« Previous PageNext Page »