Compromise and Conceit

Infernal adventuring…

recent posts

about

An Australian statistician and role-player living in Japan, interested in the theory and practice of role-playing…

  • The Epidemiology of D&D: What is the Best Ability for a Fighter?

    I’ve decided to begin a long-term research project aimed at understanding the underlying epidemiology of Dungeons and Dragons. This research project will consist of a series of (hopefully) increasingly complex simulations of battles between D&D PCs and various nemeses, to answer some key questions in character development and perhaps also to investigate some key controversies in the game. Once I have developed my simulations I hope to extend the project to Exalted, and I might diversify beyond that too.

    The simple weight of experience in D&D means that most people know, or feel they know, how D&D works and how the roll of the dice determines a PC’s fate. I have noticed that sometimes our intuitive understanding of these things can be wrong, and I’d like to investigate D&D in enough detail to understand how it works. I’ll write a separate post about some of the principles of the research project, but in this post I’ll present the first analysis.

    Introduction

    In this post a million battles are simulated between a million randomly-generated fighters and a single (unfortunate) Orc, Gruumsh The Bastard, who has 6 hit points and does 2d4+4 damage with his nasty falchion of fighter-crunching. Both Gruumsh and the million fighters were generated using Pathfinder rules as set out in the System Reference Document. These million battles were run in order to identify the effect of the three basic physical ability scores (Strength, Dexterity and Constitution) on survival for a standard fighter.

    Methods Summary

    Detailed methods are described at the end of the post. In essence, a million Pathfinder fighters were generated randomly and pitted against Gruumsh the Bastard in simulated battles. Fighter survival was analyzed using multiple logistic regression analysis by ability score. Survival probabilities by ability score are plotted in charts and summarized as Odds Ratios in the logistic regression analysis. No interactions or complex higher effects were considered. The distribution of hit points was summarized using a histogram, but doesn’t represent the true (practical) distribution of hit points for a fighter, since it includes fighters with unrealistically low constitution scores.

    Results

    Things didn’t go well for the million fighters. Overall survival was just 26%, with 256,584 lucky fighters making it to the end of their battle. The remaining 743,416 fighters were smashed to ribbons by Gruumsh and, in many cases, eaten. The median length of a battle was 4 rounds where the fighter survived, or 3 rounds if Gruumsh won. Figure 1 shows the probability of survival by ability score, and shows some stark differences in effect between ability scores.

    Figure 1: Probability of Survival by Ability Score

    It is clear from Figure 1 that strength is the key determinant of survival for a first level fighter. Only 0.4% of the weakest fighters survived, compared to 55% of the strongest. Constitution has barely any effect on survival, and dexterity is only important at the extreme ends of its range.

    Table 1 summarizes the results of multiple logistic regression of mortality. In this table, the odds ratio of death is given after adjusting for the other two ability scores, so removes the confounding effect of high or low values in other relevant ability scores. All odds ratios are given relative to the lowest value of the corresponding ability score, so for example those with strength 18 – 19 have an odds ratio of mortality of 0.003 compared to those with a strength of 2-3.

    Table 1: Multiple Logistic Regression of Death by Ability Score
    Variable Odds Ratio 95% Confidence Interval P value
    Strength
      2 to 3

    Ref.
      4 to 5

    0.21

    0.06 – 0.66

    0.008
      6 to 7

    0.07

    0.02 – 0.21

    <0.001
      8 to 9

    0.03

    0.01 – 0.10

    <0.001
      10 to 11

    0.02

    0.01 – 0.06

    <0.001
      12 to 13

    0.01

    0 – 0.03

    <0.001
      14 to 15

    0.006

    0 – 0.02

    <0.001
      16 to 17

    0.004

    0 – 0.01

    <0.001
      18 to 19

    0.003

    0 – 0.01

    <0.001
    Dexterity
      2 to 3

    1
      4 to 5

    0.87

    0.69 – 1.10

    0.236
      6 to 7

    0.76

    0.61 – 0.94

    0.012
      8 to 9

    0.65

    0.53 – 0.81

    <0.001
      10 to 11

    0.54

    0.44 – 0.67

    <0.001
      12 to 13

    0.45

    0.36 – 0.55

    <0.001
      14 to 15

    0.37

    0.3 – 0.45

    <0.001
      16 to 17

    0.3

    0.24 – 0.37

    <0.001
      18 to 19

    0.23

    0.18 – 0.28

    <0.001
    Constitution
      2 to 3

    1
      4 to 5

    0.9

    0.73 – 1.11

    0.307
      6 to 7

    0.86

    0.71 – 1.04

    0.113
      8 to 9

    0.82

    0.68 – 0.99

    0.044
      10 to 11

    0.72

    0.6 – 0.87

    0.001
      12 to 13

    0.63

    0.52 – 0.76

    <0.001
      14 to 15

    0.55

    0.45 – 0.66

    <0.001
      16 to 17

    0.48

    0.4 – 0.58

    <0.001
      18 to 19

    0.41

    0.34 – 0.49

    <0.001

    There is no difference statistically between a constitution score of 6-7 and a score of 2-3 – everyone with constitution scores in this range are purely at the mercy of the dice. In comparison, increasing strength from 3 to 4 reduces the odds of death by a factor of five, and fighters with a strength of 18 have an odds of mortality 300 times lower than fighters with a strength of three. Truly, fortune favours the strong.

    Figure 2 shows the odds ratio of mortality for constitution with its 95% confidence intervals, as a graphical alternative to a portion of Table 1 (we promised Gruumsh we would describe his victory in pretty pictures).

    Figure 2: Odds Ratio of Survival by Constitution Score

    Figure 2 suggests that hit points are not as important to combat survival as the ability to smash your opponent into the dirt. Once the Toughness feat is incorporated into simulations, constitution is likely to become even less important, and should probably be treated as a dump stat by players. Given that choosing the Toughness feat is equivalent to making a large increase in constitution, but this increase in constitution gives a barely-statistically-significant reduction in mortality, it seems likely that this feat is not a very useful choice. If Gruumsh is willing, this will be investigated in subsequent analyses[1].

    The distribution of strength ability scores under the 4d6 choose-the-best-three method is shown in Figure 3. This method shifts the scores significantly to the right: only 754 fighters had a strength of 3, compared to 16,141 who had a strength of 18. The mean strength was 12.24 and the median 12, a shift of three from a standard 3d6 distribution and a huge change to the extreme values.

    Figure 3: Distribution of Strength Scores Under 4d6 choose-the-best-three

    Nearly 5% of the sample had at least one physical score of 18; but this method is still not perfect, with only 3 of one million fighters having a score of 18 in all three physical attributes (one of these three, who also had an intelligence of 15 and a charisma of 16, was beaten to a bloody pulp by Gruumsh in just three rounds. His liver, apparently, was exquisite when grilled lightly and eaten on rye bread with a dark ale).

    Figure 4 shows the distribution of hit points in this sample of 1 million fighters. This is not the distribution one would actually see in a sample of actual Pathfinder fighters, since in a real game most fighters will have non-negative constitution bonuses (unless their player has read this post, I suppose). This histogram shows an interesting effect, however: even when constitution is unrestricted, under a 4d6/choose-the-best-three system there is a heavy concentration of hit points in the range of 4 – 10. Median hit points in this sample were 6, and the average hit point total was 6.2: in fact, the hit point distribution looks remarkably close to a uniform distribution on the range 4 – 10!

    Figure 4: Distribution of Hit Points

    Survival was not strongly associated with hit point value: those with 1 hit point survived in 20% of battles, while those with 14 hit points survived in 50% of battles. This extra importance of hps relative to constitution is driven entirely by the extra die roll (the d10 for hps) which suggests that constitution would be of much greater importance if hit points were fixed at first level; equivalently, it may be that the roll of constitution is washed out by the random determination of hit points, and if so one can expect that constitution will be more important at later levels when the law of large numbers cancels out the random effect of dice rolls on survival. For the same reason strength will probably reduce in importance over levels, since its effect is not compounded with level as constitution is. This is an issue that will need to be investigated, although if survival probabilities are replicated at second level it’s unlikely we will have much of a sample size of high level PCs[2].

    Conclusion

    At first level, strength is far and away the most important ability score for fighters, and constitution is so insignificant as to be almost a dump stat. A fighter with strength of 18 has only 1/300th the odds of death of a fighter with strength 3 when fighting a single Orc. Overall survival rates were low even in the toughest fighters, and in the absence of feats it appears that Pathfinder is an extremely nasty environment for solo adventuring.

    Future research will investigate the role of feats in enhancing survival, and their importance relative to ability scores. The results presented here are preliminary, but it appears that in min-maxing fighter PCs the wisest choice is to prioritize strength, then dexterity, then constitution. If one is developing a PC with the intention of long-term survival these findings may be reversed, but the experimental results have not yet been collated.

    Finally, the results presented here suggest that the assignment of a 1/3 challenge rating (CR) to Orcs in Pathfinder may be unwarranted. Although data are not shown here, in the testing stage this simulation program was run on Goblins (also CR 1/3) and the fighter survival rate was much higher. It may be the case that Orcs are far more challenging than a CR of 1/3. It’s not clear how Pathfinder assign their CRs, but it seems natural to suppose that a creature with a more than 50% chance of defeating an average human fighter is more than CR 1. Are Pathfinder’s CRs accurate? In any case, basic advice to fighters in Pathfinder would be: hunt Goblins, not Orcs, they’re much lower risk for the same xps.

    Methods

    For this analysis the fighters were generated according to the following rules:

    • All ability scores were generated using 4d6 choose-the-best-three, rolled in order: This is not orthodox Pathfinder but enables simultaneous estimation of the probability distribution of ability scores under this commonly-used rule, and enables analysis of the effect of ability scores across their full range – not just in the high values that one would usually assign to a PC’s prime characteristics
    • No feats were assigned to the fighter: for this first analysis the effect of raw scores was the topic of analysis, so no special abilities were given to the fighters. These million meat-shields were cast into battle with only their raw talents at their disposal
    • All fighters had the same equipment: raising a levy of a million fighters takes only a minute in 64 bit R, but it’s clearly a costly imposition on the citizenry, so all fighters were assigned standard kit consisting of chain mail armour, a standard shield, and a longsword. If we can secure a sufficiently large research grant from Waterdeep, subsequent battles we will allow random variation in armour types in order to choose the best armour
    • Racial abilities were not tested: no racial ability score adjustments or size bonuses were tested. Only raw scores were used. In future battles, racial ability scores will be incorporated into the PCs. Anyway, who cares if a halfling lives or dies?

    The results of all battles were summarized as two numbers: length of the combat in rounds, and whether or not the fighter lived or died (Gruumsh is a bastard, and his survival status is essentially irrelevant). Survival probability was plotted by ability score, and also analyzed using multiple logistic regression to assess the odds ratio of survival for any level of any ability after adjusting for all other abilities. Histograms of hit points and ability score (strength) were also obtained for reference purposes. The odds ratio of survival at different values of one score (constitution) was plotted with 95% confidence intervals.

    No ethical approval was obtained for this study, and anyone with concerns about the ethics of the study can raise the issue with Gruumsh. Informed consent was not obtained from any subjects (though Gruumsh seemed pretty eager to participate, and said “smash human!” many times, so could probably be said to have given active consent). No medical care or counselling was offered to survivors of the battles, and no reward was offered. The lucky minority who survived probably went off to start a farm or something, but we don’t know because follow-up to assess general physical health or emotional needs was not offered. Experience points were not distributed to the victors, because if we did Gruumsh would have gained enough levels to take over the world and no one wants that. Gruumsh was allowed to feast on the remains of his vanquished foes, because culturally sensitive research techniques are very highly prized at the Faustusnotes Military Academy. All simulations were conducted in R version 2.15.0, and all analyses were carried out in Stata/MP 12 because R sucks for things like making simple tables. The analyst was not blinded to the participants in the study, but if you think he had any interest in scanning a million records of a .csv file looking for fighters to favour, you’re an over-optimistic fool. This study was also not registered with CONSORT, but it’s unlikely that it would get published in any public health journal, so there was no need, really, was there?

    —

    fn1: Actually, Gruumsh is unlikely to get a choice. We’ll just roll up the fighters and send them in his direction.

    fn2: Actually, if we run a series of level-by-level simulations we could test whether the probability distributions of levels given in the D&D DMG are correct, and come up with empirical estimates of the true proportion of the population who are higher level!

  • Finding an Apartment in Japan

    On Sunday I signed a contract to rent a new apartment, which I will move into next weekend. Currently I’m living in a kind of Granny flat (called an ohanare) behind the home of my landlord, in South Kichijoji near the famous Inokashira park. I like Kichijoji a lot, so our new apartment will be in nearby Mure, about 5 minutes’ walk from Inokashira Park, and 1 minute’s walk from the sports fields and other recreational parts of the same park. It’s probably 10 minutes’ walk from the Studio Ghibli Museum, so my cat can hunt Kodama. It’s a ground floor apartment, what in Japan is called a 2DK and would in Australia be called a 1 bedroom apartment with study.

    Finding an apartment in Tokyo is so much easier than anything like it in Australia. Every year at least the Sydney Morning Herald produces a report like this that tells you what you need to do to find a rental apartment in Sydney: prepare a CV (including one for your dog!), dress nicely, form a personal relationship with a real estate agent, and offer more than the advertised rent. My experience of house-hunting in Sydney was terrible: queues of people turning up to look at one apartment, only to have the agent be late or not turn up; looking at apartments while people still lived in them (and being interrupted by unannounced inspections when I lived in an apartment!), and routinely being lied to about the place: turning up to a “3 bedroom” house to discover it was actually 2 bedrooms and a living room, for example.  Also, rents are shocking: the most recent census tells me that median weekly rent in Australia is $285, and my last two apartments were considerably more. Six years ago in Sydney I paid $360 a week for a 1 bedroom + study apartment in Stanmore, about 45 minutes by public transport from the city centre. Before I left Sydney I paid $190 a week for a 1 room apartment in Redfern – Sydney’s most dangerous suburb – on a street with signs warning against car-jackers. It had a shared laundry that occasionally would be smeared with vomit. Before I left London I was paying about 60,000 yen a month (about $150 a week) for a single room barely larger than a walk-in closet, in a share house with three other people and no real living room.

    My rent in my new apartment is about $1200 a month – about $300 a week – for a house with 2 air-conditioning units, 15 minutes walk from Kichijoji station, in the area that is voted number one place to live in Tokyo every year. It’s trivial to find a one room apartment in the heart of Kichijoji for $190 a week, and there’s zero risk of car-jacking. I also didn’t have to fuss around with open inspections, offering higher rent, or special CVs. I just visited a few real estate agents, told them what I was looking for, and they drove me to a variety of apartments around Kichijoji.

    In essence, it’s easier and cheaper to rent an apartment in Tokyo than it is in Sydney. Finding an apartment is a very smooth process, and if you don’t have a pet there’s a huge range. Going even a small amount above what I am paying gets you a very large and comfortable place. If you want to live in Shibuya I think it’s a very different story, but then, on on the flipside, Chiba is even cheaper. The big down side to renting in Tokyo is the upfront money – it’s not 100% of apartments, but for the majority of places you’ll have to put down 3 months’ rent upfront, of which you will only get back 1. If you have a cat, it’s 4 months rent upfront. It is possible to find places that don’t make this demand, but they’re often older or not so common. Still, the average real estate will have 2 or 3 such places, and there is a huge number and range of housing stock.

    I’m not sure why this difference exists, but it’s one of the key reasons I think the cost of living in Japan is low. Share housing is not very common even in Tokyo, because it’s quite easy to live alone in the suburb of your choice for between $500 and $800 a month – even students can live alone. I think it’s a mixture of things, but I wonder how much of it is a legacy of the 1980s housing boom, how much of it is ageing population, and how much of it is weak property rights. I suspect a lot of it has to do with the latter, which makes it easy to throw up new buildings higgledy-piggledy even in expensive suburbs. Also there seems to be a lot more corporate investment in housing – why, I don’t know. Tax arrangements here seem to be less favorable to family homes, though I don’t know the details. But renting is cheap and easy, and real estate agents – precisely contrary to their behavior in the west – are polite, honest, and very helpful. It’s as if renting were a respectable lifestyle decision, and a viable way to live. Have you ever heard of such a notion?

  • Did a Vampire Steal Your Home?

    It will also eat your children

    I think by now that it’s well known and accepted that Britain’s ex-Prime Minister, Tony Blair, is a vampire. Questions remain as to what magic he was using to enable him to go outside during the day – was it fairy blood, or the souls of Iraqi children? Alchemists across the multiverse want to know his secret. But beyond the arcane details, the facts are pretty clear: the working class people of Britain made a kind of unholy pact with their vampire overlord – in return for allowing him to do whatever unspeakable things he wanted to do to them, he would deliver unto the poor of Britain a ponzi scheme unlike any other through the magical ritual of “the housing market,” and they would be rich forever – the “end of boom and bust” as his blind necromancer-eunuch described it. In exchange for a little of their blood, they thought they would have it good forever. I don’t think they understood, though, just how depraved he was.

    I guess that’s the inevitable consequence of giving too many rights to vampires. Especially vampires so old and powerful that they can work their hypnotic magic on a whole nation. He should have been staked back in the 1990s when he was first beginning to whisper his sibilant lies into the ears of the weaker minds in the labour party, and enslaving Brit Pop bards to his ferocious will.

    However, the really sad thing about the Faustian pact that the working people of Britain signed with the vampire is that they never got their side of the deal, and it’s now becoming patently clear that they have been sold up the river, their hopes and dreams destroyed elegantly and completely by their undead ally. The truth is clearly laid out in this article from the Guardian, which reveals that the proportion of people in Britain who own their own home has declined from 43% in 1993 to 35% now; and is expected to drop lower, possibly to as low as 27%, if the economy remains stagnant. Furthermore, the second fruit of the unholy deal, ever-increasing wealth, has not been delivered. The Guardian reports that working Britons are suffering poverty at astounding rates:

    these 3.6m British households have little or no savings, nor equity in their homes, and struggle at the end of each month to feed themselves and their children adequately. They say they are unable to cope on their current incomes and have no assets to fall back on, leaving them vulnerable to something as simple as an unexpectedly large fuel bill.

    But hang on, I hear you say – didn’t Britain recently have a so-called “housing boom” during which huge amounts of investment poured into the private housing market? How can it be that there was a housing boom but less people now own their own homes than before the boom started? It’s as if – shocking! – the boom served to cement ownership of property amongst the wealthiest sections of society, and concentrated land ownership away from the hands of the poorer half of society – and we’re not talking about the bottom of society here, or the so-called “undeserving poor.” The Experian research specifically excluded the most deprived parts of society and the unemployed – there are 3.6 million households in the UK who are working full time but have “no assets to fall back on” and have likely been squeezed permanently out of the home-buying market.

    This is not what that vampire promised. Not at all. The vampire promised those 3.6 million households that they would be able to buy a home and gain a little financial security, and maybe some wealth. Instead, they’ve been locked out permanently. It hasn’t been remarked upon in the discussion of this new phenomenon of “generation rent,” but I think an important point needs to be recognized: this is the single biggest increase in inequality in a generation. In Britain, owning a home is a very important financial goal. It protects you in retirement, gives you secure capital, and ensures that your children have some form of endowment to protect them if (as easily happens in societies as unequal as Britain) they find themselves sliding down the income scale compared to you. Furthermore, over one’s lifetime it is meant to be cheaper than renting. Now, it doesn’t have to be this way – there are other ways to prevent inequality than ensuring home ownership – but this is the way it is in Britain. And the so-called “housing boom” has ensured that the number of people who are able to access this security has declined by 20% in just 20 years. That is a huge increase in inequality, and it all happened under the stewardship of the Labour Party, who were in power from 1997 to 2010. The Labour Party and their apparatchiks in the Guardian make much of their efforts to lift children out of poverty through tax credits, but what does that matter if at the same time they have stripped away a fundamental economic goal for 8% of the population? Those children who have been saved from poverty by tax credits will simply slide back into it in adulthood, in the depressing and sad way the Experian report describes: working hard, and not even treading water. For those 2.2 million children in the Experian report, their  adult experience of the housing market will likely be a series of long, arduous lessons in that most British of sayings: free to those who can afford it, very expensive to those who can’t.

    Furthermore, much of this inequality is likely to be generational: the people who have concentrated their ownership of the housing wealth will be baby boomers from the middle and upper classes, and as the reports note, the main losers in this massive land grab have been young families. You will hear conservatives talk a lot about “generational equity” when they are worried about government debt (“leaving it to the next generation to foot the bill,” etc blah blah) but where were they during the housing boom, while a small slice of the richest generation in history were stealing land from generation X and Y? I don’t recall ever hearing anything from the ‘Tories that might have any resemblance to a warning that the “housing boom” was going to lead to a huge increase in generational inequity. But I bet you can find all sorts of that kind of lazy and shiftless argument about government debt and bailout funds.

    So where does that leave Britain? It has a stagnating economy, with a population of some 7 million working poor (2.2 million children!) who are on the brink of financial disaster, a whole generation squeezed out of the financial security of home ownership, and yet simultaneously a rental market that is suffering a lack of housing supply – so rents are skyrocketing. At the same time, private superannuation funds have been losing money for the last 10 years, making home ownership more valuable than ever, yet the “housing boom” has shaken a huge number of people out of that market – permanently.

    Which just goes to show that the people of Britain should never have cut any kind of deal with that vampire – they should have staked it, and locked its cabal of necromantic followers into a dungeon somewhere, then thrown away the key. It also makes me think that the Reign of the Vampire saw a greater increase in inequality than ever happened under his supposedly satanic predecessor, Margaret Thatcher. I wonder if many people in the British left agree with me? Let’s consider, as a salutary example, this chap: Dennis Skinner, 80 year old Labour stalwart who was “formed in the pits and the war” and has a strong dislike of toffs and Thatcherites. He refused to take a cabinet position under Blair because he might speak out against government policy, and then be exiled from the party. Dennis, mate, allow me to let you in on a secret: your boss was a vampire. Honour, decency, and any kind of morality worth having demanded that you speak out against him and his stupid ponzi scheme. But you expressly avoided putting yourself in a position where you would be able to do that effectively. Or, could it be that you just missed the important facts here? You went to dinner with Tony Blair and were somehow looking the other way when he grabbed one of the waiters, snapped his neck and drained him of his life’s blood? And you were just, kind of, you know, having a senior moment when he was telling you all about his glorious ponzi scheme that would see everyone in Britain get rich forever from borrowing money to buy each other’s houses. You may be a labour stalwart, but there’s something else you are too: an immoral fuckwit. And the next generation of kids to grow up in your area – who will never have to go down the pits because Margaret Thatcher closed them – will never be able to afford to buy a home because of people like you and their slavish devotion to a vampire. But you and your mates will continue to whinge about how Thatcher destroyed the country and made it less equal.

    As I mentioned, none of this lets the Tories off the hook – they were cheerleaders for the vampire’s stupid ponzi scheme from its very inception. But idiots like this, who are so profoundly incapable of sensible policy-making that they drop hints about returning to the gold standard, don’t have a vampire for a boss. Their claim to infamy is that they could have done better – basically a badge of pride for your average Tory. But unlike the Skinners of this world, it’s not sitting on the lapel of their coat next to a badge that says “I allowed a vampire to arse-fuck my country,” which is what most members of the Labour Party should be wearing.

    And for the record, it was obvious to me that the vampire was evil from the moment I set eyes on it. That hideous fixed grin, the soulless eyes, the voodoo carefully disguised as an economic policy … the only question that remains unanswered for me is – how does it manage to walk around in sunlight without bursting into flames?

  • Why Working with R Really Sucks

    Today, doing a little task in R, I had cause to look up the following “warning” that appeared after compiling a script:

    Warning message:
    In readLines(file) : incomplete final line found

    I couldn’t figure out what this warning meant, because the script ran fine, so I did a web search and I came across this exemplary example of why working with R really sucks: the help files are completely useless, the warning messages are cryptic and meaningless, the inbuilt editor is broken, there is no standardization of externally-developed editors, and the people who provide help online are some of the rudest people you will ever meet in computer science. This simple warning shows it all at once. I’ve complained about the dangers of R’s cryptic and meaningless warning messages again, but this example should really serve to show how they also cry wolf in a really unhelpful way.

    The linked page is a message board of some kind (I think a reproduction of the “official” R boards on a another site) where a person called Xiaobo.Gu has posted up a request for help in decoding the above warning message. The request is polite enough though not voluminous, asking “Can you help with this?” but the first response (from someone with 7328 posts on this board!) consists entirely of the following:

    Help with what? You got a warning. And it had information that should
    tell you how to edit the file if the warning bothers you.

    What is the point of a reply this rude and dismissive? This person actually took the time to reply to a post, in order simply to say “I won’t help you.” On a message board explicitly intended to help resolve problems with R. In addition to being rude it’s arrogant: there is no information abou thow to edit the file, just a pointer to the final line. We will shortly see the cause of the error, and it should be clear that no one in their right mind would consider the warning to have provided “information” of any form.

    The next reply admonishes the original poster for failing to follow the posting rules (though doesn’t say how they were breached – so is essentially another contentless reply!) and then includes a little sneering aside about the way Windows encodes ASCII text that makes me think the developers of R have an elitist refusal to engage with Windows’s flaws. It then reveals that the warning is harmless and only appears in R version 2.14.0 (unpatched).

    Why bother putting such a warning into a program? Whose idea was it to put a harmless warning in a single version of R, and why and how can a warning be a warning and also be harmless? Either something risky is going on, or it’s not. If it’s not, don’t waste my time with red text.

    Finally another person comes along to sneeringly answer the question and provide actual information:

    A warning message such as this could not be clearer.
    It means that the last line of the file does not end with a <newline> sequence ==> the final line of the file is incomplete.

    In an editor go to the end of that line and press <Enter> or <Return>
    And save.

    Alternatively configure your editor to always terminate the last line of a file with  a <newline> sequence.

    This is a sparkling gem of passive-aggressive “help.” I can see a simple way in which the warning could be “clearer:” It could say “you did not press enter or return.” Then, it would be clearer. As it is, there is no information about what is missing in the final line: it just says it is “incomplete.” How can anyone claim that a warning such as this could not be clearer?

    But then, just to top it off, this commenter has suggested that the poster configure their editor to “always terminate the last line of a file with a <newline> sequence.” This might seem to be reasonable advice, except that I get this warning in every script I write and I am using the built-in editor! This means that some muppet at C-RAN shipped a version of R with an editor configured to write scripts in such a way that they would trigger a warning. By default. Then, the very first patch they released got rid of the warning. wtf!? Is this what passes for quality control at C-RAN?

    This is why wherever possible I use Stata for my work. I need software I can trust to produce the same results every time I run it, that isn’t going to waste my time with meaningless warnings and threats in glaring red, that isn’t configured to do things wrong by default, and that performs all calculations correctly. In order to trust that my stats software will perform all calculations correctly, I really need to know that the designers have some degree of basic quality control. When I see stuff like this – simple programmatic failings in things like the default settings of the script editor – I find it really hard to believe that the correct attention has been paid to, say, the way that the program performs adaptive Gaussian quadrature.

    I also expect that the people who design this stuff will be polite when answering questions. I don’t need some passive-aggressive guy on the internet telling me off for failing to understand an extremely vague warning message that is only troubling me because C-RAN don’t have adequate quality control. The replies on that thread should have been polite requests for more information followed by an apology and a promise to fix this problem – or, if these people aren’t directly involved in C-RAN (and we know one of them is … one of R’s designers is on that thread) then a suggestion about how to alert the developers to the problem. Sneering and bullying – no thanks. I don’t get that when I contact Mathworks for help with Matlab, no matter how stupid my request.

    This is why when I teach my students about stats packages I tell them a) you can’t trust R and b) it has a nasty community. I teach them its value for automation and experimental stats, and warn them away from using it for anything that has to be published in serious journals.

    I think R is just another example of how dangerous it is to run your business on open source software, though I’m sure there are times when it’s safe. And I think it would be fascinating to see a detailed textual analysis comparing the message boards of an open source community (linux, R, latex) with a proprietary product like Stata, because in my experience there’s a world of difference between the two communities. Why  that difference exists would not only be a fascinating anthropological study, but would no doubt be of relevance to the scientific study of neckbeard behavior, because I have a strong suspicion that neckbeards are the dominant species in the open source world. Will an anthropologist somewhere take on the task?

     

  • Vampire Remains in Bulgaria

    Reports emerge of archaeologists who have discovered the remains of men staked to prevent vampirism in Bulgaria. The rate of Vampirism in ancient Bulgaria appears to have been 2 per 700 adults, though perhaps only men were staked. A rare but deadly condition, which apparently was prevented using wooden or iron rods (iron was reserved for the richer vampires) and may not have been officially sanctioned – “brave men” opened the coffins after burial to do the staking.

    I wonder if Buffy was there?

  • NASA Invented Gaia Theory; also, Frack it!

    Hippies!!!

    While enjoying my traditional Saturday morning bludge, I seem to have stumbled on an unusually rich treasure trove of internet links, each capable of sparking its own chain of reading and contemplation that could keep me going all day. It’s not standard practice at the Faustusnotes School of Propaganda to simply post up links, but in this case I’m feeling so inspired by the material I stumbled on that I thought I would put up a little linkfest; plus, of course, my own commentary. Here, then, are a few things that interested me on this Saturday morning, that I think might also interest my reader(s).

    Gaia theory’s inventor blames it all on NASA

    James Lovelock, who invented Gaia theory in conjunction with Lyn Margulis, is retiring from his career as a “freelance scientist.” The Guardian has an interview with him, which goes to show may things – including, possibly, confirmation that scientists should retire at 65 like everyone else, because they become increasingly kooky as they age. Maybe at a faster trajectory than non-scientists. Lovelock, who in addition to Gaia theory also appears to have been responsible for saving the ozone layer, is revealed to be a supporter of nuclear power and opponent of wind-farms, even though he is a strong proponent of AGW theory and famously claimed that

    by this century’s end “billions of us will die and the few breeding pairs of people that survive will be in the Arctic where the climate remains tolerable”

    a claim he has now retracted and replaced with “2ft a century sea level rise” (he has bought a cottage by the coast). He also supports fracking, a fairly controversial energy source to say the least and one that probably would have run into a great deal more opposition in the UK this summer if they hadn’t happened to just experience the wettest spring since Noah. In defense of fracking he has some virulent thoughts on German energy policy, with which I largely agree, but I think he’s out of whack with the carbon budget predictions when he talks about fracking: switching to gas alone is not enough to meet our carbon budget targets, we need much lower carbon energy sources than that, and if we can’t fall back on nuclear power then we really, really need to find something that is even lower carbon than gas. I’ve not really got an opinion either way on fracking but it seems unduly risky, and while it may be a useful way to buy some time, I strongly suspect that at least in Britain it won’t have any effect on the carbon budget – it’s just an alternative to Russian gas. Lovelock also seems to be a fan of theories of group selection, which I thought had long since been dismissed.

    Anyway, the spectacle of Gaia’s creator advocating tearing up the earth – and his advocacy of moving everyone into megacities, air-conditioning them, and not worrying about the environment at all – suggests that Lovelock’s interpretation of Gaia theory isn’t very soft and fuzzy. Interestingly, though, in describing his career he suggests that he would never have come up with the theory if he hadn’t worked at NASA:

    when I got a letter from the director of space flight at Nasa I was gobsmacked. I realised I didn’t want to spend the rest of my life as a civil servant, and I didn’t like the idea of having everything planned right up to my retirement. My boss [at the National Institute for Medical Research in London] said I’d be a fool to ignore it and out of all that, ultimately, came Gaia.

    There you have it! I’ve always suspected that NASA with their pictures of earth-rises and their extensive work on global warming are really just a pack of hippies. Lovelock has confirmed it. Get these smelly, hairy space-telescope waving lefties away from government funds before they turn us all into gay abortionists!

    A subversive school lunch blog sticks it to the man

    The Guardian features a very sweet story about a blog, NeverSeconds, started by a nine year old British (Scottish?) girl that has run afoul of its local council and created something of a social media storm. In addition to being a really cute and interesting read, the blog is a really good example of how different blogging is as a medium, and how at its best it can really create new forms of cultural interchange. The blog theme is very simple: Martha Payne, the author, takes a photo of her school lunch and posts it on the blog with a review. It started as a writing project but quickly became a social media phenomenon, and soon she was being sent photos of school lunches from around the world. Told about how a German university has several canteens and a wide selection of cheap food, she writes:

    I like all the choices you get at University but it would take me a long time to decide everyday!

    The selection of foods from around the world that one can see in her blog are really fascinating, because it’s easy to see how much healthier the rest of the world eats than do the British – even though Martha’s food choices are much, much healthier than when I had reheated chips with mayonnaise for my school lunches back in the 1970s in Britain. You are a lucky girl, Martha, to be getting chicken fajita’s in a British school lunch!

    Anyway, eventually her blog came to the attention of the school authorities and then the council that supplies the lunches. The results are visible in her sad little post, “goodbye,” in which she tells us

    This morning in maths I got taken out of class by my head teacher and taken to her office. I was told that I could not take any more photos of my school dinners because of a headline in a newspaper today.

    Such a cute and plaintiff little post, that attracted (at time of writing) 2,261 comments and 1,019 google plus ones. About one in five of the comments are people saying they have complained to the council about the request not to take photos. Even Jamie Oliver has gone into bat for her. By today, the council had reversed its decision and she will be allowed to continue posting food reviews. The Guardian article has more information about what happened, and certainly makes the Council look very stupid, but the story ends well and the council have even admitted that they got it wrong. Which is good because this simple little project tells us so much about food and school life around the world, all in a simple and powerful format.

    The Mathematical Impossibility of Central Planning

    Cosma Shalizi, who is sometimes referred to as a statistician, has a fascinating post at his website that outlines the mathematical challenge of central planning and takes apart the work of a chap called Cockshott who claims to have proven that central planning in the Soviet Union was feasible. The claim is that by collecting certain information one could turn the task of planning an economy into a linear programming problem that would have been soluble in the Soviet era. Shalizi’s post on his own site is a response to comments on another post at Crooked Timber which is a beautiful and elegant description of the mathematical impossibility of Soviet-style central planning. Commenters there are describing it as one of the most amazing posts on the internet and I’m inclined to agree: from the title through the initial disclaimer to the notes at the bottom, it’s a work of art. The logical construction is also very nice: starting by representing central planning as a perfectly plausible problem in linear programming, Shalizi shows that it would have been prima facie impossible in the Soviet era, or even now; he then goes on to develop more complex aspects of the problem, including discussing how simplifications of the linear programming problem throw up their own linear programming problems that are equally intractable. For example, one option to simplify the linear programming problem would be to assume it only needed to be run for a given assortment of goods (that we could at least determine what outcomes people want, and then program the economy to produce them at maximum efficiency). About this, Shalizi observes:

    Kantorovich had a way of evading this, which was clever if not ultimately satisfactory. He imagined the goal of the planners to be to maximize the production of a “given assortment” of goods. This means that the desired ratio of goods to be produced is fixed (three diapers for every towel), and the planners just need to maximize production at this ratio. This only pushes back the problem by one step, to deciding on the “given assortment”.

    We are pushed back, inevitably, to the planners having to make choices which express preferences or (in a different sense of the word) values. Or, said another way, there are values or preferences — what Nove called “planners’ preferences” — implicit in any choice of objective function. This raises both a cognitive or computational problem, and at least two different political problems.

    The cognitive or computational problem is that of simply coming up with relative preferences or weights over all the goods in the economy, indexed by space and time. (Remember we need such indexing to handle transport and sequencing.) Any one human planner would simply have to make up most of these, or generate them according to some arbitrary rule. To do otherwise is simply beyond the bounds of humanity. A group of planners might do better, but it would still be an immense amount of work, with knotty problems of how to divide the labor of assigning values, and a large measure of arbitrariness.

    Notice here that we haven’t even touched on the obvious political and moral problems of the program – it’s just computationally infeasible. And we haven’t yet discussed the problem of non-convexity in the programming problem.

    Shalizi also seems to be implying that a lot of the practical solutions to the linear programming problem – so-called “shadow prices” or decentralization of the programming tasks, or decoupling of programming problems within industries – either directly resemble a move towards a market economy, or assume the necessity of developing a market economy to fill the gaps created by the simplification of the programming tasks.

    Finally, Shalizi goes on to a discussion of the problems of solving this programming problem through a market economy, and points out the limited extent to which functional economies actually resemble the free markets that many neo-liberal market fetishists see as the most efficient way of solving this optimal allocation problem. The final section describes in quite pretty language the reality that decision-making in modern societies can’t follow any one ideology or system in achieving its best outcomes:

    A bureaucracy, or even a thoroughly democratic polity of which one is a citizen, can feel, can be, just as much of a cold monster as the market. We have no choice but to live among these alien powers which we create, and to try to direct them to human ends. It is beyond us, it is even beyond all of us, to find “a human measure, intelligible to all, chosen by all”, which says how everyone should go. What we can do is try to find the specific ways in which these powers we have conjured up are hurting us, and use them to check each other, or deflect them into better paths. Sometimes this will mean more use of market mechanisms, sometimes it will mean removing some goods and services from market allocation, either through public provision or through other institutional arrangements. Sometimes it will mean expanding the scope of democratic decision-making (for instance, into the insides of firms), and sometimes it will mean narrowing its scope (for instance, not allowing the demos to censor speech it finds objectionable). Sometimes it will mean leaving some tasks to experts, deferring to the internal norms of their professions, and sometimes it will mean recognizing claims of expertise to be mere assertions of authority, to be resisted or countered.

    It is, as Shalizi says at the beginning, a long and exhausting post, but it is singularly elegant in its destruction of the central foundation ideas of Soviet planning, and fascinating for doing so from a mathematical and technocratic, rather than a moral or socio-political, framework. In this sense I think it takes on supporters of central planning at their own game. It also leads to some fascinating discussion in the comments, for example about how central planners could possibly handle the problem of as-yet-undiscovered technology and how to optimally seek it.  Well worth a read on a bludgey Saturday morning!

    The ethical challenges of photographing drug-using Russian parents

    Again from the Guardian, a fascinating article about the challenges facing Irina Popova after she released photographs from an exhibition about the lives of two drug-using Russians and their daughter. Popova, a professional photographer, stumbled on the family in St Petersburg and lived with them for two weeks, photographing their ordinary lives. The pictures can be viewed here, and show two adults living chaotic and irresponsible lives that many would judge as being unsuitable for a child; but she became accustomed to them and learnt to see love and kindness in amongst the chaos; when she put the pictures on show in an art gallery, she and the director drew very different conclusions about their meaning, but they won a prize. However, when she put them on her homepage they attracted a storm of criticism, as if she herself were somehow responsible for the lives of the people she photographed.  The Guardian observes about the criticism that

    The story of the photos raises a number of fascinating issues: about exploitation, voyeurism and embedded reportage; about the moral responsibility of a photographer or any artist who deals in non-fiction; about the differences between images seen in a gallery and images posted online; and about the meaning of informed consent.

    I would add that it might also say something about the quality of criticism that online material attracts – largely knuckle dragging fools, I’d guess. This exhibition is interesting to me because it bears tangentially on the world that I researched in the first half of my career, and people’s response to that world (drug addiction, poverty, chaos and marginalization) even when it is caring and concerned for the object’s humanity can show a remarkable degree of judgment and anger. The people in these photos often serve as ciphers for the viewer’s own judgments about the social order, how people should live, and what responsibilities these two individuals owe to the state and society or their child. The article itself shows how even the best intentions of those who intervene may not be in the best interests of everyone involved: a campaign to remove the daughter to an orphanage would probably hurt everyone involved, and it’s unlikely that the people demanding such a radical move understand any of the elements of the story the way that the photographer (or the adults involved) does. The photographer herself shows a strongly conflicted view about what “should” be done about the family she lived with, or how they “should” be living. When I look at the photographs I see a family on a bad path, but who clearly love their child, and I think Australian child protection generally tries to work on the assumption that a loving family is best, even if the adults in that family are not perfect – but the article suggests that the judgmental voices on the internet think that they can tell what is best from nothing more than a few pictures on a website.

    This story covers a complex mix of issues that will surely serve to get everyone screaming at each other, but for added joy we can enjoy the commenters below the line, reducing it all to their own personal biases. My favourite is comment four:

    What hath Putin wrought?

    That’s right, every decision that these two punks made is Putin’s fault. None of it lies in a much more complex problem of social systems reproducing themselves through family structures, in alcohol abuse or just the simple human problems of raising a child when you’re poor and uneducated and living in a world where you don’t want to or can’t work in the fields that are on offer. It’s because Putin shot a tiger. Or something. This is the kind of commentary that proves we need (in the words of the Daily Mash) a separate internet for people who comment on newspaper articles.

    Fortunately, however, we do have this internet with all its wealth of Saturday morning fascinations. And so the afternoon has already come around and I’ve done nothing. That, my friends, is the best kind of Saturday.

  • Did the Tories’ Mandatory Work Activity Programme Really Fail?

    The Guardian reports today that the UK Government has released a review of its Mandatory Work Activity Programme, and that the review finds the programme didn’t have any success getting people back to work. The review was written by the National Institute for Economic and Social Research (NIESR) and can be obtained from the blog of the NIESR’s director. The Director is disappointed that the Government announced an increase in funding after the review was released, even though it finds no effect. This MWA programme is essentially one of those work-for-the-dole schemes that forces people to take an unpaid job for four weeks if they want to keep their unemployment benefits.

    I strongly oppose work-for-the-dole schemes and I wouldn’t be surprised to find they don’t work in their supposed central goal of getting the unemployed back to work; I also don’t believe that really is their central goal, and I think they’re a terrible economic idea. But I can also see that they could be successful in getting people back to work (at least, in a functioning economy – so probably not in the UK). So I was interested to check the report and see whether or not its research methodology really supported the claims. I don’t think it does, and I don’t think this report forms a solid basis for a critical review of the programme.

    This report collects data on a couple of thousand people who were referred to the MWA between May and July, and compares them with people on benefits in that time period who weren’t referred. It then follows them to the end of November (I think), and compares them with a group of non-referred subjects who are identified during the same May-July period, and assigned “pseudo-referral dates” randomly to match those of the referred people. The groups are then matched using propensity score matching. The outcomes are then the differences in the proportion of people in work or in receipt of a benefit payment at each month between treatment group and control group. The results find no difference in work outcomes between the two groups.

    I think this method won’t show what it aims to show, because it doesn’t adjust for serial dependence or right-censoring. Only 5% of the sample was recruited in May and 40% in June, so the July sample – about 45% of the sample – were followed for 1-2 months less than the remainder. Two months’ less follow-up out of six is a lot of lost follow-up. This could have been adjusted for using survival analysis. I also don’t know why they worried about assigning pseudo-referral dates to the controls. With survival analysis, everyone in the sample can be considered a control until they receive a referral, at which point they switch into the treatment group, and then their follow-up time gets assessed. This allows for the people who entered work before they had a chance to be referred to the MWA to be included in the analysis. Finally, by analyzing each week separately (as in the charts on the Director’s blog) rather than analyzing time-to-employment, the analysis includes a heavy element of serial dependence. Basically, the proportion of people out of work in any one week will contain data from many of the same people as the week before. This serial dependence leads to bias in estimates of accuracy, and doesn’t adjust for the long-term unemployed properly – if someone had been on benefits for two years in the first week of July, chances are they’ll still be there contributing to the mean in the second week of July. This should be adjusted for more carefully I think.

    If survival analysis is used for this problem, there is no reason to stop collecting referral data at the end of July. Go right up until the end of the data, because the reduced follow-up time of later entrants is handled implicitly in this method. Also, comparing proportions in terms of differences – as this report does – is not a good plan. A 10% change from 80% is very different to a 10% change from 50%. Odds ratios or relative risks should always be used. Survival analysis uses hazard ratios, essentially equivalent to relative risks in a cohort study.

    Since this method wasn’t used, I don’t think the findings are robust. It’s hard to tell from the report exactly what the outcomes were but the lack of effect on employment could simply represent a failure to properly handle follow-up time and right-censorship. The data should be re-analyzed using survival analysis on all the available information, rather than a strange and non-standard process of assigning pseudo-referral dates to people who were never referred, and then comparing differences in weekly proportions. Strange!

  • Visual Proof Game of Thrones is anti-American

    This is just taking the piss out of a previous post, really… the latest storm doing the rounds is the discovery that one of the severed heads on the battlements of King’s Landing at the end of Season 1 is George Bush.

    What’s better, I wonder, for an ex-President? Having a library named after you, or getting on Game of Thrones? Anyway, everyone has apologized profusely, so it’s all okay …

  • Multiple Cores Killed the Maths Star

    Recently I’ve been working on some problems in disease modeling for influenza, and one of the problems is to calculate the basic reproduction number for a model which includes differential disease strengths in poor and rich risk groups. Calculating this number is generally done with a method called the “Next Generation Matrix” method, and to do this one needs to calculate two matrices of partial derivatives, invert one and multiply it by the other, then calculate the eigenvalues – the basic reproduction number is the largest eigenvalue of the resulting calculation. Doing this for just one risk group in the model I’m fiddling with can be done analytically in about 7 pages of notes – it involves finding the inverse of a 5×5 matrix, but actually this is quite quick to do by hand because most of the matrices involved are wide open spaces of zeros. However, once one extends the model to four risk groups the calculation becomes nastier – it involves inverting a 20×20 matrix, then finding the eigenvalues of a product of 20×20 matrix. Even recognizing that most of these matrices are zero elements, one still ends up with a fiendish hand calculation. On top of this, the matrices themselves contain many separate values all multiplied together. I started this by hand and decided today that I want to take a shortcut – a student needs to use some basic values from this soon and neither she nor I are going to get it done analytically before our deadline.

    So tonight I came home and, after a nice dinner and an hour spent with my partner, I spent about an hour programming Matlab to do the calculation numerically for me. I now have the two values that my student needs, and if she needs to tweak her model it’s just a few presses of a button on my computer to get the updated reproduction number. Also, it’s a matter of a second’s work to test any other parameter in the model, and with a few loops I can produce charts of relationships between the reproduction number and any parameter. It’s nice and it was fairly trivial to program in Matlab. In this instance Matlab saved me a couple of days’ work fiddling around with some enormously tricky (though not mathematically challenging) hand calculations.

    On this blog a short while back I investigated a weird probability distribution I had encountered at Grognardia. For that calculation, rather than going through the (eye-bleedingly horrible) tedium of attempting to generate a mathematical expression for the probability distributions I wanted to analyze, I simply ran a simulation in R with so many runs (about 100,000) that all random error was stripped out and I essentially got the exact shape of the theoretical underlying distribution I wanted.

    In both cases, it’s pretty clear that I’m using a computer to do my thinking for me.

    This is very different to using a computer to run an experiment based on the theory one developed painstakingly by hand. Rather, I’m using the brute number-crunching power of modern machines to simply get the theoretical result I’m looking for without doing the thinking. That Grognardia problem involved a badly programmed loop that executed a total of 4,500,000 dice results just to produce one chart. I did it on a computer with 32Gb of RAM and 12 chips, it took about 3 seconds – and I didn’t even have to program efficiently (I did it in R without using the vector nature of R, just straight looping like a 12 year old). The resulting charts are so close to the analytical probability distribution that it makes no difference whatsoever that they’re empirical – that hour of programming and the 3 seconds of processor time short circuited days and days of painstaking theoretical work to find the form of the probability distributions.

    Obviously if I want to publish any of these things I need to do the hard work, so on balance I think that these numerical short cuts are a good thing – they help me to work out the feasibility of a hard task, get values to use in empirical work while I continue with the analytic problems, and give a way to check my work. But on the flip side – and much as I hate to sound like a maths grognard or something – I do sometimes wonder if the sheer power of computers has got to the point where they genuinely do offer a brutal, empirical short cut to actual mathematical thinking. Why seek an elegant mathematical solution to a problem when you can just spend 10 minutes on a computer and get all the dynamics of your solution without having to worry about the hard stuff? For people like me, with a good enough education in maths and physics to know what we need to do, but not enough concerted experience in the hard yards to be able to do the complex nitty-gritty of the work, this may be a godsend. But from the broader perspective of the discipline, will it lead to an overall, population-wide loss of the analytical skills that make maths and physics so powerful? And if so, in the future will we see students at universities losing their deep insight into the discipline as the power of the computer gives them ways to short cut the hard task of learning and applying the theory?

    Maybe those 12 chips, 32Gb of RAM, 27 inch screen and 1Gb graphics card are a mixed blessing …

  • Sex Trafficking, Radical Feminism and the Misappropriation of State Power

    Continuing (belatedly) my series of posts on sex work, public health and feminism, in this post I will discuss how I think anti-sex work feminists have coopted the movement against human trafficking in their political campaign against the sex industry. I have shown before that I don’t think moralist campaigners against sex workers have the best interests of the women in the industry at heart, and that anti-sex work feminists show a similar instrumentalization of the women in the industry: both groups aim to use sex workers as pawns in their broader social reform agendas. In pursuing this agenda, I think these campaigners have also worked to try and redirect the focus of national and supra-national movements against trafficking, and have stirred up alarm about the level of criminality and security risk associated with the problem. I also think their academic contributions to the issue have been weak and biased.

    Human trafficking and people smuggling

    Human trafficking is defined in the framework convention against trafficking as the

    recruitment, transportation, transfer, harbouring or receipt of persons, by means of the threat or use of force or other forms of coercion, of abduction, of fraud, of deception, of the abuse of power or of a position of vulnerability or of the giving or receiving of payments or benefits to achieve the consent of a person having control over another person, for the purpose of exploitation. Exploitation shall include, at a minimum, the exploitation of the prostitution of others or other forms of sexual exploitation, forced labour or services, slavery or practices similar to slavery, servitude or the removal of organs

    Sex trafficking is a part of this, and is a serious crime against humanity. Human trafficking doesn’t necessarily involve just sex trafficking, however: it can happen on European fishing boats or even to hair-dressers on US army bases. It’s also different to people smuggling, which is the movement for profit of humans across borders, without the intention of exploiting them upon arrival. Many women use people smugglers of varying degrees of nastiness and depravity to cross borders illegally to do sex work, but this doesn’t mean they’re being trafficked. For example, when I started working in public health in the 1990s Asian sex workers would come to Australia under very tough contracts to work as sex workers. These contracts would require them to work for 6 months or a year without receiving payment, in order to repay the “cost” of being smuggled into the country. Most of the women in these situations knew what they were getting into, and weren’t particularly interested in cooperating with customs and immigration agents. Their working conditions were harsh and very bad things could and did happen to them, but they weren’t generally deceived or coerced into the work, and were willing to take the risks because of the potential lucrative benefits that would accrue to them after they discharged their contract. Conflating these women with trafficked women is a dangerous error.

    But conflating people smuggling and sex trafficking, and exaggerating the security and public health risks of the latter, is an important tool in the rhetorical arsenal of modern anti-sex work campaigners. As the developed world began to bend its considerable state power to the abolition of human trafficking, anti-sex work campaigners saw an opportunity to use the apparatus of border control and the rhetoric of slavery to ratchet up the pressure on the sex industry generally. Domestic campaigns against the legalization of prostitution have tried to have all prostitution redefined as coerced, and a similar approach has been used in attempts to get human trafficking laws in the USA changed to remove the requirement for evidence of fraud, deception or coercion (such as happened in New York). Simultaneously, campaigners have exaggerated both the numbers of people involved and the seriousness of the organized crime involved and its security ramifications for countries “at risk.” Shifting the goalposts so that all prostitution is redefined as trafficking serves the anti-prostitution movement well, but it has bad effects in both diverting attention from more serious forms of human trafficking, and in stigmatizing and criminalizing sex workers. It also tends to lead to more punitive immigration law, which affects poor women coming from poor countries without visas in increasingly harsh ways. Sex worker organizations are clear that these laws don’t work to protect women in the industry, but make their lives harder and more dangerous.

    False statistics and shoddy research

    Although successful in bringing state power to bear on poor women, and redefining prostitution as slavery, anti-trafficking campaigners have had remarkably little luck in proving that the problem is significant. Although they claim that Sweden’s anti-prostitution laws led to a large reduction in trafficking, they have very little evidence of the presence of trafficked women in the developed world. For example, a 6 month intensive project in the UK failed to find evidence of a single trafficked worker, and the major report into trafficking in the UK, Big Brothel, was greeted with harsh criticism by public health researchers for both its methods and its ethics. This report in the Guardian shows how 71 arrests for trafficking in 1998 became a Daily Mirror headline of “25,000 sex slaves in the UK” by 2008, through a series of misrepresentations by organizations that obviously have increasing amounts to gain from misrepresenting the issue. Academic critics of the Big Brothel report wrote that

    The report builds a damning picture of indoor sex work on the basis of data whose reliability and representativeness is extremely doubtful and a methodological approach that would be considered unethical by most professional social researchers. It makes claims about trafficking, exploitation and the current working conditions of women and men employed in the indoor sex industry on the basis of that data.

    Unfortunately, these activists had the ear of governments in Sweden and the UK, and laws were changed redefining any form of sex work as exploitation. The result of this in Britain was that women were forced to work alone, without managers, receptionists, colleagues or security staff, in dangerous privatized settings – the worst possible scenario for sex workers. This doesn’t stop women doing sex work, which is lucrative and flexible work; it doesn’t stop poor women risking their life savings in the hands of people smugglers to come to the UK to work; but it certainly makes this process less reliable and more dangerous. Meanwhile, other forms of slavery – on construction sites and fishing boats and farms all around the developed and developing world – receive much less attention than the headline issues of sex slavery. Furthermore, anti-sex work campaigners have exerted considerable pressure on developing nations, such as Cambodia, through organizations such as the US State Department, to try and get them to redefine the internal, voluntary movement of poor rural women to sex work in the city as “trafficking.” Given the large demographic and labour market shifts occurring in some of these countries, it is inevitable that women will move to the city to do sex work, and such redefinitions will simply see them cast out of the industry they chose and forced into more dangerous and less rewarding industrial jobs. Why should they lose this choice, just because some western feminists want to change the nature of social relations in their own backyard through rhetorical gains against prostitution overseas?

    Globalization, sex work and reaction

    The free movement of global labour has thrown up a lot of challenges to both the old right and the old left, and some of their responses to those challenges haven’t been particularly pretty. Both sides of politics have responded to this aspect of globalization with knee-jerk isolationism and racism. When sex and prostitution enter the mix, especially the sleazy low-rent end of the sex industry that migrant labourers often fill, the responses can become hyperbolic. But although it may be particularly seedy and unsavoury, it is ultimately an industry like any other – where a need exists and people are willing to fill it, they will; and where the profit is to be made, people will act as brokers for the movement of labourers in and out of the industry. Just as in the construction, cleaning and other service industries, this will often mean that unscrupulous dealers attempt to cut immigration corners, and in this case the high profits that they may obtain mean that both the brokers and the labourers are willing to risk a lot of money on the enterprise. This may make the brokers seedy and unpleasant, but it doesn’t necessarily make them slavers. It also puts many poor women from poor nations at odds with the law, places them in vulnerable and dangerous positions, and seriously increases the risk that bad things will happen to them. But the answer to this is not to redefine the behaviour of these people as slaver/victim, to create false statistics and attempt to bring the full might of the US state to bear on both the originating countries and the women in question. Rather, it is to liberalize immigration laws, to make immigration enforcement as humane and reasonable as possible, and to always insist on a clear line of understanding between the types of crimes involved – prostitution and people smuggling are not the same as trafficking, and redefining trafficking will not protect the women who willingly risk immigration prosecution to make money in this or any other industry.

    Of course, none of this is inconsistent with the real goals of anti-sex work campaigners, which are to alter the nature of gender relations to a reactionary, illiberal model that has little in common with the aims and aspirations of most people, and certainly does not serve to liberate women. In my last post in this series, I will use the evidence I’ve gathered so far, and the public utterances of key figures in the movement, to describe why I think the ultimate goals of the radical feminist anti-sex work movement are reactionary and dangerous for women everywhere, and why I think this movement is both misogynist and backward-looking.