• I have now had quite a bit of experience working with large datasets in Stata, and consistent with my previous efforts on this blog to publicize pr0blems with statistical software and solutions to computer problems, I thought I’d explain how I do it and why it’s a good idea to use Stata for large data. I approached this problem in 2008, when I was living in London and working with National Health Service (NHS) data. At that time it was a seemingly insoluble problem and there wasn’t much information out there about how to solve it; I guess since then things have improved, but just in case the information is thin on the ground, I thought I’d write this post.

    What size is “large”?

    When I researched solutions to the problem of analyzing large datasets in Stata, many of the people I contacted and the websites I looked at thought I meant data consisting of hundreds of thousands of records – this is a common size in statistical analysis of, e.g. schools data or pharmaceutical data. I was working with files of 100s of millions of records, up to 30Gb in size, and back in 2008 very few people were working with this size. Even now, this is still pretty uncommon in epidemiology and health services research. Four years of outpatient data from the NHS will contain about 250 million records, and the chances are that the correct analysis you need for such data is a multi-level model (facility and patient being two levels) with binary outcomes. With this kind of data most health researchers make compromises and use the linear probability model, or other approximations and workarounds. Most researchers also use SAS, because SAS is the only software package capable of analyzing files that don’t fit into RAM. However, it takes an enormous amount of time to do a logistic regression on 250 million records with SAS – my colleague would leave it running all day, and work on a different computer while he waited for it to complete. This is not acceptable.

    Why Stata?

    I’m not a fascist about statistical software – I’ll use anything I need to to get the job done, and I see benefits and downsides in all of them. However, I’ve become increasingly partial to Stata since I started using it, for these reasons:

    • It is much, much faster than SAS
    • It is cheaper than SAS or SPSS
    • Its help is vastly superior to R, and the online help (on message boards, etc) is much, much politer – the R online help is a stinking pit of rude, sneering people
    • R can’t be trusted, as I’ve documented before, and R is also quite demanding on system resources
    • Much of the stuff that epidemiologists need is standardized in Stata first – for example, Stata leads the way on combining multilevel models and probability sampling
    • Stata’s programming language, while not as powerful as R, is still very flexible and is relatively standardized
    • Stata has very good graphics compared to the other packages
    • SAS is absolutely terrible to work with if you need automation or recursive programming
    • Stata/MP is designed to work with multi-core computers out of the box, whereas R has no support for modern chips, and SAS requires some kind of horrendous specialized set up that no one with a life can understand

    So, while I’ll use R for automation and challenging, recursive tasks, I won’t go near it for work that I really need to get trustworthy results on quickly, where I’m collaborating with non-statisticians, or where I need good quality output. I gave up on SAS in 2008 and won’t be going back unless I need something that only SAS can do, and I don’t think SPSS is a viable option for serious statistical analysis, though it has its uses (I could write a very glowing  post on the benefits of SPSS for standardizing analysis of probability survey analysis over large organizations).

    The big problem with Stata is that, like R, it is vectorized, so you need to load the entire data file into RAM in order to be able to do any analysis on it. This means that if you want to analyze very large data sets, you need huge amounts of RAM – whereas in SPSS or SAS you can load it piecewise and analyze accordingly. Furthermore, until Windows 7 came along it was not possible to give more than 700Mb of RAM to any program (unless you were using Mac OS X/Unix), so you couldn’t load even medium-sized files into RAM. Sure, you could use Windows Professional 2000 or some such nightmare mutant package (which I tried to do) but it’s hell on earth to go there. Your best option was Mac OS and a huge amount of RAM.

    I’m going to now prove that it’s better to buy Stata and invest in 32 or 64 Gb of RAM, than to keep working with SAS. And I’m not going to fall back on hazy “productivity gains” to do so.

    Conditions for analysis of large datasets

    The core condition for analysis of large datasets is sufficient RAM to load the entire dataset – so if you expect your basic analysis file to be 12Gb in size, you’ll need a bit more than that in RAM. If the file is coming in a size larger than this, you’ll need a database package to access it – I use MS Access, but anything will do. If the file comes in text (e.g. .csv) format you can break it into chunks in a text editor or database package and import these into Stata sequentially, appending them together. Also, don’t be discouraged by larger file sizes before you import – Stata has very efficient data storage and by careful manipulation of variable types you can make your data files much smaller. Also, if you are importing sequentially you can drop variables you don’t need from each chunk of file before appending. For example, if you receive NHS data there will be a unique id derived from some encryption software that is about 32 characters long. Turn this into an integer and you save yourself about 16 bytes per record – this adds up over 250 million records. Some spatial data is also repeated in the file, so you can delete it, and there’s lots of information that can be split into separate files and merged in later if needed – in Stata it’s the work of a few seconds to merge a 16 Gb file with another 16 Gb file if you have sufficient RAM, whereas working with a single bloated 25Gb file in SAS will take you a day. It’s worth noting that SAS’s minimum sizes for a lot of variable types are bloated, and you can shave off 30-40% of the file size when you convert to Stata.

    So, loop through chunks to build up files containing only what is relevant, compress them to minimum sizes, and use a judiciously constructed master file of IDs as a reference file against which to merge data sets with secondary information. Then, buy lots of RAM. You’ll then have the dual benefits of a really, really nice computer and a fast statistical analysis package. If you were working with large datasets in SAS, you’ll have cut your analysis time from hours to seconds, increased the range of analyses you can conduct, and got yourself improved graphics. But how are you going to convince anyone to buy you that computer?

    Stata and a large computer is cheaper

    Obviously you should do your own cost calculations, but in general you’ll find it’s cheaper to buy Stata and a beast of a computer than to persist with SAS and a cheap computer. When I was in the UK I did the calculations, and they were fairly convincing. Using my rough memory of the figures at the time: SAS was about 1600 pounds a year, and a basic computer about 2000 pounds every three years: total cost 6800 pounds every three years. Stata costs 1500 pounds, upgrades about every 2-3 years, and a computer with 32Gb of RAM and 4 processors was about 3000 pounds. So your total costs over 3 years are about 2300 pounds less. Even if you get a beast of an apple workstation, at about 5000 pounds, you’ll end up about even on the upgrade cycle. The difference in personal satisfaction and working pace is huge, however.

    Conclusion

    If you work with large datasets, it’s worth your while to switch to Stata and a better computer than to persist with slow, clunky, inflexible systems like SAS or SPSS. If you need to continue to interact closely with a large SQL backend then obviously these considerations don’t apply, but if your data importation and manipulation needs are primarily flat files that you receive in batches once or twice a year, you’ll get major productivity gains and possibly cost savings even though you’ve bought yourself a better computer. There are very few tasks that Stata can’t solve in combination with Windows 7 or Mac OS X, so don’t hold back – make the case to your boss for the best workstation you can afford, and an upgrade to a stats package you can enjoy.

  • In my previous post on Obamacare commenter Paul has suggested I’m putting too much faith in government intervention to reduce inequality or contain costs. I’m about to go away for the weekend so don’t have much time to attend to my blog (nor may I next week, when classes start) but here’s a pair of questions that I think are related to that question, and to the (apparently still unsettled!) government vs. private debate. These two questions possibly also show how much we don’t know about health systems.

    Did the NHS reduce health inequality?

    One of the big claims of the NHS is that it reduced entrenched health inequality by giving poor people access to healthcare they were previously denied. I’ve implied before that I’m not confident the NHS has achieved that much in this regard, and pointed to the existing health inequalities in the UK as evidence of this. I’m loathe to say that it achieved nothing, but this fascinating and excellent paper from the British Medical Journal, published in the year 2000, suggests that despite the estabishment of the NHS, health inequality in Britain has persisted for 100 years. So is it the case that the huge intervention in the market that is the NHS achieved nothing in reducing inequality?

    Does the US health system need political, not system reform?

    It’s true that there’s very little evidence that private health markets reduce inequality or contain costs, but there are only two developed nations that have actually conducted this experiment (the US and Switzerland), and although the experiment is ongoing in many developing nations they don’t provide a good health policy laboratory (due to all the development issues and tropical diseases they are dealing with simultaneously). Switzerland, obviously, is a bit special, so there is really only one major economy that is actually trialling an even close-to-privatized system, and there’s a big problem with the experiment going on there: the USA is a plutocracy, not a democracy, and its capitalist system is pretty busted (see, e.g. “Too Big to Fail”), in the sense that it is heavily captured by special interests and the political system in which it is embedded is corrupt, unrepresentative and basically not democratic. Furthermore, the USA has a significant race problem that doesn’t exist in other places, is historically very specific to the USA, and creates a whole set of social problems that a place like Switzerland or Australia doesn’t have to cope with. So is it possible that a root-and-branch political reform, based on breaking the sectional interests in the US economy and the power of the super-rich to influence politics, would enable a purely private health system to function? If so, it’s unlikely that any attempts to salvage the market-based system that are based on regulation or minimal government intervention are going to work, because of the power of those sectional interests. Should proponents of market systems for health care be looking to developing nations for their examples?

    I’m off to take an extended bath in the country! So comments won’t get much attention until next week …

  • My weekly TOC from the New England Journal of Medicine (NEJM) contains some interesting articles about cost containment in health care, including some discussion about what Obamacare might or might not do to contain costs, based on lessons from Massachusetts (RomneyCare?). Apparently the Affordable Care Act contains some quite innovative and also potentially punitive measures to force hospitals to reduce rates of readmission after discharge, but has missed some other opportunities in the mix. The NEJM has clearly moved on from debate about whether Obamacare is a good or bad thing, and debate about whether it’s going to be constitutionally viable, to discussion about how it will change the health industry. I wonder if this is going to make the NEJM (the world’s top medical journal) look stupid in about 6 months’ time…?

    The first article in the NEJM contains a discussion of the problems facing the health system in Massachusetts, which famously introduced a version of Obamacare a few years ago. This system essentially forces people to take up a health insurance plan, penalizing those who don’t, on the assumption that such a model will improve equity and reduce costs through expanding the risk pool. Unfortunately, it appears that the plan hasn’t led to any serious levels of cost containment – according to the NEJM article, costs have risen significantly since 2006. The basic cost for a “bronze” package is $275 per month, which I’m pretty sure is significantly higher than my “silver” care package in Japan, and is $100 higher than it was just a year or two ago. The article claims that this is because of problems in the payment system in the Massachusetts system: it’s not the insurance plan per se, but the way in which hospitals are paid for providing services. Hospitals have been paid under a fee-for-service plan, and this encourages providers to charge for extra services and make extra money. Massachusetts is solving this problem in two ways: by moving to “global health payments” and by forcing people into “accountable care organizations” (ACOs). The former is a system of payment based on the expected total cost of a person’s illness, rather than the services provided within it, designed to penalize providers (i.e. hospitals) who fail to deliver the required care within the expected cost framework associated with any given disease. The latter are large organizations charged with taking responsibility for the care of a group of patients (similar, I think, to the NHS’s clinical care commissioning), and composed of large numbers of amalgamated insurance purchasers. Essentially, “global payments” are an attempt to exit from the fee-for-service system towards a payment system that discourages over-servicing, and ACOs are an attempt to merge small healthcare providers together to give them more negotiating power (as an example, all public employees are going to be combined in one ACO). These two reforms in tandem look suspiciously like an attempt to force the Massachusetts system towards something that looks like the NHS. There’s a hint here as to a basic fact – on the commissioner (health insurance) side, cost containment is best achieved by consolidating commissioners in order to achieve greater bargaining power with providers, and the maximal example of this is the government-run single-payer system. It appears that Obamacare is likely to have a lot in common with this system.

    The next article describes Medicare’s attempts to reduce costs through reducing readmissions, and describes some initiatives in Obamacare to reduce readmissions in order to contain costs. Reducing readmissions is something of a holy grail in healthcare cost containment, because readmissions are a theoretically avoidable source of significant healthcare costs – you got your operation, left hospital but had to come back for some kind of medical complication, so you represent a hospital admission that can be prevented through hospital-focused quality control (rather than lifestyle change). Reducing these leads to reductions in overall costs, and better patient experience (no one wants to go to hospital). The article includes a fascinating discussion of what Obamacare aims to do to reduce readmissions – penalizing the worst offenders – but also discusses why for many hospitals reducing readmissions is not a viable financial goal:

    Unless they are at maximum capacity, hospitals face two major economic disincentives to reducing readmissions for the specified diagnoses: the direct costs of the program itself and decreased revenues resulting from successful interventions. Interventions to create and sustain reductions in readmissions typically average $100 to $200 per discharge and often have spillover effects, decreasing hospitalizations for nontargeted diagnoses and reducing readmissions from any cause even outside the 30-day window and across payers. Although these effects are desirable outcomes for patients and payers, they detrimentally affect hospitals’ finances.

    This is an interesting and often-overlooked aspect of the private healthcare market: it’s not in the interests of hospitals to reduce rates of illness, since they get paid by insurers to treat sick people. And for hospitals that don’t have outrageously high rates of treatment failure, a little bit of treatment failure is (financially) a good thing[1]. This kind of concept just does not apply in the publicly-run systems of the UK and Australia. The article goes on to introduce the concept of a “warranty” rather than a penalty, under which hospitals have to provide care for a condition at a given price and as part of that care have to offer a warranty – so readmission to hospital gains them no financial benefits, since they have to correct any post-surgical complications without extra payment. This forces hospitals to get it right the first time.

    Unfortunately it’s not that simple. A third article in the NEJM challenges the idea that reducing readmissions is a worthy goal. It points out that, although numbers of readmissions may vary substantially, rates may not (and rates are what it’s all about in this business). Furthermore, there is evidence that readmissions, although apparently preventable, are often outside hospital control:

    The growing body of evidence suggests that the primary drivers of variability in 30-day readmission rates are the composition of a hospital’s patient population and the resources of the community in which it is located — factors that are difficult for hospitals to change.

    There is also evidence that readmission rates may increase when service improves, or that patients may be happier when readmission rates improve, because good quality continuity of care may identify additional health care needs:

    whereas some studies have shown that sustained efforts can reduce readmission rates somewhat, others have shown that interventions aimed at improving care coordination and access to follow-up care actually increased the rate of readmissions, presumably because of improved access to needed care, with commensurate improvement in patient satisfaction

    The authors of this article also point out that in focussing on readmissions the system privileges cost containment over quality of patient experience (i.e. productivity), which is a kind of cost containment in itself, though perhaps less quantifiable in a fee for service system:

    over the past decade, we have seen very little improvement in patient safety, and although mortality rates have declined for a few conditions, they remain high for most others. Many of these deaths are preventable. Yet we are focusing tremendous resources on preventing rehospitalizations for three conditions that account for approximately 10% of all hospital admissions in the Medicare population.

    They suggest that we should focus on patient safety, and if we’re going to focus on avoidable readmissions we should focus on very narrow time frames (3 or 7 days), the conclusion being that hospital readmissions are not where we are going to find our savings. I think this is another example of how “efficiency gains” never materialize in health – regardless of the health care system you are experiencing, when you hear politicians talk about “efficiency gains” you should think “oh! a wanker speaketh!” Especially if they talk about sacking back office staff (but a statistician would say that).

    The thing I think is most fascinating about these three articles is what they imply about the state of knowledge of health care systems today. Despite 100 or so years of development in health insurance and health systems, modern health care theory is incapable of working out exactly what we want from healthcare. Even worse, modern healthcare markets are constructed without any real understanding of what kind of product we’re buying when we participate in them – we know we want “health” but we don’t know whether being readmitted to hospital is a good or bad thing, and people buying health care (insurance companies) and selling it (hospitals) not only can’t agree on a mechanism for setting prices, but the hospitals are profiting from selling (in some cases) a product no one wants to buy, but are forced to buy the hospitals’ own mistakes (hospital readmissions). What kind of private market works under these conditions? It’s not an information asymmetry, it’s an information fuck up. Also noteworthy is the gradual movement of the Massachusetts system towards a form of NHS-lite, in order to reduce costs. Is that telling us something?

    The NEJM is holding a fascinating debate on health system performance and planning, within the (admittedly limited) context of the US system. As this debate plays out it gives the rest of the world a fascinating insight into how weak the theory underlying privatized markets for health care really is. It’s not that market systems are fundamentally right or wrong – it’s more that we haven’t worked out how to price health care, so price signals just don’t work. Can such a thing be done? And if so, would an entirely private market in health care work? All my posts to date claim that an entirely private market is a disaster and an impossibility – but is the problem not the notion of a private market per se, but that we don’t yet have a sophisticated enough understanding of health (and, conversely, of markets) to be able to construct such a thing in the first place?

    fn1: obviously a lot of treatment failure gets you shut down and your managers charged with negligent homicide, so it’s a bad thing[2]

    fn2: no one, however, is suggesting that hospital administration are maintaining such a balancing act. What they *are* doing is prioritizing what they consider to be more important issues, like quality of patient care (see article three).

  • Your zombie getaway vehicle…

    As a holiday location Hakone is a little over-rated, but as a zombie survival spot it has some good points. It lacks the defensibility of Mount Takao, but its relative remoteness, inaccessibility, climate and water features offer some survival opportunities. It’s also quite a beautiful spot to be torn apart by zombies in, though that’s not our intention here. Hakone is located about 2 hours’ journey West of Tokyo (if you catch the bullet train part way) in the province of Kanagawa, and consists of an ancient volcano caldera, with a lake in the middle of thickly forested hills. From some locations within the caldera it is possible to see Mount Fuji, though the weather around here is quite bad and it is often cloudy or obscured. The nearest large town is Yumoto, which is on the far side of the hills from the lake. Hakone itself is accessible through one or two twisting, winding roads that ascend and then descend the caldera. There is also a train and a cable car. The lake is a tourist resort in spring, summer and autumn, and features a rather amusing ferry fashioned on a pirate theme (see below), as well as a variety of paddle boats and smaller cruisers. There are three towns on the lakeside, linked by a single road that winds through thick forest, and by the pirate ferry.

    Pirates vs. zombies!

    Review

    Location: Hakone is reached by only a small number of winding roads, which feature many switchbacks and narrow, easily blockable chokepoints. The road is, of course, lined with towns, but these are small and easy to pass through and mostly lower down the mountain – also unlikely to be zombified before the larger towns closer to Tokyo. It would be quite easy to pass beyond a certain point and block the road against zombie encroachment, forcing them to work around the blockade and struggle up through densely forested steep mountains.

    This brings us onto a new aspect of zombie hording theory: how do zombie hordes proceed? I presume they flow along the path of least resistance, which in an urban setting would mean that they flow along roads, with more zombies on wider roads. Furthermore, all the work I’ve read and seen on zombies suggests that they have very limited, if any, information exchange. This leads to some interesting dynamics, because it means that they don’t behave like ants or bees – they don’t lay paths and follow them. So it seems reasonable to suppose that as they leave cities they are more likely to move down major roads, and if they don’t find prey they will slowly disperse. Thus, only small numbers can be expected to proceed along narrow, winding, low-prey-density country roads, and at some point they can be expected to stop and mill around – as we see in quite a few scenes from the major documentaries. This suggests that a remote location reached only by narrow, winding roads with low-density settlements sparsely located (as is the case on the higher reaches of Hakone) is likely to be missed by the majority of zombies.

    Defensibility: Hakone has a harsh winter – right now, in mid April, the cherry blossoms have not yet bloomed, for example, which must mean that between December and March it is very cold – it even snowed when I was there last weekend. A lot of people think that zombies freeze in the cold, which in the case of Hakone evokes the possibility of a kind of “zombie line,” above which zombies can’t pass. Defenders of Hakone could venture into the snows of late winter/early spring and slaughter frozen zombies on the slopes of the mountains. Additionally, this winter cold, though harsh on people living in Hakone, offers the possibility of a three month respite from zombie attacks – surely a valuable haven of stability in a zombiepocalypse.

    Also, Hakone has the lake, and the pirate ships. In a pinch, one could sail one of these pirate ships to the middle of the lake, dragging a couple of pedal-powered swans with it, and set it up as a safe haven far from the zombie hordes. The pedal boats could be used to travel to the shore for supplies. At least in the short term this offers an opportunity to ride out the worst of the initial period of the apocalypse, and to work out what to do next. Then, there are many areas of the shore which are so thickly forested that they are almost impossible for a zombie horde to enter rapidly without making a noise. One could clear a section of this forested area and set up some fields and a house, combining potato crops and potentially a rice field or two with fish from the lake (and seabirds). If zombies attacked, a quick hop onto a pedal-boat would get one out to the pirate ship and temporary safety. There are also, of course, buildings higher up the hillsides, which could be quite easily fortified. Unfortunately a lot of these are large hotels, which offer the advantage of being able to hide deep inside, but the disadvantage of being extremely difficult to secure.

    Concealment: Being in a bowl, most of the locations inside Hakone are visible when one enters the valley, so things like smoke and lights at night might be visible to zombies cresting the mountains. However, from the valley base each town is largely disconnected from the other towns, with the bulwark of the mountains and thick forests preventing zombies in one town from seeing another. Setting a barricade on the roads between the towns, it would be quite easy to ensure that the only zombies that approached the town did so along a narrow and defensible road – those that entered the forest would likely soon lose sight of human habitation and wander listlessly, losing their horde properties and becoming less threatening to a community.

    Sustainability: Hakone has a lot of tourist establishments, offering short-term sources of food, and it has a guaranteed supply of water (the lake and its tributary streams) as well as the opportunity to fish for food for most of the year. However, the steep mountains and harsh winter weather may make long-term survival here difficult, and it is likely that, outside of fire wood, there are no more reliable sources of energy – perhaps a dam could be built, but it would be a significant engineering task. Enterprising survivors could turn the many onsens in the region into a power source, but this would likely require the support of some skilled engineers. Thus, if one wants to make Hakone sustainable as a survival location, one would need to ensure that one had at least one engineer and someone with a fairly robust knowledge of farming when one arrived. Otherwise, it is likely that Hakone would serve best as a winter redoubt, somewhere to retreat for the first winter after the apocalypse while one attempted to work out a long term survival strategy. Things could get bitter here in winter, and not all survivors could be guaranteed to emerge at spring, depending on how well stocked the hotels were and how many survivors fled here.

    Conclusion

    Hakone offers short-term survival opportunities, the chance to build a home and spend a winter free of zombie harassment while a group of survivors tries to develop a long-term plan. The steep hillsides, harsh winter and lack of local power supplies mean that it may not be a viable long-term survival location, but its secluded location and the presence of many hotels means that it may be a suitable location for a small number of survivors to sit out the worst of the initial stages of the apocalypse. It’s worth remembering that in the first stages of the apocalypse, 30 million or so Tokyo residents are going to zombify and head into the country – having a secluded location to wait for this process to stabilize is a very useful first plan. So there are worse places to go than Hakone, but it is potentially less defensible and more difficult to survive in than Takao. It lacks one significant aspect of Takao – height, and the ability to make low-tech zombie traps. But it has one significant advantage – the opportunity to retreat to the lake centre if a horde approaches. As a short term survival choice, it is not perfect, but definitely better than attempting to survive within Tokyo.

    Addendum

    If you appreciate my thoughts on zombie survival strategies, please consider reading my novella Quarantine Breach, set in the world of 28 Years Later, which is freely available at Royal Road.

  • The dragon gets what the dragon wants

    On the weekend, the group I was playing with screwed up our GM’s adventure from the very first scene, and from that point on he spent the entire session inventing new characters, story lines and encounters as we stumbled from misunderstanding to misunderstanding, culminating in the three-way stand off depicted above. We asked our GM afterwards, and as far as we know the adventure was supposed to involve us killing a black dragon, then a necromancer reanimating that dragon, us killing the undead dragon, then us tracking down and killing the necromancer. Fairly standard stuff, and the adventure opened with the dragon attacking our tavern, so we could have set off down that path straight away.

    Unfortunately, we assumed – I think, fairly – that the dragon was too tough for us and that the only option was to negotiate with it. So we went and chatted, and the GM let us. What followed was a train wreck, that was rescued at every turn by our GM laying on an increasingly complex and entertaining adventure. Instead of three straight fights and treasure, we instead agreed to find a lich for the dragon; agreed to find the lich for a wizard called Magister Tiana who we thought was an enemy of the dragon; went to meet a dubious infernal contact of mine, who thought letting the lich go would be a good idea; investigated a crematorium; watched an auction where all the bidders were goth halflings; fought and killed the lich; made a lich compass; lost the lich to Magister Tiana; investigated the lich’s hotel room, where we thought we found evidence of a third force looking for the lich (probably the thieves’ guild); met the wizard that the lich was chasing, a chap called Malachy who was on the lam from the Wizard’s Guild; arranged a meeting between Malachy, Tiana and the Dragon thinking that there would be a three-way stand-off; fought the lich again; fought Malachy as he did a runner; and got a ride on a dragon to meet the heads of the Wizard’s Guild.

    As far as I know none of these events were meant to happen. A few aspects of the adventure that were particularly entertaining:

    • The town itself: we were in a town called Red Lanterns, that is built on the back of a behemoth tortoise. The town comes alive at night when the tortoise sleeps and sleeps during the day when the tortoise walks; this tortoise is one of 10 such beasts treading a steady path in a circle around the continent, and its pelagic nature makes it a haven for renegades – it has no laws. We didn’t bother finding any of this out when we visited the town.
    • The goth halfling auction: In this town the bodies of the deceased are cremated, and then their ashes are auctioned off to the highest bidder – our GM told us he got the idea from Star Trek Deep Space 9[1]. We naturally assumed the lich was after something in the ashes, and so we went to watch an auction and see if he was present. On the day we visited, a halfling was being auctioned off and two factions in his family were involved in a bidding war that was causing some deep tension. All of them were, of course, dressed in black, and the entire audience of bidders were halflings, bidding for the ashes of their own uncle. This scene cracks me up every time I think of it. It was interrupted by the lich outbidding all attendant halflings, who responded to his intrusion by attempting to shoot him, at which point he turned into a swarm of cockroaches and ran, with us chasing.
    • Magister Tiana and the dragon: so we first offered to find the lich for the dragon if he would leave the town alone, and he agreed. Then within a few hours a wizard, Magister Tiana, visited us and told us that she was mates with the dragon. We didn’t believe her because she didn’t tell us she knew we had arranged a deal with the dragon, and in fact we were able to cut another deal with her to get the lich for her, with a bonus if we found out who he was working for. Why would she do this if she was an ally of the dragon that had already got us working for free? I’m not sure why the GM did this, or why we cut a deal with a rival of a dragon (thinking about this for even a moment, it’s really not a good idea to double cross people with this kind of power), but he did and we did, and thus the flavour of the adventure turned into one of those “everyone’s out to get Wally, let’s get him first” type stories. They always end well!
    • The fugitive wizard: after we had killed the lich and lost his body (aren’t we smart!) we searched his spellbook and found notes in it indicating that he was chasing some guy called Malachy, who was hiding in the local wizard academy. We found him, and discovered that he was on the run from the wizard’s guild due to an “accident” in which he accidentally crashed one of their sky castles. He was on the run from the lich after a confrontation in which he somehow permanently destroyed the lich’s eye and one hand. When he found out Tiana was in town  he got all scared and started thinking of running, but somehow we convinced him to meet Tiana and hand himself in.
    • The final stand off: we arranged the final stand-off thinking that Tiana and the Dragon would turn up separately, see each other and toast one another, and we would hand Malachy to the winner and loot the loser[2] – we remained convinced she’d lied to us right up until the point that she rode in on his back, carrying the lich’s head. Thus we found ourselves in the situation depicted above, with her and Malachy having a robust chat under the watchful eye of the dragon. Things went pear-shaped because Tiana had brought the lich’s head with her, and it got loose and started trying to waste everyone so that it could catch Malachy – apparently he was quite the prize. We, naturally, sided with the dragon, and then Malachy did a runner while dragon, Tiana and lich were engaged in fearsome battle. We caught Malachy and dragged him back, and that was that.

    I think this adventure is a credit to the GM. Every part of it was fabricated on the spot to help us continue charging around the town making mistakes, and although we were starting to suspect we’d cocked it up, at no point did he let on which bits were in the plan and which weren’t – we were convinced the halfling auction was in his original notes, for example. He was creative and energetic throughout the whole process, he managed to tie together disparate elements of the plot even as he was making them up on the spot, and somehow at the end everything was resolved neatly and clearly – all of this in the space of about 5 hours. I think this kind of creativity and flexibility is the mark of a good GM, especially when it’s in response to your having thrown all his preparation out the window from the first encounter. We didn’t intend any of this disruption, we just genuinely misinterpreted the purpose of that first battle – like most players, if he had said “guys, this adventure is meant to involve you fighting this dragon” we would have taken it on, but he didn’t, and so we did what comes naturally to a bunch of cowards, and supplicated the damn lizard. But he didn’t correct us, presumably having faith that he could somehow muddle up an adventure regardless, and that’s what happened. He told us later that he decided many of the plot elements based on our assumptions, so that we were driving the plot forward, which is also a very fine thing to do. The man was an improvisational genius.

    If there is any lesson in this for better adventure planning, I guess it’s that you shouldn’t make an adventure’s entire plot hinge on players deciding to fight a dragon – many players assume dragons are too tough for them, and if the first encounter of the day is a dragon they will assume negotiation is the key. But it also shows that if you’re a good GM with a healthy attitude, even when your players completely cock up your plans from the very start, you can still make a great adventure. And our GM this day was not just a good GM – he was a great GM. This is GMing at its finest, in my opinion.

    Finally, to top it all off, once we’d finished for the day we offered to do a test fight against the dragon, to see if our first decision was right. It was a close thing, but we killed it. So even our decision to negotiate was wrong!

    fn1: And they say Star Trek never benefited humanity!

    fn2: we were stupid and evil!

  • He went thattaway …

    During this weekend’s gaming session, we killed a lich. Good job! おつかれさま!Unfortunately, we didn’t know where our victim’s phylactery was, so we couldn’t actually eliminate the lich – we needed to find the phylactery, but the only way to do that was to let the lich reanimate its body and then follow him to his lair, kill him again and this time destroy his phylactery. All in a day’s work, etc. We had his body and disembodied head (gagged, of course) and we’d stuffed the head in a sack, then stuffed the sack in a barrel of water, then put a lid on the barrel, so we could make plans without fear of being overheard. So we hatched a scheme to follow the lich, but there was unfortunately a small hitch – we knew that this lich had the power to dissolve his body into a swarm of cockroaches and scuttle away to his dark and stinky home. We were in a city, where following a swarm of cockroaches is impossible. So, we designed a lich compass. This is how our plan worked:

    1. Carve a chunk of flesh from the lich’s body, near one of its injuries where it is unlikely to notice the piece is missing
    2. Put this chunk of flesh in a jar large enough for a cockroach to move around
    3. Take the body and the head to a park, build a fire large enough to make a lich think we’re trying to destroy its body, but not large enough to completely incinerate the body
    4. Chuck the lich on (and its head, of course)
    5. After about 10 minutes of watching, tell each other that the job is surely done, and walk away
    6. Hide
    7. Once the fire has died down, the lich body will slowly reanimate, and the lich will collect his head and go shambling, smokily, to his lair
    8. Follow him
    9. The big risk is that he will turn into a swarm of cockroaches. This is pretty likely, since his clothes will have been destroyed and he’s covered in burns – he probably wants to go home in a slightly less conspicuous form
    10. When he turns into cockroaches, pull out the jar
    11. All the cockroaches will be heading in the same direction – this means that the cockroaches in the jar will be trying to push through whatever side of the jar is closest to the lich’s path
    12. Follow the compass!

    Obviously this plan has a large number of flaws – I count 11 (anyone can follow a compass![1]). Our compass was based on two big assumptions:

    1. The lich doesn’t have a detailed knowledge of the location of every piece of his body, but the pieces are somehow spiritually connected
    2. When the lich casts a spell on himself, all parts of his body are affected

    Obviously some GMs would take issue with these assumptions, but I don’t think they’re unreasonable. If the GM accepts this particular approach to Lich corporeality, then I think our compass should work. Also, perhaps, it would work on Vampires.

    Unfortunately, we never got to test our assumptions because a crazy wizard teleported in, stole the lich’s head, and teleported out again. The GM then revealed that the lich can disconnect his soul from his body, leaving the body to rot, so all we ended up with was a jar full of rotting flesh. But I think if the wizard hadn’t appeared, we would have been good to go – he disconnected his body because he’d been teleported out, not because he knew we were hiding there.

    Incidentally, when we killed the lich it made me aware of a problem that GMs face with these undying types of creatures, which is that the nature of their invulnerability is really hard to describe. Sure, we all know that you can’t kill a lich permanently, but of course if the players don’t have the phylactery they’ll do a damn good job of making the lich essentially eternally dead. Cut it into tiny pieces and feed them to fish in 7 oceans, burn the body and scatter it to the winds, etc. We were on a city that was on the back of a massive, slow-moving tortoise, and I recommended trying to find a way to feed the body into the tortoise – it wouldn’t be  dead but a thousand years of digestion would surely make the matter irrelevant. At this point the GM has to figure out a way to make the creature’s return plausible. Obviously it can be done but, particularly for things like trolls and the like that aren’t undead spirits in a temporary physical shell, it’s damn hard to explain. And in the case of liches, the Monster Manual seems to imply quite strongly that it’s their original body they’re in (the demi-lich is still tied to its body’s dust and skull) – so how exactly does it work for a lich?

    This is further proof that as one advances in levels in D&D, it’s important to put points into your various lore skills. If one hasn’t read Mordenkainen’s seminal texts on undead corporeality, how will one know the best way to build a lich-compass?

    fn1: Actually I’m pretty sure that there would be systems and/or GMs where a skill check would be required …

  • You kids get off my lawn!

    Over the Easter weekend my blog was the victim of a Jesus-jacking. A blog post I wrote a year ago for Easter, What Kind of Undead was Jesus?, suddenly started attracting huge numbers of hits – from 0 on April 4th to 126 on April 8th and then back down to three (so far) today. All these hits were due to web searches on terms like “Jesus lich,” “Jesus is a lich,” and “Jesus was a lich.” I think it was also posted on facebook, plurk and twitter. This year there seems to be a meme going around about how Jesus was a lich – “happy lich Jesus day” and a hashtag to go with it. Somewhere in that my post got picked up and rebroadcast a little, especially on Facebook. It even made it to that sump of New Atheism, Pharyngula.

    Anyway, I want to point out that I thought of this a year ago and I have the stats to prove it.

    On a tangential note, what is wrong with you people? Somehow over the holiest weekend in Christendom, approximately 300 people felt the need to do a google search on whether or not Jesus was a lich. Don’t you people have something better to do? Like, I don’t know, go to Church? Or play computer games? Or watch a movie? Or have sex? Or go away on an Easter break? I’m guessing that there are about 1 billion people living in nominally christian societies on this earth, which means that something like 1 person in every 4 million people feels the need to find out whether a figure who is a prophet in two of the world’s biggest religions was a lich by googling it on a public holiday. I’ve been away from christendom for a while now, but surely Easter hasn’t become so boring that you have to google Jesus’s undead status? Get a life people! Or at least keep your google searches for a working day …

  • Not someone you want to go bowling with…

    Twenty-five years ago today the Grim Reaper appeared on Australian television to warn us about the dangers of HIV. You can see the ad through this article about the anniversary. I was 14 at the time, and it was truly terrifying. I think it did its job, and scared Australians into sexual responsibility, though now that we have treatments and testing and the like, people may be beginning to become complacent again. Although it now seems a bit hammy, I think it also compares favourably with British health and safety adverts – it’s not as tacky, and makes its point much more succinctly and believably. I particularly like the nod to the holocaust when the narrator says it could kill more people than world war 2 – a nice touch, very understated but very effective.

    There was some controversy at the time, because some people interpreted it as likening gay men to the Grim Reaper (at that time it was largely a disease of gay men), but unlike in the USA there was a much better relationship between government, health workers and gay activists, and the controversy didn’t damage the ad’s effectiveness. Of course now people think that the kinds of things being said in this advert were hyperbolic or alarmist, because Australia has largely escaped the problem of HIV – another complaint made at the time was that this ad was overdoing it, and would contribute to that general suspicion people have that government health messages are just intended to scare us. But take one look at the situation in Africa and it’s clear that Australia dodged a very, very scary epidemic, and with our large drug-using population it was possible that HIV could have crossed to the heterosexual population by the early 1990s. It didn’t, and we can thank Australia’s early and very impressive response for our very lucky escape. Part of that response was this cute guy with his scythe and his slightly tatty cape, and we Australians should all be thankful for whatever small part he played in keeping us safe. So, thanks and … happy birthday Grim Reaper! If you get laid at your party, remember that prevention is the only cure we’ve got!

  • image

    Spring has come, and the big event of the early spring is the cherry blossom viewing festival, hanami. These are the arrangements at 9am this morning in Inokashira park, Tokyo. All the blue mats are a spot someone has reserved for their viewing party, and the people sitting on the mats are likely junior members of a company or club, who have been sent to guard a spot (basho tori) until evening. Partying under the flowers is a serious business, involving a lot of alcohol, possibly a grill or bbq, and a wide range of foods and snacks. If you look carefully you can see that the people on the mats have left their shoes at the edge: this is an important consideration in these parties. All the major parks in Tokyo will have arrangements like those pictured above, and many also lay out rubbish bins and entertainment (Inokashira park has signs up asking us to take our rubbish home, so is going for a “zero rubbish” approach to partying). The flowers we’ll be looking at are like this:

    image

    Typically one has several hanami: one with family, one with friends, one with any circle or club of which one is a member, and one with work. I am probably going to do four: two with my partner on Saturday (one in the park shown here); one with my university, on Monday, at which we will also welcome the new students; and one on Friday next week with my partner, when we head North to Ikaho to enjoy their famous cherry blossom scenes. If I had a role-playing group of longer standing, I would also be doing a hanami party with them, but we don’t seem to be planning any.

    Early April is a really nice time in Japan, because everything renews: spring has come, the cherry blossoms are out, new businesses open and all the “freshmen” enter companies and universities. It has a strong feeling of renewal and energy that’s very nice. The strong party atmosphere of the hanami season really serves to reinforce the feeling of new beginnings, and it’s a famous part of Japanese culture, reflecting a strong (and some say unique) Japanese sense of the impernanence of all things. Japanese life is characterized by seasonal festivals, and the main stages of a Japanese year are not marked by fixed historical or religious dates (like Christmas or Anzac Day) but by fluid, seasonal events: the cherry blossoms, the summer fireworks, the Autumn leaves, and finally the winter equinox. It’s at these times that one is reminded both of the strong role ritual continues to play in everyday Japanese life, and the continuing connection that Japanese culture has to paganism.

    It’s also at times like this that one gets outrageously drunk with random strangers, and eats too much. Wish me luck toasting the impernanence of my sobriety …

  • Heading for a fall?

    China’s rapid economic rise has been the topic of much debate over the past few years, and I think that this rise has some implications for western political economic theorists that are quite fun to explore. The orthodox view of China’s rise seems to be that it is going to continue to grow rapidly for a while to come, and that this growth is a serious threat to world stability. Of course a lot of the kind of thinking you read on China is just bog-standard journalistic stupidity, not worthy of much time and heavily influenced by that strange blend of insecurity and arrogance that seems to characterize cheap western journalists’ approach to Asia. A lot of it also looks like a very close copy of what was said about Japan both before World War 2 and again during Japan’s meteoric rise of the 70s and 80s. However, generally, no matter how poor the quality of journalism on Japan, foreign policy seems to have been much more level-headed, and China has been allowed to do its thing largely in peace since the 1970s. In response, China has changed radically over that time: it’s adopted many elements of the free market, turned its back on much of the Maoist principles that led to disasters like the Cultural Revolution, and has even come close to admitting and apologizing for Tienanmen Square (though it hasn’t). Also, most of the UN’s millenium goals have not been met, but those that have been met are largely due to China: it has made huge inroads into health and social problems that other developing economies have failed to dent, so something is going right in China. On the other hand, some people think that China is heading for a crash, and that this crash is going to be bad, based on bad fundamentals; this goes very much against the orthodoxy and is almost a heretical claim, but it is out there. Certainly China’s GDP growth is hard to believe from the perspective of most developed economies.

    I think these changes, and the way the world is beginning to reorient economically and politically around Asia, raise interesting questions for political economists in the West. I think that a lot of people are ignoring the possible theoretical challenges that China’s rise may pose for a variety of Western disciplines, and I want to consider them here. Let us suppose that China continues to liberalize politically without becoming democratic, and let us assume also that China follows the trajectory many people seem to believe it is capable of, and continues to develop without an economic crash – that is, it maintains an economy that has, essentially, the characteristics of a bubble without collapsing – suppose instead that it makes a soft landing, with the party putting the brakes on growth where necessary and slowing things down at the right time – this seems to be what many people believe will happen. I think this raises some challenging questions for market neo-liberals, marxists and possibly also Keynesians, that I’d like to consider here.

    1. Is market capitalism the best model?

    Modern Western political economics seems to have pretty much given up on any kind of economic system except market capitalism, but most economic theorists seem resigned to the existence of boom-and-bust cycles in capitalism: the challenge is not in getting rid of them, but in managing them. But every bust is a tragedy for a minority of the population, and creates (minor) political upheaval. Eliminating boom-and-bust would be a boon for capitalism, but despite the Gordon Brown’s infamous claim to the contrary, it doesn’t seem possible. So if China can develop over the next 10 years without experiencing such a catastrophe, then the Chinese may be able to claim to have developed a capitalist model free of busts; but their model, for all its capitalist points, is not market capitalism. Is managed capitalism a better capitalist model than market capitalism, and can it be achieved in a democracy? Of course, other Asian nations have shown similar economic models – Japan and Thailand spring to mind – but they eventually faced busts as they liberalized. If China avoids the bust (and there’s no guarantee it will) while maintaining greater than 5% annual growth over 2 decades, what does this tell us about the relative merits of managed vs. market capitalism? I think this possibility raises challenging questions for liberal economists and Keynesians alike.

    2. Are economic freedoms and political freedoms really intertwined?

    A common mantra of neo-liberal economists and market liberals generally is that economic and political freedoms are intricately intertwined; that you cannot genuinely have one without the other. In its most extreme form any form of government interference in markets must necessarily reduce political freedom too; in more reasonable forms, it’s not possible to advance to a proper level of political and social freedom if large portions of the population don’t have economic freedom. But this doesn’t happen in China: a society without fundamental political freedoms is developing a strong market economy, which (although I have no proof) I think is much more economically free than the classical liberal model would expect given the lack of political freedoms. Is the market liberalist model of the essential interconnection of these two freedoms fundamentally wrong? If so, under what conditions? I can think of a model of economic and political freedom in Australia which depends on strong, prescriptive social institutions (union membership and compulsory voting) that are quite unique in the developed world – and Australia also has a remarkable economic history over the past 30 years. Is some restriction on political freedom essential for achieving economic freedom?

    3. Was historical materialism completely wrong?

    As I understand it, historical materialism describes stages of economic development that societies pass through, and argues that transition to a new stage occurs through social and political upheaval. Typically, marxists believed that the communist revolution could only occur once society had developed through some “objective” standards, to the point of industrialization, and that the social and political upheaval that heralded the coming of the communist utopia would generally only be achieved when society contained a sufficient critical mass of politically conscious industrial workers. Generally, therefore, marxists preferred to be active in industrialized societies with strong unions and social democratic parties – places like the UK and (famously) Germany. But the most successful communist societies – Cuba and China – were underdeveloped relative to the historical materialist model, and their revolutions occurred through military action amongst peasants by a vanguard of (often foreign-educated) members of the elite, not the industrial working class. Communist China has existed since 1949, so in 2021 it will become the longest-lived communist nation on Earth (supplanting the USSR, 1917-1989); sooner if you factor in the period of instability in the USSR that followed the revolution (the equivalent period having occurred before 1949 in China). So unless something drastic happens in the next 10 years, it appears that historical materialism’s predictions were, are and will be thoroughly and utterly wrong. Not only that; while the USSR and the Eastern European communist states, founded in a strong industrial working class, were inflexible and oppressive, China and Cuba have shown themselves to be much better able to adapt to the flows of history, and have shown themselves capable of survival through pursuing political, economic and foreign policy reforms that were unthinkable to the founding nations of the communist ideal. Of course, it could just be that there are cultural influences at work – Cuba is far from the only South American country to have tried communism, and the rest (like Nicaragua) were very flexible in their interpretation of the tenets of Marxism; and Vietnam is another example of an Asian communist country that gave classical Marxism the flick very quickly. But historical materialism presents itself as some kind of fundamental theory. Whichever way you slice it, unless China really goes under in the next 5-10 years, I think Marxists need to accept that their view of history is completely stupid and wrong. And when they do, I’d like an apology to my Grandfather for the despicable actions of the USSR in the Spanish civil war – actions that were based in an application of historical materialism to a country that was very close to the Latin American and Asian exemplars of a society ready for a communist revolution.

    4. Is parliamentary democracy the only model of consultative government?

    I think that the Chinese one-party state is actually quite a consultative political system – through cadres and local party structures I think it gathers information on the needs and opinions of ordinary Chinese and adapts its policies accordingly. People don’t get to vote for their leaders, but I think there are ways in which the leadership is influenced by ordinary opinion. I think this is a crucial part of the process by which the country has been able to engage in near-continuous reform since 1970, without many significant internal upheavals. I also think that this is an important difference between China and the USSR, whose leaders acted like new Tzars. Furthermore, it is clear that the Chinese leadership listen and react to foreign opinion, though never (obviously) against their own interests. So I wonder if they have created a kind of consultative government that responds to public pressure without elections. If it were possible to quantify differences in political responsiveness, would the Chinese leadership be found to be significantly different in accountability to, say, Obama, Bush or Sarkozy? Especially on foreign policy issues, China has avoided some quagmires that the entire world was very clearly telling Bush and Blair they should avoid; but it has also implemented significant reforms in economic and social policy that one would not expect of a communist leadership. Is this a sign of careful listening? And if so, does this mean that consultative government can be achieved without elections – is it possible it could be more desirable? If not desirable as a whole, does it offer any lessons in public accountability and responsiveness that western democracies can learn from? Was, e.g., the Australian Labor Party a more responsive and consultative government under Hawke not because of his leadership but because of its strong system of local branches and union representation? Is the problem with modern political parties that they are poll-driven spin-monsters, or that they lack the grassroots membership necessary to maintain a level of consultative interaction with the community? And if so, are they still genuinely democratic, even though they maintain the semblance of democracy through elections? If democracy is reduced to just a shell-game of voting and polling, is it any better than a politically restrictive but socially consultative dictatorship? Is the only difference one of sustainability, in that a dictatorship can go pear-shaped after a change of leader, while a democracy can’t? And if so, how do we explain the continued smooth transitions of leadership in Chinese communism?

    5. Are democracies less militaristic than dictatorships?

    In my previous post on China’s military budget, I noted that China is actually a pretty good international citizen, with low levels of military spending and very few imperialist projects. In short, China doesn’t go to war easily. In the past 20 years it hasn’t gone to war at all, while the USA has gone to war at least four times – on one occasion “accidentally” lobbing a missile into a Chinese consulate, an act that China chose not to respond aggressively too. How is it that a one party state that is, let’s face it, militarily pretty impregnable even when it isn’t spending much, is so uninterested in military adventures? One idea that occurred to me with the anniversary of the Falklands War is that China doesn’t have any domestic democratic pressure to go to war. China manipulates militaristic sentiment domestically, some would argue quite cynically, but is perfectly capable of putting a lid on demands for war. On the other hand, democratic leaders can benefit significantly from military intervention, whether they seek it out (as Bush did in Gulf War 2) or it comes to them (as in the Falklands). They have a lot of incentives to manipulate jingoistic sentiment,  and I think recent events show that they are quite happy to do so when it suits them. Before world war 2, wars of colonial conquest were a given in Western political theory – the idea that you don’t invade some tinpot country when it suits you would have been quite alien to the way of thinking of most democrats in London or Washington or Paris. Perhaps for dictators war is much less likely to be a net positive politically than it is for democrats? But this idea doesn’t stand up by itself – dictators have a long history of stupid wars, and the worst wars of the last century only occurred after democracies slid into dictatorship. So what is the particular property of China’s one-party state that makes it so averse to wars of choice? Some cultural thing? Something about its particular political constitution? If so, is there a class of dictatorships – like China – that are much less likely to go to war than a modern democracy? Are the properties of this class fragile and easily changeable (so that, e.g., China could just suddenly flip into a military expansionist mode tomorrow), or does it have something to do with the aforementioned consultative style? Is it simply a function of China’s stage of development? Is there something about the sheer size and diversity of China that means the political class have to tread very carefully to avoid tearing the country apart?

    I don’t claim to have a view one way or the other on any of these questions, but I think they pose interesting challenges to the mainstream of western political economics as I perceive it through my (layman’s) perspective. If China successfully negotiates its development phase, and especially if it can resolve the Taiwan issue peacefully, then I think political economists are going to have to accept that their theories are challenged by the new models (and some of the older ones) springing up in Asia. I doubt we’ll see much change, but that doesn’t mean we can’t consider the possible ramifications of a peaceful, stable, economically and environmentally sustainable China, if such a beast emerges over the next 10-20 years. How will Western democratic and economic ideologies change in the Asian century?