Today I am celebrating my first publication in my new job, and since it’s about a topic I’ll probably be coming back to a lot in the next year, I thought I’d cover it here. It’s not much of a publication – just a letter in the journal Addiction – but it covers what I think is an interesting topic, and it shows some of the complexity of modern health policy analysis. The article, entitled Equity Considerations in the Calculation of Cost-Effectiveness in Substance Use Disorder Populations[1], can be found here[2]. It’s only 400 words, but I thought I’d give an explanation in more detail here, and explain what I’m trying to say in more detail. The background I’m presenting here may be useful for some future material I’m hoping to post up here. I’ll give a brief overview of the “cost effectiveness” concept, explain what the problem is that I’m addressing in this paper, and then give a (slightly) mathematical example in extremis to show where cost-effectiveness analysis might lead us. I’ll also add some final thoughts about cost-effectiveness analysis (CEA) in fantasy populations, with perhaps a final justification for genocide. Or at least an argument for why Elves should always consider it purely on cost-effectiveness grounds.
Cost-Effectiveness Analysis, QALYs and the IDU Weight
Traditional epidemiological analysis of interventions is pretty simple: cholera, for example, kills X people, so let’s prevent it. However, we run into problems when we have limited resources and need to compare two different interventions (e.g. turning off a pump vs. handing out disinfectant pills). In this situation we need to compare which intervention is more effective, and we do this by assessing the cost per life saved under each intervention – if turning off the pump is cheaper and saves more lives, then it’s better. This is usually represented mathematically as the ratio of the cost difference between the intervention and some control (the incremental cost) and the effect difference (the incremental effects). The ratio of the two is the incremental cost effectiveness ratio (ICER). This is what I used in assessing clerical interventions to prevent infant mortality. However, when we are dealing with chronic diseases the incremental effects become harder to measure, because a lot of interventions for chronic illness don’t actually save lives: they extend life, or they improve the quality of life a person experiences before they die. In this case we use Quality-Adjusted Life Years (QALYs). These are usually defined by conducting a study in which people are asked how they would weight a year of their life under some condition relative to fully healthy – or, more usually, relative to their health as it is now. For example, blindness in one eye might be rated a QALY of 0.9 relative to being fully-sighted. There is some interesting debate about whether these ratings should be assessed by those who have the condition or the community as a whole; the logic here can be perverse and complex and is best avoided[4].
So in essence, you rate one year of life as having the value of 1 when fully healthy, and then other states are rated lower. We can use the issue of Voluntary Testing and Counselling as an HIV intervention to see how this works.
Example: Voluntary Testing and Counselling
It’s fairly well-established that good post-test counselling can successfully reduce a person’s risk behavior, so if you can get people at high risk of HIV (e.g. men who have sex with men (MSM)) to undergo voluntary testing, you can catch their HIV disease at an early stage and get them to change their behavior. In theory, doing this fast enough and effectively will reduce the rate at which HIV spreads. Furthermore, catching HIV earlier means initiating treatment earlier (before it becomes symptomatic), and early treatment with anti-retroviral drugs leads to longer survival[5]. However, discovering one is HIV positive is not a pleasant experience and knowing you are HIV positive lowers your overall quality of life, even if the disease is asymptomatic. So if the survival benefits of early testing don’t outweigh the loss of utility, then it’s not worth it. So 10 years ago, when treatment extended your life by perhaps 10%, but testing reduced your remaining QALYs from 1 to 0.9, then the benefits might not outweigh the costs. Additionally, treatment is expensive, and it might be more cost effective on a population level to run health promotion campaigns that reduce risk behavior: reduced risk behavior means less infections, means less QALYs lost to HIV.
In essence, it’s a kind of rigorous implementation of the old bar room logic: sure I’d live longer if I didn’t drink, but why would I want to?
Recently, however, some analysts have introduced a sneaky new concept, in which they apply a weight to all QALY calculations involving injecting drug users (IDUs). The underlying logic for this is that IDU is a mental illness, and people with a mental illness have a lower utility than people without. This weight is applied to all QALY calculations: so a year of life as a “healthy” IDU is assigned a value of, e.g. 0.9, and all other HIV states (for example) are given a value of 0.9 times the equivalent values for a non-IDU.
What is Wrong with the IDU Weight
This has serious ramifications for cost-effectiveness and, as I observe in my article, fucks up any attempt to get a cost-effectiveness analysis past the British NICE, since it breaks their equity rule (for good reason). In addition to its fundamentally discriminative nature, it’s also technically a bit wonky, and in my opinion it clouds cost effectiveness analysis (“which treatment for disease X provides better value for money?”) with cost-benefit analysis (“who should we spend our money on?”). It’s cool to do the latter vs. the former, but to cloud them together implicitly is very dangerous.
Technical Wonkiness
Suppose you have a population of IDUs with a weight of 0.9, and you need to compare two interventions to prevent the spread of HIV. One possible intervention you could use is methadone maintenance treatment (MMT), which is very good at reducing the rate at which IDUs take injecting risks. You want to compare this with some other, broader-based intervention (e.g. voluntary testing and treatment, VTT, which also affects MSM and low-risk people). Then the average QALY for an MSM with asymptomatic HIV is about 0.9 (to pick a common value). Because you’ve applied the weight to IDUs but not to (e.g.) MSM, the average QALY for an IDU with asymptomatic HIV is 0.9*0.9=0.81. Now suppose that you implement MMT: this intervention reduces the risk of transmission of HIV, but it also treats IDU’s mental illness, so the weight for all the successfully-treated IDUs drops away and you gain 0.09 QALYs per IDU you treat; but then you gain 0.1 additional QALYs for every case of HIV prevented by the MMT intervention. This means that VTT has to be almost twice as effective as MMT to be considered cost effective, if they cost roughly the same amount. That is, in this case the cost-effectiveness of MMT is exaggerated relative to VTT by dint of your weighting decision – even though half of the benefits gained don’t actually have anything to do with reducing the spread of HIV (which implies you can prevent half as much HIV for the same QALY gains). On the other hand, if you implement an intervention that doesn’t treat IDU but does prevent HIV in IDU (such as needle exchange), its effectiveness will be under-estimated due to the IDU weight. In both cases, introducing the cost-benefit element to the analysis has confused your outcome.
Opening Pandora’s Box
The real problem with this IDU weight, though, is if we decided to extend the logic to all cost-effectiveness analysis where identifiable groups exist. For example, we could probably argue that very old people have lower QALYs than younger people, and any intervention which affects older people would gain less benefit than one which affects young people. An obvious example of this is anything to do with road accidents: consider, for example, mandatory eye testing vs. raising the minimum driving age. Both would result in lower rates of injury (and thus gain QALYs) but the former would primarily affect older people, and so would be assigned lower effectiveness, even if it prevented a hugely greater number of injuries[6]. When we start considering these issues, we find we’ve opened Pandora’s box, and particularly we’ve taken ourselves to a place that no modern health system is willing to contemplate: placing a lower value on the lives of the old, infirm, or mentally ill. As is often the case with social problems, the marginalized and desperate (in this case, IDUs) are the canaries in the coalmine for a bigger problem. I don’t think any health system is interested in going down the pathway of assigning utility weights to otherwise healthy old people (or MSM, or people with depression, or…)
An Example in Extremis
Let’s consider an obscene example of this situation. Suppose we apply a weight, let’s call it beta, to some group of recognizable people, who we call “the betamaxes.” Now imagine that these people are the “carriers” for a disease that doesn’t afflict them at all (i.e doesn’t change their quality of life) but on average reduces the quality of life of those who catch it to a value alpha. Suppose the following conditions (for mathematical simplicity):
- The people who catch the disease are on average the same age as the betamaxes (this assumption makes comparison of life years easier; breaking it simply applies some ratio effects to our calculation)
- The disease is chronic and incurable, so once a member of the population gets the disease their future quality of life is permanently reduced by a factor of alpha
- One betamax causes one case of disease in his or her life
- Preventing the disease is possible through health promotion efforts, but costs (e.g.) $10000 per disease prevented
- Betamaxes are easily identifiable, and identifying and killing a betamax costs $10000
I think we can all see where I’m going here. Basically, under these (rather preposterous) conditions, identifying and killing betamaxes is a more cost-effective option than the health promotion campaign whenever alpha>1-beta. Obviously permanent quarantine (i.e. institutionalization) could also be cost-effective.
This may seem like a preposterous example (it is), but there’s something cruel about these calculations that makes me think this weighting process is far from benign. Imagine, for example, the relative QALY weights of people with dementia and their carers; schizophrenia and the injuries caused by violence related to mental health problems; or paedophilia. I think this is exactly why health systems avoid applying such weights to old people or the mentally ill. So why apply them to IDUs?
Cost-Effectiveness Analysis in Fantasy Communities
There’s an obvious situation where this CEA process breaks down horribly: if you have to apply it to elves. Elves live forever, so theoretically every elf is worth an infinite amount of QALYs. This means that if a chronic disease is best cured by drinking a potion made of ground up human babies, it’s always cost-effective for elves to do it, no matter how concentrated the baby souls have to be. If a human being should ever kill an elf due to some mental health problem, then it’s entirely reasonable for the elven community to consider exterminating the entire human community just in case[7]. Conversely, any comparison of medical interventions for chronic disease amongst elves on cost-effectiveness grounds is impossible, because all treatments will ultimately produce an infinite gain in QALYs: this means that spending the entire community’s money on preventing a single case of HIV has an incremental cost effectiveness of 0 (it costs a shitload of money, but saves an infinite number of QALYs). But so does spending the entire community’s money to prevent a single case of diabetes. How to compare?
Similar mathematical problems arise for Dwarves, who have very long lives: you’d have to give them a weight of 0.25 (for being beardy bastards) or less to avoid the same problems vis a vis the use of humans in medicinal treatment that arise with elves.
This might explain why these communities have never gone for post-scarcity fantasy. When you have an infinite lifespan, no intervention of any kind to improve quality of life is cost-effective. You might as well just live in squalor and ignorance, because doing anything about it is a complete waste of money.
Cost Effectiveness Analysis as a Justification for Goblin Genocide
Furthermore, we can probably build a mathematical model of QALYs in an AD&D world: some people have better stats than others, so they probably have better quality of life. We could construct a function in terms of the 6 primary stats, and obviously goblins come out of this equation looking pretty, ah, heavily downward weighted. Given that they lead short and brutish lives, and are prone to kill humans when the two communities interact, the obvious effect of weighting their QALYs from this mathematical model is pretty simple: kill the fuckers. The QALY gains from this (and the low cost, given the ready availability and cheap rates of modern adventurers) makes it a guaranteed investment. In fact, compared to spending money paying clerics to prevent infant mortality, it could even be cost-effective.
Conclusion
Cost-effectiveness analysis needs to be applied very carefully to avoid producing perverse outcomes, and the logical consequences of applying weights to particular groups on the basis of their health state are not pretty. We should never weight people “objectively” to reflect their poor health in dimensions other than that under direct consideration in the cost-effectiveness analysis, in order to avoid the risk of applying a cost-benefit analysis to a cost-effectiveness situation. Furthermore, even if we are comfortable with a “discriminatory” weight, of the “oh come on! they’re just junkies!” sort, it can still have perverse outcomes, leading to over-estimates of the cost-effectiveness of treatments for the mental illness compared to other interventions. Furthermore, we should never ever ever allow this concept to become popular amongst elven scholars.
I’ll be coming back to this topic over the next few months, I think, in a way I hope is quite entertaining for my reader(s). Stay tuned…
—
fn1: The slightly cumbersome title arose because the journal now doesn’t like to refer to “substance abuse” or “substance abusing populations” so I had to change it to the un-verbable “Substance Use Disorder”
fn2: If you download the pdf version it comes with a corker of a letter about French tobacco control policy[3]
fn3: Which is a contradiction in terms, surely?
fn4: For a full explanation of this and other matters you can refer to the famous text by Drummond, which is surprisingly accessible
fn5: In fact we are now looking at very long survival times for HIV – up towards 30 years, I think – provided that we initiate good quality treatment early, and so it is no longer necessarily a death sentence, if one assumes a cure will be available within the next 30 years
fn6: This applies even if you ignore deaths and focus only on short-term minor injuries, and thus avoid the implicit bias in comparing old people with young people (interventions that save life-years in old people will always be less “effective” than those that save life years in young people, unless the effect of the intervention is very short-lived, because old people have less years of life to save).
fn7: In fact you can go further than this. All you need is for an elven propagandist to argue that there is a non-zero probability that a single crazy human will kill a single elf at any point in the future, and the expected value of QALYs lost will always be greater than the QALY cost of killing all humans on earth, no matter how small the probability that the human would do this
October 12, 2011 at 10:28 am
An interesting post. Some thoughts on it are:
1. Your point that the IDU multiplier ” clouds cost effectiveness analysis (“which treatment for disease X provides better value for money?”) with cost-benefit analysis (“who should we spend our money on?”). It’s cool to do the latter vs. the former, but to cloud them together implicitly is very dangerous.” Is an interesting one. I’d have to query why IDUs get a multiplier but other mental illnesses or conditions don’t.
Even if we accept that IDUs are more likely to be catching HIV and nicking your TiVo simultaneously, they shouldn’t get a weighting unless the principle is extended to other groups such as imprisoned criminals. Of course that leads to weightings like “People in prison who are intended to die there should be assigned a multiplier of 0”. While logical it does suggest that Josef Fritzl shouldn’t be given so much as a tissue.
Of course, I also don’t want to say that I’m fundamentally opposed to the concept of allowing a cost benefit analysis to be factored into this, either explicitly (my preference) or implicitly (as per the IDU weighting). The reason for that is such weighting are already built into our society all over the place.
2. Under Tolkien’s mythology Elves live forever, but they are also physically reincarnated in the event of their death. As such, anything you do to an elf can’t impact their QALYs – they’re always going to live forever even after you stub them and take all their stuff.
Of course, in the same mythology, killing a human gets them an infinite span in Heaven which we can assume to have a QALY > 1.0 per year. So humans should probably be forced to pay their killers.
By contrast in most D&D settings, an elf has a long, but non-infinite lifespan [1]. As such they can perform a cost benefit analysis. And if you check your setting background there are at least a couple where the elves have decided that death or slavery were the correct options for other races. Sadly purely mathematical assessments of such things tend to be poor predictors.
[1] See various player handbooks and Elven expansion books.
October 12, 2011 at 11:58 am
I should make it clear that I don’t think that the people incorporating this weight are intending to discriminate against IDUs; I just think this is a likely consequence of the weighting. I think the reason that it happens to IDUs is because IDU is probably the only form of “mental illness” that is directly associated with transmission of a serious infectious disease. Stepping back from the cost-effectiveness analysis issue, this suggests a flaw of the medical model of drug use. This has been very effective in turning our approach to drug use from a moral to a medical one, but it appears that it has its limits when applied to cost effectiveness: I don’t think many people genuinely believe that IDU is an “illness” the same way that, e.g. schizophrenia is. Another example of the effect of widely applying the medical model is the decision by an Australian court a while back to allow IDUs to get sickness benefits while on methadone treatment, because they are mentally ill and thus not able to work. MMT is intended to help IDUs get back to work (it wouldn’t be a very effective “treatment” if it didn’t) so this is kind of self-defeating. [To be fair, I’m not sure how widespread the results of that court case were or how relevant it is to modern Australian welfare policy, or even if I’m interpreting it properly, since I’m going entirely on memory].
The prisoner issue is a very interesting one: on the face of it it seems like a fair time to apply the weight, since society as a whole has decided to reduce the utility of these people. But I think I can come up with an example of a perverse outcome of applying such a weight to prisoners, too. Suppose that prisoners are applied a utility weight β, to represent their lower quality of life. So a healthy prisoner spending one year in prison has a QALY value of β whereas a member of the community has a value of 1. Now suppose that prison is acting as an incubator for some disease (TB is a good example). For this perverse outcome to happen, we have to suppose that prison forms a kind of high-risk group, so the presence of a community of prisoners leads to a higher rate of TB in the community – this effect isn’t guaranteed but it’s a well-accepted consequence of the existence of high risk groups and I think it is a problem in e.g. Malawi. Assign TB a QALY value of e.g. 0.9. So a person living with TB in the community has a QALY of 0.9, while in prison they have a value of 0.9 x β. Then, given certain balances of rates of TB in the community in the absence of a prison community compared to its presence, one very simple and cost-effective way of gaining QALYs could be to free all the prisoners. This would arise because by freeing the prisoners we not only reduce the rate of TB in the community (by breaking the transmissive properties of the high-risk community) but we gain additional QALYs by liberating the prisoners (we gain an amount 1-β). It’s also possible that this perverse outcome occurs most severely in the communities that value prisoners lowest: where β is very low, it’s likely that the prisons are overcrowded and there is a lot of corruption and misuse of prisoners (increasing their connection with the community) and high turnover (due to e.g. imprisoning minority groups for piss-poor reasons[1]). So it’s likely that the largest impact of the prison community on TB in the non-prison community will occur when their utility weight is lowest, and the most is to be gained in TB-related QALYs by freeing them; but this effect will be exaggerated by the utility gain from freeing the prisoners. If on the other hand we were assigning them a QALY value of 1, then we assess the intervention “free the prisoners” on the same footing as the intervention “improve post-release health care for prisoners” and on the same footing as the intervention “improve testing and treatment of TB in the community.” If we don’t compare them on equal terms, we might find the intervention “free the prisoners” is more cost-effective than “improve testing and treatment of TB in the community” even though it prevents less cases of TB, and doesn’t lower the overall burden of disease due to TB as much as “improve testing and treatment.” To me this is a perverse outcome.
This mathematical wonkiness is why I think we should do all our cost-benefit calculations explicitly and separately to our cost-effectiveness analyses. Furthermore, if a government wants to start saying that healthcare resources should be reserved only for the already-healthy or the non-criminal, I want them to say it openly. In fact our government makes it very clear through its national drug policies that the opposite applies: that it is willing to put extra resources into health care for the most marginalized members of our community. Our government explicitly endorses the “Health for All” principles of the WHO, and if the community want to change this then an explicit and open debate has to happen. I like things done that way.
Your point about Tolkien’s mythology is interesting. That means actually that the shoe is on the other foot, and it’s impossible for humans to do cost effectiveness analysis of elf-genocide that doesn’t come up in the plus column. This is because killing all elves has a net loss of 0 QALYs, but at a gain of some number of QALYs due to elves killing/enslaving humans. The only issue then is whether killing them is cheaper than setting very strong immigration and border control policies. Also, elves should be very scared if they’re delicious: even if the QALY gain for a human from eating elf meat is 0.00000000001, it’s still a cost-effective way to improve utility, because there’s no net loss to the elf (provided they’re killed humanely).
Do humans go to heaven in Tolkien’s mythology? Certainly, if humans in our world genuinely believed that death took them to a better case, under a standard cost-effectiveness analysis we ought to consider killing them as martyrs as a net benefit to the community.
It’s these kinds of heartless calculations that give elven economists a bad name, and will prevent them from ever winning a Nobel Peace Prize.
—
fn1: other reasons for high turnover are also possible, of course, and corruption is another good one
October 13, 2011 at 4:16 pm
”I think the reason that it happens to IDUs is because IDU is probably the only form of “mental illness” that is directly associated with transmission of a serious infectious disease.”
I disagree. I recall hearing that cannibalism is linked to prion related diseases. Unless you’re implying that cannibalism is a lifestyle choice instead of mental failing? [1]
”I don’t think many people genuinely believe that IDU is an “illness” the same way that, e.g. schizophrenia is.”
I agree totally. I suspect the general community attitude would be that IDU is a (highly negative) personal choice. Interestingly, this means that the Daily Mail readers are more likely to support your proposition than Guardian readers.
I’m also curious if they apply discount factors to other mental illnesses, especially ones that are controllable. It’s one thing to say your QALY is lower if you suffer from rampant insanity that causes you to see little green men everywhere, but another to say that epilepsy lowers you QALY. And If IDU is the only one they pick out then it’s not really adjusting for IDU as a mental illness, is it? It’s adjusting for IDU.
“To be fair, I’m not sure how widespread the results of that court case were or how relevant it is to modern Australian welfare policy, or even if I’m interpreting it properly, since I’m going entirely on memory”
Regardless, I promise to state it as a fact next time I’m speaking to a party conference.
”in the communities that value prisoners lowest… high turnover (due to e.g. imprisoning minority groups for piss-poor reasons)”
I’m pretty sure that the jurisdictions that value prisoners the lowest tend not to worry about high turnover. Burma’s recent release of political prisoners notwithstanding.
”To me this is a perverse outcome.”
The thing I find funniest is that while our argument makes perfect sense, it actually has the effect of removing a bias in favour of removing health disadvantaged group members from the health disadvantaged group. It’s another case where the Daily Mail reader is going to be more in favour of the outcome than the Guardian reader… Though of course your other comments (i.e. WHO reference) mitigate against this if implemented too.
”Do humans go to heaven in Tolkien’s mythology?”
Their special gift from Eru Ilúvatar (God) is they die and pass beyond the bounds of the world, presumably to hang out with Eru – though possibly to toil as slaves in his angel hooker factory. If someone gave me death as a special gift I’d expect the follow up to be a total dud too.
”we ought to consider killing them as martyrs as a net benefit to the community”
This was one of my points in our post about cleric based reductions in child mortality. If acting in a clever manner will increase our lifespan and get you into heaven, but acting in a stupid manner will decrease our lifespan and get you into heaven then stupid is best of utility maximisation.
[1] Come on. Admit it. You’re pro-cannibalism decriminalization! It’s only a tiny step from that to condoning the coming Zombie-pocalypse.
October 15, 2011 at 4:28 pm
Actually, cannibalism and prion disease is kind of interesting from a disease modeling point of view… I might have to do a post on that. But even if we accept that cannibalism is caused by mental illness, prion diseases aren’t infectious to non-cannibals. That is, in my cannibal-loving zombie-apocalypse-condoning utopia, even if your boyfriend sups on the fresh brains of his enemies every night, you can’t get mad cannibal disease from him by any other means except eating him, so the disease won’t spread into the non-cannibal community. Whereas with IDUs, the disease can spread (through sex) to the non-IDU community, and in some circumstances (e.g. China now) the IDU community plays a significant role in driving the epidemic. So it’s one of the few “mental illnesses” where the role of the “illness” is relevant in driving an epidemic of infectious disease in people who don’t have the mental “illness.” I guess that if other mental illnesses had this property, then they would also be subject to this mistaken application of a “mental illness weight.” Essentially what I’m trying to get at with this paragraph is that IDUs haven’t been singled out for this form of “discrimination” because of their mental illness, but because it happens to be associated with an infectious disease that needs to be prevented anyway (But I think you knew this is what I was getting at).
I’m also curious if they apply discount factors to other mental illnesses, especially ones that are controllable.
Interesting that you ask that, because one of the papers I referenced in the article gives a reference in support of its assignment of an IDU weight, but when you track down the reference you find very few attempts at describing a QALY weight for other forms of mental illness. I think this is because it’s very hard to do a well-designed study asking the mentally ill to rate their own loss of utility. I should add here as well, just for clarity, that it’s completely correct to assign mental illness a lower utility weight if you’re researching the cost-effectiveness of a treatment for that mental illness (e.g.an anti-depressant). It’s when you weight the ill people at a lower utility in studies that aren’t about treatment for the mental illness that you build an implicit discrimination into your models.
I’m pretty sure that the jurisdictions that value prisoners the lowest tend not to worry about high turnover
I’m thinking of the USA here: it has a very poor record on incarceration. Also Australia’s record on incarceration of Aboriginal people is very much like this: high turnover, but limited respect for the rights of prisoners. I think in the US there is some horrifying statistic about 30% of young black men having been to prison at least once, and that’s got to represent high turnover (e.g. for petty drug offences). I’m assuming that countries with this sort of attitude towards imprisonment also don’t care much about the quality of their prisons. By contrast, Australia has a lot of systems in place to prevent people being imprisoned for petty drug offences, and also by international standards has quite a humane prison system.
I’m actually working on a model of prison health at the moment, and I’m explicitly avoiding weighting prisoners’ lives with a lower utility value, because I don’t want to have any perverse results thrown up by it.
The thing I find funniest is that while our argument makes perfect sense, it actually has the effect of removing a bias in favour of removing health disadvantaged group members from the health disadvantaged group
Which direction the perverse outcome works in depends on what you’re modeling. If, for example, I assess the cost-effectiveness of voluntary testing and treatment vs. needle/syringe programs, I’ll find the latter is less cost-effective than it should be because it targets IDUs and they have a lower QALY weight. So then I might reject this tactic even if it has some effect on prevalence in the non-IDU community, or even if ultimately it prevents more infections.
If someone gave me death as a special gift I’d expect the follow up to be a total dud too.
This, sir, is an excellent point.
October 22, 2011 at 9:15 pm
[…] looking at the cost-effectiveness of health interventions in fantasy communities we have shown that the infinite lifespan of elves creates analytical problems, and other commenters have suggested that the cost-effectiveness of clerical interventions to […]