Here is another post about how to evaluate the validity of a medical claim. My last one dealt with case reports as medical evidence. This discussion is about the next rung on the ladder of reliability of medical evidence — the uncontrolled trial.
Researchers can do other things to series of cases besides simply describe what the patients are like; they can manipulate the situation in various ways. For example, if a doctor looks at her series of patients and becomes convinced that a particular therapy will work for the disease, she can give the therapy to the next patient, or series of patients, that come her way with the problem, and see what happens. This would constitute one version of an uncontrolled trial, and is probably the oldest kind of treatment research doctors have used. Venerable as the technique is, it is easy to see how an experiment like this could yield misleading results.
First, the patient group is subject to the same selection bias of the case series — the assortment of people with the problem who come to see the doctor are unlikely to represent a random sample of all people with the disease. Next, the only way the doctor could decide the treatment might be helping would be to compare what happens in the patients who get the new therapy with the patients she saw in the past who did not. Such so-called historical controls are the weakest sort of control group. This is because they are subject to the same kind of selection bias as the experimental group, those who get the treatment. Worse than that, since they were seeing the doctor at an earlier time, they may not even be representative of the patients with the disease who are seeing her now. Finally, a doctor who believes that a particular treatment will work (which is, after all, why she is doing the experimental trial in the first place) is hardly the best person to decide impartially if that is so. All of us want our theories to be correct, so her evaluation is bound to be slanted — it is only human nature at work.
Uncontrolled trials like this are particularly susceptible to what logicians call the fallacy of post hoc, ergo propter hoc, translated from the Latin as “after this, therefore because of this.” Anyone who has watched late-night cable television has seen countless examples of this logical trap, in the form of personal testimonials from people who had this or that problem, took the pill or bought the product, and the problem went away. The fallacy, of course, is that the two events may be entirely unrelated, just as the fact I may drink coffee every morning before the sun comes up does not cause the heavens to move in that way.
Trials like this are also highly prone to suffer from the placebo effect, the trick the human mind plays on us to believe so much that a particular treatment is working we actually will it to happen. The power of wishful thinking in the human mind is astonishing. Even more astonishing is that, in some situations, the “useless” placebo, a sugar pill or its equivalent, actually does improve the situation, if only slightly (15-30% by most estimates). So oftentimes people get a little better no matter what the therapy. The only way we can be sure the improvement is from the therapy (as a therapy, and not a placebo) is to blind both the patient and the observer to knowing which patients got the treatment and which ones got the placebo.
A new law went into effect in South Dakota this past July that represents a serious infringement on the right of patients and their physicians to do what they think is best for the patient. The law concerns abortion, an incendiary topic. I’m not writing here to debate the rightness or wrongness of abortion; for one thing, in my experience such a discussion never changes anyone’s mind and often becomes vitriolic. Rather, I’m writing to bring attention to what I see as a precedent dangerous to good medical care.
The background to this law is that abortion opponents, stymied by several Supreme Court decisions, most notably Roe v. Wade, have over the years tried various roundabout techniques on the state level to limit severely or even prevent abortion. These measures take the form of making it more difficult for a woman to obtain an abortion. The South Dakota law follows this pattern.
The law mandates that the physician hand a woman seeking an abortion a lengthy written document. The precise language of this document is decreed by the state legislature, and the woman must sign each page of it. However you feel about abortion, the key problem here is that a legislative body is, in effect, practicing medicine. The language of the statement is not medical, it is political. Any reasonable person realizes that the principal intent of the law is to reduce (or even eliminate) the number of abortions in South Dakota.
My concern is that, if this practice is allowed, there is nothing to stop legislatures (or Congress) from passing similar laws about any medical situation they wish. From my perspective as a pediatric intensivist, I’m most concerned about intrusion into the wrenching end-of-life discussions all intensivists from time to time have with families, but in theory such laws could be about anything. Several constitutional scholars note that the law probably violates physicians’ first amendment rights to free speech.
You can read the precise language contained in the statement, as well as more about the controversy here.
Here is another post about using medical evidence. My last one dealt with the usefulness (or not) of expert opinion as a basis for evaluating the validity of a medical claim. This post discusses the next highest item on the hierarchy of medical evidence — the case report.
The simple case description is the oldest tradition in medical research, dating back hundreds of years, long before biostatistics had even been thought of. The medical journals of the nineteenth and early twentieth centuries are filled with doctors’ descriptions of individual patients with particular diseases — how the problem began, how it evolved in the patient, and what happened to the patient. The writer usually then speculated about what the particular case taught us about the malady. He would usually compare his findings to whatever previous authors had written about the disease.
We still use this technique today, although such case reports are very weak tools for deciding anything important about diseases. Any nonphysician can see the pitfalls in this sort of medical evidence. It is totally dependent upon what happens to turn up in the doctor’s individual practice, and there is no way to tell if a particular case is representative of all, or even most, instances of the disorder. It is merely an anecdote of one occurrence.
Slightly better than a case report is a collection of case reports, or case series. There is also a very long tradition of these in medical writing, and a very large series does tell us things that are increasingly likely to be true as the series gets larger. But there remains a huge problem in using case series to understand whatever the problem is: they are not a random sample of all cases; rather, they were selected in ways that easily introduce bias. For example, a particular doctor may become known as an expert in a particular disease and so people with the problem come to see him. Or perhaps only patients with more severe, unusual forms find their way to him because the milder varieties are cared for by someone else or are never seen by a doctor at all.
One can imagine a large number of other circumstances that would introduce bias into whatever the doctor writing the case series would see. These issues make case reports and case series only slightly better than expert opinion as a tool for understanding medical causation and treatment. These pitfalls well illustrate a common saying among medical scientists: “The plural of anecdote is not data.”
The difficulties with isolated case reports are not unique to medicine. Consider something like an internet message board, most of which are devoted to specific topics. People posting on the board share their experiences and opinions about many things, such as particular products that work or do not work. Vigorous debate often ensues, some of it supported by evidence of some kind, some of it not. But the effect is that of a series of case reports, and readers of the messages have no idea how representative is the experience of an individual poster.
Case reports and series are an example of what is called selection bias — the possibility, or even the probability, that the individual cases have not been selected randomly from the total group they come from, which is all people who have the disease. They are therefore not representative of the entire group, and using them to draw any conclusions about the whole group is suspect. Consider again the example of the internet message board. If thousands of posters complain about a particular product, the odds get progressively higher there is some problem with it. But people with a complaint are probably more likely to post than are those satisfied with the product, so there is built-in selection bias. Additionally, the sample will include only people with internet access and familiarity with message boards. Only a survey of everybody who bought the product, or a properly randomized sample of them, can answer the question.
One thing everyone agrees on is that our economy cannot sustain the amount of money we spend on healthcare. That was clear even before the Wall Street meltdown, and it’s even more true now. The United States spends 16% its gross domestic product (GDP) on healthcare, a figure half again that of the next highest spending country (Switzerland, at 11%). Most other European countries spend 9-10%. We also spend much more per capita than anybody else. (These figures are from 2003, but little has changed since then — if anything, it’s worse.)
Not only do we spend more than anybody else, but by many measures, as a society we get much less for our money. That is, in spite of our high healthcare bills, the United States does not compare well with other countries in many measures of health.
What’s to be done? How can we find ways to spend less on healthcare but get better value for our dollar? One answer is that much of the money we spend is on unproven or even worthless treatments. Many authorities advocate we establish an independent agency of some kind to evaluate which treatments work and which ones don’t. Britain already has such an agency, called the National Institute for Health and Clinical Excellence. The way it works is that patients and physicians are free to use a non-approved treatment, but insurance won’t pay for it. As you can read in the linked article, there are some vocal opponents to the agency, something probably inevitable. But in the words of its director: “We are not trying to be unkind or cruel. We are trying to look after everybody.”
Opponents of such a concept complain this represents rationing of health care, because inevitably it would mean that patients won’t get all they may want. This is true, but in fact we’re already rationing care; we just use a more insidious method. Anyway, what else are we to do? There is simply not enough money, especially now, to pay for everything. It’s time we recognize that. An independent evaluation agency of this sort would make the decision-making process transparent and fair.
Here is another post taken from my newest book, How to Talk to Your Child’s Doctor. This one concerns what I call the egotist.
Egotism is a common trait among doctors, although most of us keep it under adequate control when dealing with patients. Throughout this book, the ideal doctor-parent encounter has been described as a collaboration among equals, each of which brings expertise to the exchange; the doctor knows medicine, the parent knows the child. This is the ideal, although sometimes the reality falls short of it. The way our medical system is now structured gives more power and influence to the doctor side of the relationship than the patient side. As you read in chapter 7, things were not always this way; a century ago a surplus of doctors with treatments of doubtful usefulness scrambled to attract patients. These days, however, physicians have many more therapies that actually work, plus the benefit of an enormous medical establishment behind them. So now doctors are usually the ones deciding who gets what treatment. In spite of that fact, good, experienced doctors will do their best to use their power over patients lightly, always inviting parents and patients to share in the authority.
Physician egotism can get in the way of good communication in several ways. A simple manifestation is the tug-of-war over whose time is more valuable, the doctor’s or the parents’. A good example of this conflict is the doctor who schedules far more patient appointments than he can accommodate in a day, then seems unaware of how keeping a parent waiting for hours can poison the atmosphere even before the evaluation has even begun. Parents usually understand long waits when they take their child to the doctor for an unanticipated acute problem. If the waiting room is full of children just like theirs, there is little the doctor can do except see them each in turn. But the subspecialist who packs his waiting room with too many scheduled patients is proclaiming, in effect, that his time is far more valuable than that of parents, who often must take off a full day’s work to bring their child to see him.
The egotistical doctor is one who tends to forget that the patient is the center of everything, the reason the parents are there in the first place. He forgets that the encounter is about the child, not the doctor. This attitude can show itself in a persistent tendency to turn the subject of the conversation away from the child and toward the doctor. The result may be harmless, as when a garrulous doctor is genuinely trying to relax the parents and their child with a friendly conversation about other things, or it may be more toxic, as when a doctor constantly talks about himself and what he does. The latter can be particularly trying to parents who have waited a long time to see the doctor, only to find their brief time with him taken up by extraneous chatter.
Although it can be annoying to parents, excessive egotism in your child’s doctor is generally a minor issue in the big picture of getting your child the evaluation she needs. I say this because, although there are exceptions to everything, for the large majority of doctors I have met who are more egotistical than the average, their self-centeredness does not get in the way of their medical skills. In fact, some subspecialties, such as high-risk surgery, almost require the physician to have a huge ego if he is to perform such surgeries effectively .
So it is largely a matter of the personal taste of the parents. If you find yourself irritated when talking with an excessively egotistical doctor, and if you think this is interfering with his proper evaluation of your child, the best thing to do is to be persistent in turning the conversation back to your child at every opportunity. Of course, if you are really irritated by his manner or the way he treats you, do your best not to see him again.
Most doctors are paid by some version of how they have always been paid–what is loosely called “fee for service.” The notion is simple: the doctor gets paid for each encounter with a patient, whether this is an office visit or a major surgical operation. Thus the more the doctor does, the more he gets paid. It seems fair. After all, shouldn’t we get paid for the work we do?
The fee-for-service model, however, has long had critics. The most fundamental objection is that it rewards doctors for doing things. In an environment like that, one in which the more one does the more he gets paid, the doctor is tempted to do things that don’t need to be done, or choose the thing to do for which he gets paid more rather than the thing that pays less. In the ideal situation, of course, the doctor won’t consider those things–rather, he will put the patient’s best interests first. But doctors are human, and fee-for-service presents a doctor with a fundamental conflict of interest.
The alternative to fee-for-service is some sort of payment system in which the doctor is paid a salary that is the same no matter how much he does. This is currently done by some large practices, including the Mayo Clinic. Critics of this salaried system believe that, without financial incentives, doctors will simply not work as hard. Other physician groups blend together the two approaches, with a salary supplemented by some sort of bonus for doctors who do more.
The debate is more than a financial one–it is also a philosophical one. Most free market enthusiasts strongly support fee-for-service; those who favor tighter regulation of medical care, often including one of various single-payer models, are typically open to salary-based payment models. Salary-model systems also are frequently used by Health Maintenance Organization (HMO) systems.
Historically, organized medicine has been strongly opposed to paying physicians by any other method than fee-for-service. Seventy-five years ago, physicians who accepted salaried arrangements were ostracized by their peers and sometimes even penalized. This view has changed to some extent, but I think it is still fair to say that most American physicians favor traditional fee-for-service.
For myself, I favor a salary model (or salary plus a modest incentive for extra productivity) for what I do. I work in the field of intensive care, which lends itself well to this. Other specialties are somewhat different. My job, in effect, is to be like a firefighter waiting in the firehouse–if the PICU is busy, I work harder; if there are less patients, I can ease up. After all, we pay firefighters whether or not they are fighting a fire.
I see no reason why physician payment strategies must be all one thing or all another. It seems to me that whatever evolves from our current chaotic situation could find a place for both approaches. Hard work should be rewarded. However, and this is a big however, we need to understand the inherent conflict of interest of traditional fee-for-service medicine. Also, not all rewards for hard work need be financial ones.
This is another post about how non-physicians can understand how physicians use evidence. As I noted before, medical evidence has a hierarchy of reliability. The least reliable of these is expert opinion. This seems counter-intuitive: why is expert opinion the worst sort of evidence? Should not the experts know what they are talking about? In general, of course, experts do know what they are talking about — that is what makes them experts. But a closer examination of the matter shows why this kind of evidence is the weakest and most subjective; after all, it is one person’s opinion (or sometimes a committee of persons’ opinions), and opinions can be incorrect.
We need to look closely at just why a particular individual is considered an expert, and by whom. Credentials are important: where did the person obtain her training, where does she work now, and what is her standing among her peers? These seem obvious questions to ask, but these days an astonishing number of people with dubious or no credentials can write a book, put up a website, or start a blog, and, if they are persuasive marketers, can convince others they are experts. Fortunately for parents, the same wide-open quality of the internet allows one to search the background, credentials, and accomplishments of any putative expert. Wise parents will do this as a matter of course before deciding whether to take the expert’s advice on any important matter about their child’s health.
Experts who advise you to do one thing or another with your child typically base their advice to you on their own interpretation of the available medical research. They have the knowledge and training to understand the often esoteric medical literature. In addition, most experts themselves do research in the relevant field. Those are important and useful things. However, there are still good reasons why we should regard such opinions as the worst kind of data — better than nothing, but sometimes only barely so.
For one thing, the reason an expert holds a particular opinion may be because she was taught that way by her teachers, who may have, in turn, been taught the same thing by their teachers. Medicine is practical and empirical enough that such received, traditional opinions will not be tossed out unless they are consistently wrong. It is also true we physicians venerate our medical forebears to the extent that misguided opinions can occasionally persist long after they should have been discarded. So sometimes the answer to why doctors do something a particular way is that we have always done it that way.
Experts also form their opinions based upon what they have seen in the past. If their experience is long, often they have seen quite a few instances of whatever is under discussion, and that experience should count for something. On the other hand, memory is a tricky thing; sometimes we recall things in ways that can ultimately prove misleading. For example, the more striking and dramatic things tend to stay in our minds better than the more mundane things, and medical experts are not immune to this phenomenon. For example, I know that I remember unusual manifestations of certain cases for decades, and this inevitably colors how I approach the next child with that particular problem. Even though I know the case was unusual, I naturally think of its circumstances whenever I care for another child with whatever the disorder was. This is an example of what we call recall bias.
There are other kinds of bias that may affect the judgment of medical experts, and some of them are not innocent things like tricks of memory. Medical experts are no different from other kinds of experts, such as foreign policy pundits or stock market analysts, in that we, too, may have agendas that are not obvious to parents listening to our advice. Any controversial subject will lead to partisanship, and medical debates are no different. Parents considering the advice of medical experts should be alert to what a particular expert’s agenda might be. This is not necessarily a sinister thing; I think the great majority of experts advocating one position or another do so because they truly believe it is the correct one. But it is still a real thing.
Sometimes, however, medical experts may have agendas that are not so innocent. For example, there have been recent examples of experts touting one treatment over another when they have an undisclosed interest in the outcome. The conflict of interest could be intellectual, such as past friendships or associations with researchers of a particular treatment, or they could be crassly commercial. The ethical boundaries are, in theory, quite sharp, but there have been well-known examples of medical experts recommending a particular treatment when they have a financial conflict of interest regarding the outcome.
The best experts, of course, are the ones who tell you the basis of their opinions. They do not just pontificate about what is best for your child — they tell you how their reading of the best evidence led them to their informed opinion. They interpret for you the meaning of the scientific evidence. On the whole, most medical experts will do this for parents. Even so, it is a good thing for parents not to take whatever the experts say entirely on faith; it is better to have some grasp of how medical evidence is collected and analyzed. And besides, even experts can be wrong.
This is the first of several posts about how to read the medical news. How can a non-physician interpret the latest breathless bulletin about some new breakthrough? Are there commonsensical tests one can apply to the story to see how valid it might be? There are. One useful notion is that we have ways of grading the validity of different sorts of evidence. I’ll get to those in later posts. First, it is useful to know where we’ve come from.
Medicine is not totally a science. It makes use of science and struggles as best it can to adhere to scientific principles, but medical knowledge, perhaps because it involves humans, will always contain an element of human fallibility. This is more true of some aspects of medicine than others. For example, we understand the science of how a damaged bone heals far better than we understand the science of how a damaged mind does. Medicine is occasionally more or less science-inspired intuition. A good share of the frustration non-physicians have with medicine stems from this misunderstanding of what medicine fundamentally is. Yet in spite of all the inherent vagaries of medical practice, the fact remains: we can know things, there are tools to help us do this, and these tools are based on the scientific method.
Throughout most of its history medicine had no scientific footing at all, and for centuries it was as much a branch of philosophy as it was anything else. The theories of how the body worked were fanciful notions with little or no basis in observed reality. The Scientific Revolution changed that. Ever since William Harvey decided in the early seventeenth century to test thousand-year-old teachings of how the blood circulated by actually looking at what happened, the principle of experimental observation has been a foundation of medical science. Although the accepted ideas of disease causation remained outlandish to our modern eyes for many years after Harvey’s pioneering experiments, the world of medicine had still changed profoundly and permanently. Henceforth the best physicians would actually test to see if their theories had any basis in observed reality.
For Harvey and the thousands of medical researchers who came after him, if they had a theory about how the body worked, they tested it with experiments. But it is also important to understand we can use sound principles of scientific observation even when we have no idea, no theories, about why things happen. For example, we can compare two ways of treating a disease though we have no notion of what causes the disease. It is often the results of such naïve observations which lead to theories of disease causation, theories which can then be tested with further observations. But we can still reach some useful conclusions about what works and what does not work without having any understanding at all of the reason why.
The vital distinction to make is this: although we can have good observations without underlying theories, we cannot have useful theories that are not based upon some kind of valid observations. What does this principle mean for today’s parents? From the outset, parents should demand of anybody making a claim about how children’s bodies work that they show good evidence in support of their theories. And good evidence can be hard to find.
The physicians of Roman times, whose theories Harvey disproved, saw no need to provide any experimental evidence at all. These days parents hear conflicting evidence everywhere; all the partisans in these debates have evidence. The problem is to decide among the conflicting claims whose evidence makes sense and whose does not. Can parents do that? Again, without the need to make a biostatistician of anyone, I believe they can. What they need to learn, and then practice, is how to look critically at medical claims.
The first step is to understand that not all evidence is equal, even if the people collecting it wear a long, white coat and have degrees after their names. Like good detectives, we need to weigh the validity of different kinds of evidence, and medical scientists have established a hierarchy of kinds of evidence to help decide which is the most likely to be correct. The hierarchy runs from least reliable to most likely to be right. Parents may be surprised to learn the least reliable category is expert opinion, what an individual expert or even a committee of experts say is correct. Next in believability comes various forms of what are called uncontrolled studies, and most reliable are what are called randomized, controlled trials (RCTs).
You will read about the first of these, expert opinion, in the next post.
I’ve written before about how children from poor families have a higher chance of needing PICU care than do children from more affluent families. Eligibility for Medicaid is a good marker for this; nearly half the population of most urban PICUs is made up of children on Medicaid, even though the national average (it varies a little from state to state) for children on Medicaid is about 20%. So poor kids are more likely to become critically ill.
Now a new report from the Robert Wood Johnson foundation, a renowned health policy organization, lays out how poverty correlates so closely to poor health. This chart is the most telling. It measures a somewhat vague quantity, something they call “children in less than very good health.” They obtain this value by surveying parents, so you could quibble about the validity of whatever it is the term measures. That quibble would make sense to me if the numbers weren’t so striking.
But they are striking. For example, among white, non-Hispanic children, 20% of poor children have “less than very good health,” compared with 6% of well-off children. The differences among black and Hispanic children are much more dramatic. Nearly 50% of poor, Hispanic children are not in optimal health.
What this means to me is fairly obvious, and it has been obvious for a long time — health status is linked to socio-economic status. We shouldn’t need a study to tell us that, but it is helpful to have such a graphic demonstration of the effect. I’m sure it’s partly because poor families can’t afford health insurance. But that isn’t the whole story — all of these poorest children, the group with the most severe health problems, would qualify for Medicaid, even in the states with the most stringent requirements.
Thus whatever we do about healthcare reform will be closely linked to what is happening in the economy. Perhaps the best thing we can do for healthcare is reduce poverty.
There is a long tradition in folklore, one shared by shamans and occultists, that knowing the true name of something gives you power over it.
Many years ago I had a very sick patient in the PICU who one morning, totally out of the blue, broke out in a bright, red rash all over his body. The boy had many critical problems already and, although the rash didn’t seem to be causing him any additional difficulties, it was dramatic. I worked my way down the list of usual things that cause such rashes and nothing seemed likely, so I asked a skin expert, a dermatologist, to come and take a look at it.
The dermatologist, a distinguished professor in a three-piece suit, gravely looked at the rash, removed his half-glasses, and pronounced that the child had “a generalized erythroderma.” I was then a bit rash myself at times, so I demanded: “I can see his skin is red all over (which is what “generalized erythroderma” means translated into English), but what is the rash from?” The professor was not amused. He had named the thing — that alone was useful and important.
What I call “Rumpelstiltskin syndrome” is the tradition in medicine that putting a name to a disorder, for example a set of symptoms, goes a long way toward controlling the problem because it gives our minds power over it. It is a little like the fairy tale in which Rumpelstiltskin conceals his name from the miller’s daughter. She is in his power until she happens to learn it, after which she is on top of the situation. No one wants to be a diagnostic enigma — we feel better when we have a name to call our malady, even if we can’t do anything about it.
Classifying diseases and relating them to one another, and in the process giving them names, is an ancient tradition in medicine. Called nosology, physicians tried, like botanists studying plants, to derive a sort of family tree of diseases. Even if there was no effective treatment, at least naming the disease would allow prognosis, the art of predicting what will happen to the patient. That is a useful thing. For a very long time in medical history it was nearly all physicians had to offer.
It also can be a first step in figuring out the cause of a disease. For example, before we knew anything about the human immunodeficiency virus (HIV), doctors identified patients with an unusual cluster of symptoms and signs and called the problem acquired immunodeficiency syndrome, or AIDS. Now we know what causes AIDS, but at first all we could do was describe and name it.
So, like the miller’s daughter and Rumpelstiltskin, knowing the ‘true name’ of a medical condition is helpful. But it also is true that that the simple naming act makes us feel better about the situation, if only because now we know what to call it.
