In previous posts I described various kinds of medical evidence, ranked from less to more reliable. This time I’ll lay out how researchers collect the best evidence — something called the randomized, controlled trial. These are the gold-standard for clinical research in medicine. They are also the most difficult and expensive kind of clinical research to do.

The concept is simple. If, for example, an investigator wants to test if a particular pill is effective against a certain disease, she identifies a group of patients with the condition, gives half of them the pill and the other half a fake pill, called a placebo, and sees what happens. The key to the validity of the trial is that assignment of a patient to the group that gets the drug or the placebo is truly random (this is often done by computer code), and that neither the patient nor the investigator knows which patient is in which group until after the trial is over, at which time the code disguising the two groups is unsealed and the data analyzed. The randomized, controlled trial eliminates nearly all the problems you read about in earlier posts, although even randomized trials have occasionally been bedeviled by the discovery afterwards that the control and experimental groups did, in fact, differ in key ways.

Randomized, controlled trials are the centerpiece for a recent movement in medical practice called evidence-based medicine. The term could, of course, allow doctors to use any sort of evidence, but what people mean by evidence-based medicine is that doctors should be guided by the results, whenever possible, of proper randomized, controlled trials. If there are not any such data available, then doctors should use a formal, defined process for evaluating what evidence there is. They should weigh the evidence much as we are doing now, ranking it from expert opinion, through case series, uncontrolled trials, and up to any controlled trial information there is available.

There is even a name assigned to this formal evaluative process — meta-analysis. The notion of meta-analysis is that the results of several, disparate studies, which by themselves might be inconclusive, can be pooled together to reach a conclusive, composite answer. Of course one cannot make a silk purse from a sow’s ear; the summation of several bad studies can simply be a single bad study. However, meta-analysis does have the ability to make explicit how we judge the validity of medical research.

If randomized, controlled studies are the gold standard, why are we even discussing any of those less useful methods? Why not just do that kind of research for everything? The answer is two-fold. For one thing, relatively few disorders have been the subject of randomized trials because they are extremely complicated and expensive experiments to plan and carry out. They often take years to map out and execute, followed by another year or more to analyze the data. A second issue is that it is not really possible to devise a randomized, controlled trial to examine many of the medical questions parents — and pediatricians, too — have about their children, even if we had the time and money to do it.

Randomized trials are best suited to testing some kind of therapy or intervention. But the intervention must be of the sort that neither the researcher nor the subject knows if they are using the experimental treatment or the placebo. This is feasible for a pill, although even then it can prove difficult. For example, one of the trials about the effectiveness of fish oil in reducing the risk of heart disease was complicated by the fact the fish oil smelled and tasted a certain way, alerting the subject to what group he was in.

For some things it is difficult even to devise a placebo — a surgical procedure, for example. Some questions are so important to answer that patients have undergone (with their informed consent, of course) sham surgery as a way to blind both the subject and the evaluator to which group the patient was in. Considering how difficult it is to set trials like this up it is understandable why so many things, important things for children’s health, have never been studied with a randomized, controlled trial, and very likely never will be. We will just have to decide what to do with data which, although still usable, are intrinsically less reliable. That is unfortunate, but that is the reality.

For myself, I find this notion comforting — it means there still is a place in medical practice for intuition and common sense.

Here is another post in understanding the proper use of medical evidence. Medical researchers can use several techniques to try and get around some the pitfalls you read about in earlier posts. Various kinds of population studies are examples of how they do this.

These kinds of studies can be retrospective, meaning we look back in time at things that have already happened to people, or prospective, meaning we follow forward, in real time, a group of patients to observe what happens. They are most often used to determine if there is any association between something, say a drug or an environmental exposure, and a disease.

Finding such an association doesn’t prove causation. We must beware of another logical fallacy, cum hoc ergo propter hoc, or “with this, therefore because of this” in the Latin. But a very strong association between two things is reasonable evidence for a link of some sort between them, such as one thing at least partially causing the other, or else both being caused by some third thing. After all, where there’s smoke, there’s usually fire.

One common tool of this sort of research is the case-control study. In this technique the researcher tries to identify a group of people who have the thing she’s studying (the case group) and match it with another group of people who are just like the case group in such things as age, sex, and whatever else might confound the analysis, but who don’t have the disease (the control group).

The two groups can then be compared, looking for things that the case group has that the control group doesn’t; such things could then be associated with the disease, and maybe even cause it. The crucial part of this process is choosing a good control group. Sometimes each individual case has its own individual control, matched to it as closely as possible. A major problem with case-control studies is that investigators can only match the two groups for confounding variables they know or think at the time may be important for the disease they are studying; it may turn out later some completely different thing was important, something they didn’t control for when they matched the two groups because they didn’t know it mattered.

Here is a simple, hypothetical example of what I mean. Suppose an investigator has a theory a certain food is protective against cancer of the large intestine (colon). She identifies persons who developed the cancer, matches them with others who don’t have the cancer, and then does a detailed analysis of the diets of the two groups to see if there were differences between cases and controls in how much they ate of whatever the theoretically protective food item is. If she doesn’t match the case and control groups for the variable of having a family history of colon cancer, she will be seriously misled by the results because colon cancer is one of those malignancies that tend to run in families.

Case-control studies, if they use interview techniques, can be quite susceptible to recall bias. For example, imagine you are trying to determine if exposure to a particular food is associated with a medical problem. You assemble a group of patients who have a disease and ask them about exposure to whatever food you are investigating. Then you ask the same questions of those in the control group. The trouble is, usually researchers are investigating a potential connection like this because it’s already believed at least by some people, perhaps many people, that there is one. If it’s already widely believed there is such an association, unless you design your survey very carefully, the patients in the case group are more likely to recall an exposure to whatever the food is than are your controls.

Another way to examine, say, the theory a particular environmental agent or activity is associated with developing a disease is to do a prospective cohort study, selecting two groups of people who differ only in their exposure to the thing being studied, and then follow them over time to see what happens. This kind of study is much less likely to be plagued with recall bias because you are not asking people to remember what already happened to them. (It is also more time-consuming, often taking years.) If the hypothesis is correct, then those exposed to whatever the agent or activity you’re studying should be more likely to get the disease than are those who aren’t exposed. Like case-control studies, however, the key is in the selection of the control group — it must not differ from the case group in any way that could affect susceptibility to getting the disease.

Properly done, these kinds of population studies give much more reliable data than do the techniques I discussed in previous posts — expert opinion, case reports, and uncontrolled trials. However, they don’t lend themselves well to treatment trials, assessing what works, because the population researcher is generally just passively observing things. The best way to determine if a treatment works is to use the gold standard, known as the randomized, controlled trial. You will read about how those work in a later post.

Here is another post about how to evaluate the validity of a medical claim. My last one dealt with case reports as medical evidence. This discussion is about the next rung on the ladder of reliability of medical evidence — the uncontrolled trial.

Researchers can do other things to series of cases besides simply describe what the patients are like; they can manipulate the situation in various ways. For example, if a doctor looks at her series of patients and becomes convinced that a particular therapy will work for the disease, she can give the therapy to the next patient, or series of patients, that come her way with the problem, and see what happens. This would constitute one version of an uncontrolled trial, and is probably the oldest kind of treatment research doctors have used. Venerable as the technique is, it is easy to see how an experiment like this could yield misleading results.

First, the patient group is subject to the same selection bias of the case series — the assortment of people with the problem who come to see the doctor are unlikely to represent a random sample of all people with the disease. Next, the only way the doctor could decide the treatment might be helping would be to compare what happens in the patients who get the new therapy with the patients she saw in the past who did not. Such so-called historical controls are the weakest sort of control group. This is because they are subject to the same kind of selection bias as the experimental group, those who get the treatment. Worse than that, since they were seeing the doctor at an earlier time, they may not even be representative of the patients with the disease who are seeing her now. Finally, a doctor who believes that a particular treatment will work (which is, after all, why she is doing the experimental trial in the first place) is hardly the best person to decide impartially if that is so. All of us want our theories to be correct, so her evaluation is bound to be slanted — it is only human nature at work.

Uncontrolled trials like this are particularly susceptible to what logicians call the fallacy of post hoc, ergo propter hoc, translated from the Latin as “after this, therefore because of this.” Anyone who has watched late-night cable television has seen countless examples of this logical trap, in the form of personal testimonials from people who had this or that problem, took the pill or bought the product, and the problem went away. The fallacy, of course, is that the two events may be entirely unrelated, just as the fact I may drink coffee every morning before the sun comes up does not cause the heavens to move in that way.

Trials like this are also highly prone to suffer from the placebo effect, the trick the human mind plays on us to believe so much that a particular treatment is working we actually will it to happen. The power of wishful thinking in the human mind is astonishing. Even more astonishing is that, in some situations, the “useless” placebo, a sugar pill or its equivalent, actually does improve the situation, if only slightly (15-30% by most estimates). So oftentimes people get a little better no matter what the therapy. The only way we can be sure the improvement is from the therapy (as a therapy, and not a placebo) is to blind both the patient and the observer to knowing which patients got the treatment and which ones got the placebo.

A new law went into effect in South Dakota this past July that represents a serious infringement on the right of patients and their physicians to do what they think is best for the patient. The law concerns abortion, an incendiary topic. I’m not writing here to debate the rightness or wrongness of abortion; for one thing, in my experience such a discussion never changes anyone’s mind and often becomes vitriolic. Rather, I’m writing to bring attention to what I see as a precedent dangerous to good medical care.

The background to this law is that abortion opponents, stymied by several Supreme Court decisions, most notably Roe v. Wade, have over the years tried various roundabout techniques on the state level to limit severely or even prevent abortion. These measures take the form of making it more difficult for a woman to obtain an abortion. The South Dakota law follows this pattern.

The law mandates that the physician hand a woman seeking an abortion a lengthy written document. The precise language of this document is decreed by the state legislature, and the woman must sign each page of it. However you feel about abortion, the key problem here is that a legislative body is, in effect, practicing medicine. The language of the statement is not medical, it is political. Any reasonable person realizes that the principal intent of the law is to reduce (or even eliminate) the number of abortions in South Dakota.

My concern is that, if this practice is allowed, there is nothing to stop legislatures (or Congress) from passing similar laws about any medical situation they wish. From my perspective as a pediatric intensivist, I’m most concerned about intrusion into the wrenching end-of-life discussions all intensivists from time to time have with families, but in theory such laws could be about anything. Several constitutional scholars note that the law probably violates physicians’ first amendment rights to free speech.

You can read the precise language contained in the statement, as well as more about the controversy here.