7.4: Quasi-Experimental Study Designs

In 1979, in their classic textbook "Quasi-Experimentation: Design and Analysis Issues for Field Settings" Cook and Campbell describe quasi-experimental designs and their limitations. That work is the basis for the following discussion. Quasi-experimental designs get their name because they are not true experimental designs, where patients are randomly assigned to treatment and control groups. In quasi-experimental designs, outcomes may only be measured at the end of the study (rather than at the beginning and end), there may not be a control group, or patients may be assigned to the control and treatment groups by methods other than randomization.

Quasi textbook

Why bother with quasi-experimental designs? Why not just structure every study as a randomized controlled trial (RCT)? Unfortunately, it is not always possible or practical to do an RCT. It may not be ethical to do a RCT in some cases (for example, of tobacco use), it may be too expensive, especially for early or exploratory studies, or it may be inconvenient or administratively difficult as in an educational study. Also, quasi-experimental studies can sometimes provide a more natural, generalizable environment that better establishes effectiveness (as opposed to efficacy).

A short-hand proposed by Cook and Campbell and adopted by many others uses the following code to describe quasi-experimental designs:

R = randomization
On  = observation at time n
X = intervention (i.e. surgery or giving a drug)


Post-test only design

The post-test only design is the simplest and probably weakest quasi-experimental design. It has the form:

X   O1
An example is a surgeon who develops a new technique for performing an inguinal hernia repair. He may perform this repair on the next 20 patients in his practice ("X" in the diagram above), and then report the success rate for this small case series 2 weeks later (time O1). Obviously, we have no idea whether this new repair is better or worse than the old repair since there is no comparison group. Another example is the otherwise healthy patient who presents to her general internist with a cough and fever of 5 days duration, but no signs of pneumonia. There is about a 60% chance that she will be given an antibiotic like azithromycin, and when she feels better a week later, she is likely to credit the drug (as is her physician). But this is a post-test only "study", and is misleading. In fact, for patients like this, antibiotics do little or no good (Echt, 1991).

On the other hand, there are some situations where this is the only type of evidence we have for an intervention, and that is probably sufficient. A facetious example is the effectiveness of parachutes for persons jumping out of an airplane (Smith, 2003), while a real-life example is surgery to drain an abscessed gallbladder. But as we discussed earlier with surgery for appendicitis, even when the primacy of a therapy seems obvious and is widely accepted, it may not be the only or even the best approach.

Pretest/post-test design

An improvement to the post-test only design, particularly when there the patient has a chronic illness or there is a longitudinal process being evaluated (such as education or performance), is the pretest/post-test design. It adds an observation before the intervention:

O1    X   O2
You have all been part of this study design. For example, in a study of an educational intervention to improve knowledge about evidence-based medicine, learners might take a pretest, attend some classes, and then take a post-test. While this design can tell us about improvement or worsening in the group studied following the intervention, it doesn't tell us a) whether they would have improved anyway, or b) whether another approach would have been more effective. To learn that, you have to add a control group.

Non-equivalent control groups post-test only

Control groups are an important way to account for threats to validity, especially history, since both groups are (hopefully) exposed to the same external events and influences. When patients are randomized to treatment or control groups, they are assumed to be equivalent because of the randomization. When some other approach is used to assemble the treatment and control groups, they are assumed to be "non-equivalent". The simplest non-equivalent control group design uses only a single post-test:

X    O1

An example would be a study of patients with inguinal hernia, where some patients choose surgery while others choose to just wait. After 6 months, functional outcomes are measured (O1).

Non-equivalent control groups pretest/post-test

The next step in the evolution of quasi-experimental designs is to add a non-equivalent control group to a pretest/post-test design:

O1    X   O2
O1          O2
An example would be a study comparing a new protocol for treating hyperglycemia in hospitalized patients, with patients in one ward treated using usual care and patients in a second ward treated with by adjusting their basal insulin dose. O1 would represent the initial glucose level and O2 the glucose level 6 hours later. While it seems like a fairly strong design, without randomization unknown confounders and unmeasured differences between groups can bias the results.

Time series

Time series designs resemble the pretest/post-test designs, with the exception that there are multiple measurements before and after the intervention:

O1    O2    O3    X   O4    O5    O6
This type of study can provide better protection against historical threats to validity. Even better protection is provided by the addition of a non-equivalent control group:

O1    O2    O3    X   O4    O5    O6
O1    O2    O3          O4    O5    O6

Below is a hypothetical example of data from a time series study with a non-equivalent control group of an intervention to increase the percentage children walking to school:

Time series
If you look at the graph of the control group, you can see that there are annual fluctuations in the rate of walking to school. Having multiple measurements before and after the intervention, as well as a comparison group, makes a more convincing case for the efficacy of the intervention. Time series are sometimes called "historically controlled studies", especially when outcomes are measured months or years before and after the intervention.

As you can see, the above designs are combinations of pretest/post-test vs post-test only; multiple pre and post-tests (time series); and presence or absence of a non-equivalent control group.

Ecologic studies

Ecologic studies compare the incidence or outcome of disease in different geographic regions. They are subject to a range of biases, including ascertainment bias, since different regions may approach the diagnosis of disease differently. This is particularly true of conditions like depression or alcoholism that sometimes carry negative social connotations that vary by culture or ethnicity. An example of an ecologic study is one of the association between increasing primary care physician supply and the incidence and mortality rates for colorectal cancer in 67 Florida counties. They found a strong negative correlation between primary care physician supply and both incidence (-0.46) and mortality (-0.29). That is, having a greater proportion of primary care physicians in a county was associated with a reduction in the incidence and mortality due to colorectal cancer. The opposite was seen for subspecialty physician supply.  (Roetzheim, 2001) The authors adjusted for demographic and other differences between counties. This is important, because it may simply be that it is more economically advantageous for primary care physicians to locate in areas where there are healthier patients or better insured patients, and that these patients are less likely to get cancer.

The "ecological fallacy" occurs when one draws conclusions about individual members of a group based on the average performance of the group. For example, although rates of heart disease are lower in Italy than in the United States, one cannot assume that every individual Italian is at lower risk of heart disease than every American. Ecologic studies sometimes take advantage of migration of populations. If the incidence of a disease in the home country is higher than in the country to which they migrate, a pattern of increasing incidence in subsequent generations is often seen as immigrants adopt the new country's lifestyle and habits. An example is the rate of gastric cancer in Japanese immigrarnts to the United states, which is lower than that in Japanese citizens.

Previous section Next section