Before You Analyze the Data, Understand
It.
Before jumping into data analysis, it’s important to
understand how the data was obtained and whether additional information is
available. As an example, imagine that an educational consultant brings you, a
statistician, the approval rates for two subjects taught in both day and
evening programs at several schools where she works. She wants to examine
students’ grades in these two time slots to evaluate their interest in the
subject—and possibly offer materials better suited to their intended careers.
Suppose the sample is large and that several variables
are available, such as gender, age, employment status, and school location. But
you, being a thoughtful statistician, begin to wonder: should I simply compute
the overall pass rates for each shift? And then you start asking questions.
Are older students less available to study than
younger ones? Are there more young people in the day program? Do students who
work have the same interests as those who don’t? Are there more non-working
students in the day program? Could the school’s location—perhaps with many
students from underprivileged areas—affect their future career paths?
You’re identifying variables that may influence the
outcome variable—in this case, the grades—and considering how these factors
might shape the results. You realize that if you lump all day students together
and all evening students together, you might be setting up a misleading
comparison.
These influential variables define the need for stratification in your
analysis. If not accounted for, they become confounding variables—and if ignored, they can
distort results. This situation is known as Simpson’s Paradox. (See the related post: Simpson’s Paradox: When Data Misleads.)
Here we highlight two real, though simplified,
examples of Simpson’s Paradox: one involved a suspicion of gender
discrimination in the 1970s at the University of California, Berkeley; the
other concerns a controversial result about kidney stone treatments, published
in the British Medical Journal
in the 1980s.
In the Berkeley case, the numbers seemed damning:
graduate programs had admitted 44% of male applicants, but only 35% of female
applicants. However, a deeper investigation revealed something surprising. Men
tended to apply to departments with lower competition, while more women applied
to highly competitive ones. When admission rates were analyzed by department, a
small but statistically significant bias in
favor of women emerged—meaning that, proportionally, women were
more likely to be admitted.
In the kidney stone study, researchers initially
concluded that a newer, less invasive treatment was more effective than
traditional surgery. But doubts emerged—until the patients were split into two
groups according to the size
of their kidney stones. This more appropriate analysis showed that the new
treatment had been more often used on patients with small stones, and that only in this group was it
actually more effective.
Simpson’s Paradox often misleads us in performance
assessments and data analysis when relevant subgroups are ignored. If the data
is analyzed only in aggregate, without accounting for possible sources of
variation, the conclusions may be deceptive. That’s why it’s essential to examine subgroups before
rushing to interpret results for the whole. Always think carefully about
potential confounders—and include
them in your analysis. Because Simpson’s Paradox is real, and it happens.
Simpson’s Paradox is a statistical
phenomenon in which a trend that appears in separate groups disappears—or even
reverses—when the groups are combined. This occurs because of confounding variables,
which influence the relationship between the variables under analysis.
References
·
Bickel, P. J.; Hammel, E. A.; O’Connell, J. W. Sex bias in graduate admission. Data from Berkeley. Science, v.187, n.4175, p.398–404, 1975.
·
Charig, C. R. Comparison
of treatment of renal calculi by open surgery, percutaneous nephrolithotomy and
extracorporeal shockwave lithotripsy. Brit. Med. J. (Clin. Res Ed), v.292,
n.6524, p.879–882, 1986.
No comments:
Post a Comment