
Social
science
Try
it and see
In the
social sciences, it is often supposed, there can be no such thing as
a controlled experiment. Think again
IN THE
scientific pecking order, social scientists are usually looked down
on by their peers in the natural sciences. Real scientists do experiments
to test their theories-or, if they cannot, try to look for natural phenomena
that can act in lieu of experiments. Social scientists, it is widely
thought, do not subject their own hypotheses to any such rigorous treatment.
Worse, they peddle their untested hypotheses to governments, and try
to get them turned into policies.
The Campbell
Collaboration, whose second annual conference has just taken place in
Philadelphia, exists to change both this perception and the reality
behind it by advancing the cause of "evidence-based" social
policy. The collaboration is an international, independent, non-profit
organisation that brings together social scientists, statisticians and
policymakers. Its aim is to assemble and evaluate the best available
evidence for the effectiveness of various social interventions. In particular,
that means evidence from experiments.
Governments
require sellers of new medicines to demonstrate the safety and effectiveness
of their products. The accepted "gold standard" of evidence
is a randomised controlled trial, in which a new drug is compared with
the best existing therapy (or with a placebo, if no treatment is available).
Patients are assigned to one arm or the other of such a study at random,
ensuring that the only difference between the two groups is the new
treatment. The best studies also ensure that neither patient nor physician
knows which patient is allocated to which therapy. This "double-blinding"
reduces the risk that wishful thinking or other potential biases may
influence the outcome. Drug trials must also include enough patients
to make it unlikely that chance alone may determine the result.
Get
real
Yet the
medical industry is held to a higher standard of evidence than that
to which governments hold themselves. This is bad, because, as Carol
Fitz-Gibbon, a Campbell Collaboration participant from Durham University,
in England, points out, school education amounts to about 15,000 hours
of compulsory treatment. Social welfare and criminal-justice interventions
can be similarly invasive. But few education programmes or social initiatives
are evaluated in carefully conducted studies prior to their introduction.
A case
in point is the "whole-language" approach to reading, which
swept much of the English-speaking world in the 1970s and 1980s. Whole-language
holds that children learn to read best by absorbing contextual clues
from texts, not by breaking individual words into their component parts
and reassembling them (a method known as phonics). Unfortunately, the
educational theorists who pushed the whole-language notion so successfully
did not wait for evidence from controlled randomised trials before advancing
their claims. Had they done so, they might have concluded, as did an
analysis of 52 randomised studies carried out by the US National Reading
Panel in 2000, that effective reading instruction requires phonics.
To avoid
the widespread adoption of misguided ideas, the sensible thing is to
experiment first and make policy later. This is the idea behind a trial
of "restorative justice" which is about to begin in the English
courts. The experiment, initiated by Lawrence Sherman, a criminologist
from the University of Pennsylvania (with the support of England's Lord
Chief Justice), will include criminals who plead guilty to robbery or
serious assault. Those who agree to participate will be assigned randomly
either to sentencing as normal or to participation in a conference in
which the offender meets his victim and discusses how he may make emotional
and material restitution. The purpose of the trial is to assess whether
such restorative justice reduces reoffending. If it does, it might be
adopted more widely.
Other randomised
trials going on in Britain include an evaluation of the educational,
nutritional, social and psychological effects of free breakfasts in
English schools, a trial of smoke-alarm installation, and a trial of
sex education in Scottish secondary schools.
We have
control
The idea
of experimental evidence is not quite as new to the social sciences
as sneering natural scientists might believe. In fact, randomised trials
and systematic reviews of evidence were introduced into the social sciences
long before they became common in medicine. Iain Chalmers, a founder
of both the Campbell Collaboration and its older and better established
medical sibling, the Cochrane Collaboration, identifies an apparent
example of random allocation in a study carried out in 1927 of how to
persuade people to turn out to vote in elections. And randomised trials
in social work were begun in the 1930s and 1940s. But enthusiasm later
waned. Brian Sheldon, a social worker from the University of Exeter,
in England, suggests this loss of interest can be attributed, at least
in part, to the fact that early experiments produced little evidence
of positive outcomes.
Thomas
Cook, a pioneer of controlled experiments in education at Northwestern
University, in Chicago, suggests that much of the opposition to experimental
evaluation stems from a common philosophical malaise among social scientists,
who doubt the validity of the natural sciences, and therefore reject
the potential of knowledge derived from controlled experiments. A more
pragmatic factor limiting the growth of evidence-based education and
social services may be limitations on the funds available for research.
Nevertheless,
some 11,000 experimental studies are known in the social sciences (compared
with over 250,000 in the medical literature). Randomised trials have
been used to evaluate the effectiveness of driver-education programmes,
job-training schemes, classroom size, psychological counselling for
post-traumatic-stress disorder and increased investment in public housing.
And where they are carried out, they seem to have a healthy dampening
effect on otherwise rosy interpretations of the observations. An examination
of 308 studies from the criminal-justice literature, by David Weisburd
of the University of Maryland and his colleagues, found that randomised
trials were significantly less likely to report positive outcomes than
non-randomised studies. Analysis of work in other areas gives similar
results.
The problem
for policymakers is often not too few data, but what to make of multiple
and conflicting studies. This is where the Campbell Collaboration comes
into its own. Rather than initiating research, it is designed to evaluate
existing studies, in a process known as systematic review. This means
attempting to identify every relevant trial of a given question (including
studies that have never been published), choosing the best ones using
clearly defined criteria for quality, and combining the results in a
statistically valid way. The Cochrane Collaboration has produced more
than 1,000 such reviews in medical fields. The hope is that rigorous
review standards will allow Campbell, like Cochrane, to become a trusted
and authoritative source of information.
The evidence-based
policy movement has its detractors, however. Notably, many object on
ethical grounds to the idea of randomly denying half of a population
the potential benefit of a new service or initiative. But there is a
rejoinder: to ask for evidence that the intervention in question is
not itself harmful.
For example,
who could object to driver education programmes in schools? But three
different studies of a total of about 15,000 students have shown that
such programmes are likely to increase road deaths. That is because
training programmes, while not producing significantly safer drivers,
cause young people to obtain driving licences at an earlier age. And
more young drivers means more accidents.
Or take
the approach to criminal deterrence commonly known as "scared straight".
The first completed Campbell Collaboration review is an analysis of
such programmes, which introduce juvenile delinquents to prison inmates
who portray conditions inside in harsh, or at least realistic, terms.
The theory is that exposure to the grim realities of life in prison
will deter at-risk youths from future crime. A nice idea-but wrong.
Using a
variety of statistical techniques, Anthony Petrosino, a criminologist
at the US Academy of Arts and Sciences, combined the results of the
seven available randomised studies of scared-straight programmes. This
meta-analysis, as it is known, strongly suggests that participation
in a scared-straight programme substantially increases the likelihood
of subsequent arrest among participants.
Because
they are cheap to run and politically popular, scared-straight programmes
are widespread in the United States. Yet the obvious conclusion from
Dr Petrosino's analysis is that such programmes are not only harmful
to participants, but also place everybody in society at increased risk
of crime. A few more counter-intuitive results such as that, and experiment-based
social science might at last be given the respect that it deserves.
©
Copyright The Economist Newspaper Ltd 2002. All rights reserved.