Before Any Major Policy, Sacrifice a Goat
We need more goat sacrifices
A survey of failed social interventions—$3 billion in School Improvement Grants that did nothing, Chief Diversity Officers who didn't diversify campuses, One Laptop Per Child's zero effect in Peru, and a vaccine message that backfired—makes the case that well-intentioned programs routinely fail. Drawing on Peter Rossi's satirical Iron Law of Evaluation and the illusion of control, the author argues we should ritually remind ourselves that we know nothing before meddling in each other's lives.
In the social sciences, nothing ever dies; it merely becomes embarrassing.
- GrantMoyer
Isn't trying a plausible sociological intervention then measuring its effectiveness a win regardless of how effective it ended up being? Either way, we know more about what's likely to work for future interventions. It's the scientific method.
These interventions target very complex systems. It's not unexpected that we need to try a bunch of things that don't work before we find something that does, to the extent that most of the things we try are ineffective, right?
Edit: I wrote the above before reaching the end of the article. By the end, the article reaches a similar conclusion, albiet with less science and more mysticism.
Maybe that means it's a good article, leading me to reach its conclusion before taking the stance itself. On the other hand, the article's leading sections gave such a strong impression that the author thought something needs to be drastically changed about the current system — the opposite stance from its conclusion — on seemingly shakey grounds, that I almost didn't finish reading it. I'm not sure what to think, aside from please leave innocent goats alone.
- shit_game
> One of my favorite genres of psychology experiment is called something like: “the study is probably bullshit but the effect is probably real”. At the top of that heap is The Illusion of Control, a classic from 1975 and a favorite of Psych 101 classes everywhere. Usually, this study gets summarized as “people value their lottery tickets more when you let them pick the numbers vs. when you pick the numbers for them.” It’s as if people think that the mere act of picking imbues their ticket with a bit of juju, which makes them less willing to part with it.
> That would be a beautiful study, but it wasn’t the study that was actually conducted. The “lottery tickets” in question were in fact football cards, each one with a picture of a player, their team, etc. Some participants were given the opportunity to choose their football card and some had a card handed to them. Later, just before the drawing, all of the participants were asked how much it would take to buy their card back. On average, the pickers asked for $8.67, while the not-pickers said $1.86.4
> I’m sure you can see the obvious and less exciting explanation for these results: when people were given the chance, they picked their favorite players. The people who didn’t get to pick ended up with a card they liked less. This “illusion of control” effect is merely a “reality of people having preferences and acting on them” effect.5
Alternatively, people who are knowledgeable of a subject will have selfishly biased opinions […]
- n923263
> It’s as if people think that the mere act of picking imbues their ticket with a bit of juju, which makes them less willing to part with it.
No, the mere act of picking imbues their ticket with a bit of guarantee that you haven't maliciously selected numbers that don't win to cheat them. And I'd expect people's psychologies to pick up on that even if they can't explain why.
Even if that actually was the study that was conducted, it still wouldn't support belief in juju as an explaination.
And not considering that people might react strategically to the mechanisms of your experimental/intervention setup does seem to be a reccuring problem in social science/policy.
- ryan_glass
One very successful social intervention in the UK was the push to reduce drink-driving. Starting around the 80s I think, maybe earlier. There was a big media campaign with TV ads as well as implementing and enforcing stricter laws. What I find interesting is that many in the older generation still found it more acceptable to have a few drinks and drive, whereas those growing up during this period largely find it very socially unacceptable.
- zbentley
This article makes an “ok” case for social science being in its infancy (I think it is in some ways, but is more mature than widely thought in others), and a very good case for most social interventions being ineffective due to their being undertaken in methodologically flawed ways.
Most interventions are embarked upon by working backwards from unproven (often political) theories and an extremely limited range of possible tools (“give money”, “take money”, “physically punish”). That leads to the mother of all searching-only-under-the-streetlight fallacies.
But that doesn’t necessarily indict the social science academy/frontiers of understanding (which, again, are plenty flawed—just perhaps not quite as stone-aged as the author suggests). Usually, it indicts the state/designers of the intervention in question. Blaming the sciences sometimes sounds like blaming car airbag designers for motorcyclist deaths.
I’m reminded of the conservation efforts for the spotted owl. Famous academic naturalists were asked to design a conservation plan that worked with logging. They did. The state replied “fine, but we need to clear-cut 90% of the forest you asked for, so make do with 10% of the habitat”. The program did poorly, but not because the academic understanding of how to address the problem was as flawed.
- mwhitfield
The article talks about measurement of impact as though it's a totally value-neutral idea, but it's absolutely not:
* Measurement requires substantial logistical overhead; both up-front in creating a plan for measurement, and during intervention due to the extra record keeping involved. This is a (probably frequently large) drain on program efficacy. Just listen to any schoolteacher's take on standardized testing, evaluation rubrics, etc.
* Some contexts, and some types of intervention are more amenable to measurement than others. Insisting that nothing which is not measured can be valuable biases our decision-making towards the most measurable actions, which are likely not the best. Strong correlation with simplistic actions, short timelines, and lack of responsiveness to individual context.
* Goodhart's law.
Author (and everyone) would benefit from the book Seeing Like a State.
- AdmiralAsshat
> When Arkansas added a work requirement to Medicaid, the number of Arkansans with health insurance went down without any increase in employment.
The author is assuming good faith on behalf of the Arkansas state government that the purpose of the requirement was to increase employment, rather than merely cutting down on the number of citizens successfully receiving Medicaid.
- trgn
tough article. 90% is a tongue in cheek ridicule, with lots of data points, and then the bait and switch at the end, with far less argumentation.
mainly, interventions are experiments, and therefore we need more of them, you never know. but they are not experiments, otherwise we wouldn't repeat failed ones over and over, or let existing failing ones run on and on.
maybe the answer really is more simple, that most programs are indeed politically motivated endeavors, indeed largely signals of a worldview, less programs to bring about material improvement.
fwiw - i dont know the answer, but having your cake and eat it, being perceptive of the absurdity, yet, not daring to call it out, is an author's disposition, less a truth. so maybe, if we do the one hand other hand game, accept the bleak truth glimpsed in the facts, but also accept that there's nothing to be done about it within existing frameworks, and not point at it as something desirable.