If you’ve read the news the past few years you might think that all undesirable weather is caused by climate change and all increases in crash rates are caused by distracted driving.
These reports mainly offer insight into the people who write them. People who are curious ask, “why did 39 people got run over in crosswalks during a drought?” People with an agenda ask, “are cell phones and carbon dioxide to blame?”
It’s similar to the difference between “is Saddam Hussein stockpiling weapons of mass destruction?” and “can we find evidence to link Saddam Hussein to WMD?” The so-called “Downing Street Memo” accused the Bush administration of asking the second question instead of the first. The memo was supposed to be scandalous, but I’ve seen that sort of thinking too many times to be surprised.
It’s how prosecutors approach criminal cases. Once they decide who did it, they’re looking for evidence to support their belief. They’re not trying to convince themselves any more. They’re trying to convince the jury.
It’s how a lot of traffic committees approach sign requests. They ask “can we post this sign?” instead of “should we post this sign?” They had already decided they wanted the sign. They are looking for evidence to support their decision — and only for evidence to support their decision. I’ve seen hearings where only people who support a proposal are allowed to speak.
When you run the process, you can make an ambiguous situation come out your way. You can make facts that are against you look like facts in your favor.
A professor asked one of his students why a paper used an unusual statistical significance test. The answer was, it was the only test that said the results were statistically significant. All the commonly used tests said the experiment was a failure.
In the idealized scientific method you learned in high school, this wouldn’t be allowed. You would decide ahead of time what test to use to measure success. But science isn’t that neat in reality. Sometimes there are good reasons. Sometimes…
Billions of dollars can be won or lost depending on whether the FDA approves a new drug. Companies realized if they ran several clinical trials and only reported the ones that said the drug worked, they could make more money. The government has been trying to stop that for 20 years.
The study I wrote about claiming that speed cameras are good for you used different statistical methods on different parts of the data, depending on which gave better results on that part of the data. It cites Ezra Hauer’s Observational Before-After Studies in Road Safety as a reference, but fails to mention Hauer’s warning that the per-vehicle crash rate declines as a road gets busier. This was important because the road with cameras did get busier after cameras started ticketing.
Did you read the references of the last paper on traffic safety that made the news? Probably not. You didn’t read the paper. You read a press release rewritten as an article.
When you read the paper and the list of references you can see what was left out. Corrections for traffic volume, for example, in a couple papers. Or you can see what was added, like fabricated safety data in a third paper.
What’s important is to be skeptical. Ask questions. What isn’t being said? Why was that anaylsis method used? Why was the data split up that way?
Remember Henry Fonda in 12 Angry Men? Almost anybody can be convincing if nobody is prepared to show a little doubt.
The opinions expressed in this post belong to the author and do not necessarily represent those of the National Motorists Association or the NMA Foundation. This content is for informational purposes and is not intended as legal advice. No representations are made regarding the accuracy of this post or the included links.
