I wrote recently about choosing evidence to fit a conclusion rather than a conclusion to fit evidence. Christie Aschwanden at 538 wrote a good article on that backwards journey. She called the process “p-hacking,” derived from the term “p-value.”
In traditional science published experimental results have to be statistically significant. Did the lab rats die from bad luck, or because a chemical killed them? You prove the latter with a statistical test. The resulting p-value gives the chance the result was due to bad luck instead of chemical toxicity.
Ideally, strictly following the scientific method, the data analysis technique should be fixed in advance. If your plan said fit a curve to these data points, people can tell if you had to work too hard to make the curve fit the data. (Chi squared per degree of freedom is one model quality metric I learned in school.)
In reality, scientists tend to generate a pile of data and figure out what to do with it later. They also go digging through somebody else’s pile of data looking for something good. The second kind might be called “retrospective analysis” or “meta-analysis.” When you see those terms, watch out for cherry-picking.
Let’s look at road safety statistics. These are normally retrospective.
Suppose in 2015 a state had 300 traffic deaths, 100 of them “alcohol-related.” In 2016 the count dropped to 280, 97 alcohol-related.
My conclusion would be the state was slightly safer, but the change was not statistically significant. You’re likely to hear the opposite on the news: there is a crisis of drunk drivers demanding tough new laws.
The reason those stories get published is the decline in “alcohol-related” deaths was not as fast as the overall rate. Those 97 deaths represented 35% of the total, compared to 33% the year before.
If percentages don’t break the right way, absolute numbers might. Say deaths increased to 350 and alcohol-related deaths increased to 110. Then the increase in absolute number makes the news. And if there isn’t any way to spin the data they pick a different subject for the press release. That’s called publication bias.
In honest science publication bias is recognized as a problem. One new journal, Journal of Negative Results in Biomedicine, wants to you to submit papers with findings like “our mice didn’t get cancer when we fed them junk food.” Because otherwise the literature will have the one experiment that says junk food causes cancer, and not the 19 unpublished experiments that say it doesn’t.
Much of what gets published on traffic regulation is written by people with an agenda who are not going to publish anything politically unsound. We could use a compilation of reports like “nothing happened when we changed the speed limit.” Instead we have one forward-looking controlled experiment and a bunch of retrospective analyses that only got published because the authors sliced the data to meet their sponsors’ needs.
If you want to try your hand at slicing data to suit your agenda, go to the 538 article I mentioned above. There’s a tool you can use to prove that Republicans are bad for the economy. Or if you’re so inclined, you can prove that Democrats are bad for the economy. Same data, different analysis.
In the old days you had to do some thinking to lie with statistics. Now computers will do it for you.
The opinions expressed in this post belong to the author and do not necessarily represent those of the National Motorists Association or the NMA Foundation. This content is for informational purposes and is not intended as legal advice. No representations are made regarding the accuracy of this post or the included links.
