randomizing sample selection and processing order minimizes these procedural biases, makes the interpretation of the results clearer, and makes them more likely to be replicated.
many of the replication experiments blinded and randomized, but it’s not known if the original experiments did. all that is known is that for the 15 animal experiments, only
one of the original studies reported randomization and none reported blinding
. but it would not be surprising if many of the studies neither randomized nor blinded.
study design and statistics
according to one estimate, over half of the one million articles published each year
have biased study designs
, contributing to 85 per cent of us$100-billion spent each year on (mostly preclinical) research being wasted.
in a widely reported commentary, industry scientist and former academic glenn begley reported being able to reproduce the results of only
six of 53
academic studies (11 per cent). he listed
six practices
of reliable research, including blinding. all six of the studies that replicated followed all six practices. the 47 studies that failed to replicate followed few or, sometimes, none of the practices.
another way to bias findings is by misusing statistics. as with blinding and randomization, it’s not known which, if any, of the original studies in the reproducibility project misused statistics, because of the studies’ lack of transparency. but that, too, is common practice.
a dictionary of terms describes a slew of poor data analysis practices that can manufacture statistically significant (but false) findings, such as
harking
(hypothesizing after the results are known), p-hacking (
repeating statistical tests until a desired result is produced
) and following a series of data-dependent analysis decisions known as a “
garden of forking paths
” to publishable findings.