Skip to content

The Problem with Positive Results

June 29, 2012

It is a commonly known problem that journals have a bias toward publishing positive results, or results which say X causes Y rather than results which say X and Y are unrelated. To get a paper published, scientists are incentivized to falsify or misinterpret results. Not all scientists fall to this temptation, but on the margin, some do some of the time. Also, scientists have control over many different variables while testing, especially in the softer sciences. With so many “researcher degrees of freedom“, tests can be run with dozens, or even hundreds of specifications and modifications until the results the researcher wants to see are found. If you torture the data long enough, it will confess.

The second problem is that people who want to discover the truth never get to see the negative results and are fooled into thinking they are all positive. So, suppose there are 10 studies on the effect of rock climbing on lung cancer. 9 of them find that rock climbing does not cause lung cancer, and then don’t get published so no one sees them. The last finds rock climbing causes lung cancer (5% significance, would happen about 50% of the time over 10 trials!) and that’s the only one future researchers will see. One study which shows rock climbing causes cancer, despite the fact that overwhelming evidence says it does not. Not only that, newspaper articles will be filled with speculative causal connections, and the scientist will have their careers launched for warning us about the horrible risks of rock climbing.

The chart below shows the distribution of z statistics, which show that there is a statistical abnormal amount of results which are just barely statistically significant.

Replications and Falsifiability
The Scientific Method is the greatest tool humanity has to discover patterns in our world and to separate the intellectual wheat from the chaff. If you have a hypothesis, you subject that hypothesis to a test designed to falsify the hypothesis. Only when the hypothesis has been subjected to a variety of tests over time is it accepted as true. Alas, journals don’t seem to get this. Hypotheses are tested by people with a vested interest in proving them right, and once statistical significance is found, never tested again. Replications are rarely published and are low prestige. Falsifications are attacked, as if they are impugning the personal honor of the researcher rather than just trying to discover the truth. In my opinion, if a scientist objects to having their work retested or refuses to release their data, that is as good as admitting they are charlatans.

There are a couple ways to address this problem. First, I think that any major study should automatically have another group try to replicate their results as a matter of procedure. Second, I don’t think a journal should publish any result from a scientist not willing to disclose their data and methodology. Third, I think that journals should approve of papers before they find out what the results are. A scientist should submit their idea, methods, and tests, and only after a journal has approved or rejected the test do they release the actual results. That would go a long way toward eliminating the bias toward positive results.

Further Reading
Problems with Psychology replications
47 out of 53 of cancer studies couldn’t be replicated
Yong on Science and Replication on Econtalk
Edward Leamer Let’s take the con out of econometrics
Feynman on Cargo cult science
Complex Equations help you get published, but lower impact. My guess is that there is a similar effect for complex statistical techniques.
Researchers with p values close to .05 are less likely to share their data.
A statistics heavy explanation

No comments yet

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: