Tutorial 2: Replication

How Does Replication Change the Scene?

If the base rate is low, then a positive result is no guarantee than one's hypothesis is correct, even when the study is well-powered. That said, there are things one can do to improve the pre-study probability that a hypothesis is true. One of these is to replicate previously conducted studies (or alternatively, to perform novel tests of previously investigated hypotheses).

As an example, let's assume that we are looking to detect genes associated with a certain mental disorder. There are 100,000 candidate genes, and we expect that 100 genes will be associated with the disorder. This means that our base rate is 100/100,000, or 0.001, which is pretty reasonable (and possibly optimistic) in fields like behavioral genetics. Our experimental methods have a power of 0.6, meaning that a true association will be detected 60% of the time. False hypotheses will yield positive results at a rate of 5%.

If we test all 100,000 genes, we should expect approximately (0.6 × 100) + (0.05 × 99,900) = 5,055 positive results, of which only 60 will reflect true associations. This means that if a gene tests positive, the probability that it is true is only 60/(60 + 5055) ≈ 0.012, just over 1%. So most genes with positive test results will NOT be associated with the disorder.

Let's take the genes that got positive test results and test them again. Now, of the 60 true associations, (0.6 × 60) = 36 genes will test positive, and there will (0.05 × 5,055) ≈ 253 false positives. In other words, the probability that a gene that tests positive twice in a row is really associated with the disorder is 36/(36 + 253), or just under 13%. That's still not that high, but we can see how replication can increase our ability to separate the true hypotheses from the false. If we replicate the double-positive results yet again and get a third positive result, the probability that the hypothesis is true rises to over 63%. Assuming we continue to get positive results, each subsequent result makes us more confident that our hypothesis is correct. This graph illustrates how, assuming we continue to get positive results, we become increasingly confident that our hypothesis is true.


We Should Be So Lucky

True hypotheses will not always yield positive results (hence the concept of power), and false hypotheses will not always yield negative results (hence the concept of false positives). There are also several other factors that further complicate the picture. These include the different strategies researchers use to determine whether to replicate and which studies to replicate, as well as the ell-known bias against the publication of negative results and replications1. In our paper2, we explore the impact of these factors and suggest some strategies for coping with them.

In the game you just played, you saw that the ability to replicate can help to separate the true hypotheses from the false, as initial results do not always reflect the reality of the situation. However, wish as we may, we don't have unlimited freedom to replicate as much as may be desirable (or even necessary). There are limits to time, labor, funds, and opportunity. Next up, we'll see how how those limitations affect our ability to use replication to effectively sort true hypotheses from false.


1. Franco A, Malhotra N, Simonovits G (2014) Publication bias in the social sciences: Unlocking the file drawer. Science 345:1502-1505.
Fanelli D (2012) Negative results are disappearing from most disciplines and countries. Scientometrics 90:891-904.
Nosek BA, Spies JR, Motyl M (2012) Scientific Utopia: Restructuring incentives and practices to promote truth over publishability. Perspectives in Psychological Science 7:615-631.

2. Once the paper is published, there will be a link to view it.