After how many p-values between 0.025-0.05 should one start getting concerned about robustness?

Like insects sensitive to the infrared spectrum, researchers have evolved a special sensitivity to p-values between 0.025 and 0.05. Sure, you’ve got your p < .001’s and n.s.’s, but your life as a scientist might depend on your special sensitivity to 0.025<p<0.05.

 Our increased sensitivity to these p-values might make us forget what a small part of the p-value spectrum we are talking about here – just 2.5%. At the same time, we are slowly realizing that too many high p-values are rather unlikely. This lead Michael Inzlicht to wonder:

Editor question: After how many p-values <.05 but >.025 should one start getting concerned about robustness? What's fair?

— Michael Inzlicht (@minzlicht) May 26, 2015

Now if someone who is so serious about improving the way he works as Michael Inzlicht wants to know something, I’m more than happy to help.

I wanted to give the best possible probability. Using this handy interactive visualization it is easy to move some sliders around and see which power has the highest percentage of p-values between 0.025 and 0.05 (give it a try = it’s around 56% power, when approximately 11% of p-values will fall within this small section). If we increase or decrease the power, p-values are either spread out more uniformly, or most of them will be very small. Assuming we are examining a true effect, the probability of finding two p-values in a row within 0.025 and 0.05 is simply 11% times 11%, or 0.11*0.11=0.012. At the very best, published papers that simply report what they find will contain two p-values between 0.025 and 0.05 1.2% of the time.

NOTE 1: Richard Morey noted on Twitter this calculation ignores how researchers will typically not run two studies in a row, regardless of the outcome of the first study. They will typically run Study 2 only if Study 1 was statistically significant. If so, we need to calculate the conditional probability that Study 2 found a significant effect between 0.025-0.05, conditional on the probability that Study 1 found a significant effect between 0.025-0.05 (with 56% power). Thus: p(0.025<p<0.05|p<0.05, assuming 56% power). This probability is 21%, which makes the probability across two studies 0,21*0,11=0.023, or 2.3%. 

We can also simulate independent t-tests with 56% power, and count how often we find two p-values between 0.025 and 0.05 in a row. The R script below gives us the same answer to our question.

Note 2: Frederik Aust and Rogier Kievit remarked on Twitter that if you multiply enough probabilities, the number will always be small in the end. I agree. We can compare observing 2 p-values between 0.025-0.05 when we have 56% power with observing these two p-values when the null-hypothesis is true. If we again take the conditional probability, this is 0.5*0.025 - 0.0125. The conditional probability with 56% power was 0.023. This means the observed pattern across two studies is 1.84 times more likely under the alternative hypothesis than under the null-hypothesis. Even though we have 2 significant studies, the evidence for the alternative hypothesis is purely anecdotal. 

If you want to focus on the probability of p-values between 0.01 and 0.05, the interactive visualization shows the optimal power is around 62%, when approximately 24% of the p-values will fall between 0.01 and 0.05. Finding two studies within these p-values is not improbable (it happens around 0.24*0.24 = 6% of the time), but a third study within this interval occurs only 1.3% of the time. Again, it can happen, but not very often. 

If you want to use the frequency of p-values as an indication of the robustness of results, don't decide when to use the rule or which boundaries you will use after looking at the data. But if you always use the rule to doubt the robustness of two p-values between 0.025 and 0.05 in two study papers, and three p-values between 0.01 and 0.05 in three study papers, you won't make too many errors in the long run. An obvious exception is when authors pre-registered all their studies. 

What should you do as an editor when you encounter a set of studies with p-values that are relatively unlikely to occur? First, you can ask the authors to discuss the situation. For example, when you explicitly mention the set of studies becomes more probable when a non-significant finding is added, the authors might be happy to oblige. Second, one of my favorite solutions is to decide upon an in principle acceptance (assuming the article is fit for publication), but ask the authors to add one replication. The authors are guaranteed of publication irrespective of the outcome of the replication, but we have a better knowledge of what is likely to be true.