How a p-value between 0.04-0.05 equals a p-value between 0.16-017

This blog post is presented in collaboration with a new interactive visualization of the distribution of p-values created by Kristoffer Magnusson (@RPsychologist) based on code by JP de Ruiter (@JPdeRuiter).

Question 1: Would you be inclined to interpret a p-value between 0.16- 0.17 as support for the presence of an effect, assuming the power of the study was 50%? Write down your answer – we will come back to this question later.

Question 2: If you have 95% power, would you be inclined to interpret a p-value between 0.04 and 0.05 as support for the presence of an effect? Write down your answer – we will come back to this question later.

If you gave a different answer on question 1 than on question 2, you are over relying on p-values, and you’ll want to read this blog post. If you have been collecting larger sample sizes, and continue to rely on p < 0.05 to guide your statistical inferences, you’ll also want to read on.

When we have collected data, we often try to infer whether the observed effect is random noise (the null hypothesis is true) or a signal (the alternative hypothesis is true). It is useful to consider how much more likely it is that we observed a specific p-value if the alternative hypothesis is true, than when the null-hypothesis is true. We can do this by thinking about (or simulating, or using the great visualization by Kristoffer Magnusson) how often we could expect to observe a specific p-value when the alternative hypothesis is true, compared to when the null-hypothesis is true.

The latter is easy. When the null-hypothesis is true, every p-value is equally likely. So we can expect 1% of p-values to fall between 0.04 and 0.05.

When the alternative hypothesis is true, we have a probability of finding a significant effect, which is the statistical power of the test. As power increases, the p-value distribution changes (play around with the visualization to see how it changes). Very high p-values (e.g., p = 0.99) become less likely, and low p-values (e.g., p = 0.01) become more likely. This increase is really slow. If you have 20% power, a p-value between 0.26 and 0.27 is still just as likely under the alternative hypothesis (1%) as under the null hypothesis (only higher p-values are less likely under the null hypothesis than under the alternative hypothesis). When power is 50%, a p-value between 0.17-0.18 is just as likely when the alternative hypothesis is true as when the null hypothesis is true (both are again 1% likely to occur).

If the power of the test is 50%, a p-value between 0.16-0.17 is 1.1% likely. That means it is slightly more likely under the alternative hypothesis, than under the null hypothesis, but not very much (see Question 1).

If the power of the test is 50% (not uncommon in psychology experiments), p-values between 0.04 and 0.05 can be expected around 3.8% of the time, while under the null hypothesis, these p-values can only be expected around 1% of the time. This means that it is 3.8 times more likely to observe this p-value assuming the alternative hypothesis is true, than assuming the null hypothesis is true (dear Bayesian friends: I am also assuming the null hypothesis and the alternative hypothesis are equally likely a-priori). That’s not a lot, but it is something. Take a moment to think about which ratio you would need before you would consider something ‘support’ for the alternative hypothesis (there is no single correct answer).

As power increases even more, most of the p-values from statistical tests will be below 0.01, and there will be relatively few p-values between 0.01 and 0.05. For example, if you have 99% power with an alpha of 0.05, you might at the same time have 95% power for an alpha of 0.01. This means that 99% of the p-values can be expected to be lower than 0.05, but 95% of the p-values will also be below 0.01. That leaves only 4% of the p-values between 0.01 and 0.05. I think you’ll see where I am going.

If you have 95% power (e.g., you have 484 participants, 242 in each of two conditions, and you expect a small effect size of Cohen’s d = 0.3 in a between participants design), and you observe a p-value between 0.04 and 0.05, the probability of observing this p-value is 1.1% when the alternative hypothesis is true. It is still 1% when the null hypothesis is true (see Question 2).

This example shows how a p-value between 0.16-0.17 can give us exactly the same signal to noise ratio as a p-value between 0.04-0.05. It shows that when interpreting p-values, it is important to take the power of the study into consideration.

If your answer on Question 1 and Question 2 is not the same, you are relying too much on p-values when drawing statistical inferences. In both scenarios, the probability of observing this p-value when the alternative hypothesis is true is 1.1%, and the probability of observing this p-value when the null-hypothesis is true is 1%. This is not enough to distinguish between the signal and the noise.

When power is higher than 96%, p-values between 0.04 and 0.05 become more likely under the null-hypothesis than under the alternative hypothesis. In such circumstances, it would make sense to say: “p = 0.041, which does not provide support for our hypothesis”. If you collect larger sample sizes, always calculate Bayes Factors for converging support (and be careful if Bayes Factors do not provide support for your hypothesis). Try out JASP for software that looks just like SPSS, but also provides Bayes Factors, and is completely free.

Even though it is often difficult to know how much power you have (it depends on the true effect size, which is unknown), for any study with 580 participants in each condition, power is 96% for a pretty small true effect size of Cohen’s d = 0.2. In very large samples p-values above 0.04 should not be interpreted as support for the alternative hypothesis. Now that we are seeing more and more large-scale collaborations, and people are making use of big data, it’s important to keep this fact in mind. Obviously you won’t determine whether a paper should be published or not based on the p-value (all interesting papers should be published), but these papers should draw conclusions that follow from the data.

One way to think about this blog post, is that in large sample sizes, we might as well use a stricter Type 1 error rate (e.g., 0.01 instead of 0.05). After all, there is nothing magical about the use of 0.05 as a cut-off, and we should determine the desired Type 1 error rate and Type 2 error rate when we design a study. Cohen suggests a ratio of Type 2 error rates to Type 1 error rates of 4:1, which is reflected in the well known ‘minimum’ recommendation to aim for 80% power (which would mean a 20% Type 2 error rate) when you have a 5% Type 1 error rate . If my Type 2 error rate is only 5% (because I can be pretty sure I have 95% power) than it makes sense to reduce my Type 1 error rate to 1.25 (or just 1) to approximately maintain the ratio between Type 1 errors and Type 2 errors Cohen recommended.

Relying too much on p-values when you draw statistical inferences is a big problem – everyone agrees on this, whether or not they think p-values can be useful. I hope that with this blog post, I’ve contributed a little bit to help you think about p-values in a more accurate manner. Below is some R code you can run to see the probability of observing a p-values between two limits (which I made before Kristoffer Magnusson created his awesome visualization of the original Mathematica code by JP de Ruiter). Go ahead and play around with the online visualization to see how much more likely p-values between 0.049 and 0.050 are when you have 50% power.

P.S.: But Daniel, aren’t p-values ‘fickle’ and don’t they dance around so that they are really useless to draw any statistical inferences? Well, when 95% of them end up below p=0.05, I think we should talk of the march of the p-values instead of the dance of the p-values. They will nicely line up and behave like good little soldiers. All they need is a good commander that designs studies in a manner where p-values can be used to distinguish between signal and noise. If you don’t create a good and healthy environment for our poor little p-values, we can’t blame the children for their parent’s mistake.

P.P.S.: You might think: “Daniel, be proud and say it loud: You are a Bayesian!” Indeed, I am asking you to consider what the true effect size is, so that you can draw an inference about your data, assuming the alternative hypothesis is true. As Beyoncé would say: “'Cause if you liked it, then you should have put a ring on it.” As a Bayesian would say: 'Cause if you liked it, you should have put a prior probability distribution on it’. Yes, I sometimes get a weird tingly feeling when I incorporate prior information in my statistical inferences. But I am not yet sure I always want to know the probability that a hypothesis is true, given the data, or that I even can come close to knowing the probability a hypothesis is true. But I am very sure I want to control my Type 1 error rate in the long term. This blog is only 10 months old, so let's see what happens.