An extended version of this blog post is now in press at PeerJ.
TL;DR version: De Winter and Dodou (2015) analyzed the distribution (and its change over time) of a large number of p-values automatically extracted from abstracts in the scientific literature. They concluded there is a ‘surge of p-values between 0.041-0.049 in recent decades’ which 'suggests (but does not prove) questionable research practices have increased over the past 25 years'. I show the changes in the ratios of p-values over the years between 0.041-0.049 are better explained by a model of p-value distributions that assumes the average power has decreased over time. Furthermore, I propose that their observation that p-values just below 0.05 increase more strongly than p-values above 0.05 can be explained by an increase in publication bias over the years (cf. Fanelli, 2012), which has led to a relative decrease of 'marginally significant' p-values in the literature (instead of an increase in p-values just below 0.05). I (again, see Lakens, 2014) explain why researchers analyzing large numbers of p-values in the scientific literature need to develop better models of p-value distributions before drawing conclusion about questionable research practices. I want to thank De Winter and Dodou for sharing their data, assisting in the re-analysis, and reading an earlier version of this draft (to which they replied they were happy to see other researchers used the data to test alternative explanations, and that they did not see any technical mistakes in this blog post).
TL;DR version: De Winter and Dodou (2015) analyzed the distribution (and its change over time) of a large number of p-values automatically extracted from abstracts in the scientific literature. They concluded there is a ‘surge of p-values between 0.041-0.049 in recent decades’ which 'suggests (but does not prove) questionable research practices have increased over the past 25 years'. I show the changes in the ratios of p-values over the years between 0.041-0.049 are better explained by a model of p-value distributions that assumes the average power has decreased over time. Furthermore, I propose that their observation that p-values just below 0.05 increase more strongly than p-values above 0.05 can be explained by an increase in publication bias over the years (cf. Fanelli, 2012), which has led to a relative decrease of 'marginally significant' p-values in the literature (instead of an increase in p-values just below 0.05). I (again, see Lakens, 2014) explain why researchers analyzing large numbers of p-values in the scientific literature need to develop better models of p-value distributions before drawing conclusion about questionable research practices. I want to thank De Winter and Dodou for sharing their data, assisting in the re-analysis, and reading an earlier version of this draft (to which they replied they were happy to see other researchers used the data to test alternative explanations, and that they did not see any technical mistakes in this blog post).
In recent
years researchers have become more aware of how flexibility during the
data-analysis can increase false positive results (e.g., Simmons, Nelson, &
Simonsohn, 2011). If the true Type 1 error rate is substantially inflated
because researchers analyze their data until a p-value smaller than 0.05 is observed, this might substantially
decrease the robustness of scientific knowledge. However, as Stroebe and Strack
(2014, p. 60) have pointed out: “Thus
far, however, no solid data exist on the prevalence of such research practices”.
Some researchers have attempted to provide some indication of the prevalence of
questionable research practices by analyzing the distribution of p-values in the published literature.
The idea is that questionable research practices lead to ‘a peculiar prevalence
of p-values just below 0.05’
(Masicampo & Lalande, 2012) or the observation that ‘”just significant”
results are on the rise’ (Leggett, Loetscher, & Nichols, 2013).
Despite the
attention grabbing titles of these publications, the reported data does not afford the strong conclusions these researchers have drawn. The
observed pattern of a peak of p-values
just below 0.05 in Leggett et al (2013) does not replicate in other collected p-value distributions for the same
journal in later years (Masicampo & Lalande, 2012), in psychology in
general (Kühberger, Fritz, & Scherndl, 2014), or in scientific journals in
general (De Winter & Dodou, 2015). The peak in p-values observed in Masicampo & Lalande (2012) is only
surprising compared to an incorrectly modelled p-value distribution that ignores publication bias and its effect
on the frequency of p-values above
0.05 (Lakens, 2014).
Recently, De
Winter and Dodou (2015) have contributed to this emerging literature on p-value distributions and concluded that
there is a ‘surge of p-values between
0.041-0.049 in recent decades’. They improved upon earlier approaches to
analyze p-value distributions by
comparing the percentage of p-values
over time (from 1990-2013). Two observations in the data they collected could
seduce researchers to draw conclusions about a rise of p-values just below a significance level of 0.05. The first
observation is a much stronger rise in p-values
between 0.041 and 0.049 than in p-values
between 0.051-0.059. The second observation is that the percentage of p-values that falls between 0.041-0.049
has increased more from 1990 to 2013 than the increase in the percentage of p-values between 0.01-0.09, 0.011-0.019,
0.021-0.029, and 0.031-0.039 over the same years (the authors also analyze p-values with 2 digits (e.g., p = 0.04), which reveal similar patterns, but here I focus on the three digit data, which included p-values between for example 0.041-0.049 because trailing zeroes (e.g., p = 0.040) are rarely reported). The authors (2015, p. 37) conclude
that: “The fact that p-values just
below 0.05 exhibited the fastest increase among all p-value ranges we searched for suggests (but does not prove) that
questionable research practices have increased over the past 25 years.”
I will
explain why the data does not provide any indication of an increase in
questionable research practices. First, I will discuss how the difference in
the increase in p-values just below
0.05 and just above 0.05 is due to publication bias, where (perhaps
surprisingly) p-values just above
0.05 are becoming relatively less likely to appear in the abstracts of
published articles over the years. Second, I will explain why the relatively
high increase in p-values between
0.041-0.049 over the years can easily be accounted for by a decrease in the
average power of studies, but is unlikely to emerge due to an inflated Type 1
error rate due to questionable research practices. I want to explicitly note
that it was possible to provide these alternative interpretations of the data
mainly because the authors shared all data and analysis scripts online (http://dx.doi.org/10.7717/peerj.733/supp-7)
and were furthermore extremely responsive and helpful in answering a number of
questions I had. While I criticize their interpretation of data, I applaud
their adherence to open science principles (their Matlab code is an excellent
example of reproducible statistics), which greatly facilitates cumulative
science.
As I have
discussed before (Lakens, 2014), it is essential to accurately model p-value distributions before drawing
conclusions about p-values extracted
from the scientific literature. Statements about p-value distributions require a definition of four parameters. First,
researchers should specify the number of studies where H0 is true, and the number
of studies where H1 is true. Second, researchers need to estimate the average power
of the studies (or the average power of multiple subsets of studies, if
heterogeneity in power is substantial). Third, the true Type 1 error rate and
any possible mechanisms through which the error rate is inflated should be
specified. And finally, publication bias, and a model of how the p-value distribution is affected by
publication bias, should be proposed. It is important to look beyond simplistic
comparisons between p-values just
below 0.05 and p-values in other
locations in the p-value distribution
outside the scope of an explicit model of the four parameters that determine p-value distributions.
Are p-values below 0.05 increasing, or p-values above 0.05 decreasing?
De Winter
and Dodou (2015) show there is a relatively stronger increase in p-values between 0.041-0.049 than
between 0.051-0.059 (see for example Figure 9, reproduced below). The data is
clear, but the reason for this difference is not. Are p-values below 0.05 increasing more, or are p-values above 0.05 increasing less? A direct comparison is
difficult, because the percentage of papers reporting p-values below 0.05 can increase due to an increase in p-hacking, but also due to an increase
in publication bias. If publication bias increases, and people report less
non-significant results, the percentage of papers reporting p-values smaller than 0.05 will also
increase, even if there is no increase in p-hacking.
Indeed, Fanelli (2012) has shown negative results have been disappearing from
the literature between 1990-2007, which would explain the relative differences
in p-values between 0.041-0.049 and
0.051-0.059 observed by De Winter and Dodou (2015).
We can
examine the alternative explanation that the relative differences observed are due
to publication bias increasing, instead of due to an increase in p-hacking, by comparing the relative
differences between p-values between
0.031-0.039 and 0.041-0.049 over the years on the one hand, and 0.051-0.059 and
0.061-0.069 on the other hand. If there is an increase in p-hacking, the biggest differences should be observed below 0.05
(in line with the idea of a surge of p-values
between 0.041-0.049.
However, there are reasons to assume the biggest difference might occur in p-values just above 0.05. As Lakens
(2014) noted, there seems to be some tolerance for p-values just above 0.05 to be published, as indicated by a higher
prevalence of p-values between
0.051-0.059 than would be expected based on the power of statistical tests and
an equal reduction of all p-values
above 0.05. If publication bias becomes more severe, we might expect a
reduction in the tolerance for p-values
just above 0.05, which would lead to the largest changes in ratios above 0.05. The spreadsheets and datafiles used to re-analyze and reconstruct the data is available on the OSF.
Across the
three time periods (1990-1997, 1998-2005, and 2006-2013) the ratio of p-values in the 0.03 range to p-values in the 0.04 range is pretty
stable: 1.13, 1.09, and 1.11, respectively. The ratio of p-values in the 0.05 range to p-values
in the 0.06 range is surprisingly large to begin with (given that purely based
on power, p-values between 0.051-0.059 and 0.061-0.069 should occur
approximately equally often in the literature), and shows a surprisingly large
reduction over the years: 2.27, 1.94, and 1.79, respectively. The only larger
reduction in ratios is observed for p-values
between 0.001-0.009 (which is most likely due to differences in power over the
years, as will be explained below). This surprisingly large change in ratios over
time for p-values between 0.051-0.059
indicates that instead of a surge of p-hacking,
publication bias has become more pronounced over the years for p-values just above the 0.05 level,
which causes p-values just above 0.05
to increase relatively less over the years than p-values in all other bins (except for p-values below 0.009).
This might be
explained by the idea that where p-values
between .051-0.59 (or 'marginally significnt' p-values) were more readily interpreted as support for the hypothesis
in 1990-1997 than in 2005-2013. This idea is
speculative, but seems likely given the increase in publication bias over the
years (Fanelli, 2012). It should be noted that p-values just above the 0.05 level are still more frequent than can be explained just by the average power of the
tests and publication bias that is equal for all p-values above 0.05 (cf. Lakens, 2014). In other words, this data
is in line with the idea that publication bias is still slightly less severe
for p-values just above 0.05, even
though this benefit of p-values just
above 0.05 has become smaller over the years.
This seems
to be the driving force for the differences between p-values in the 0.041-0.049 range and p-values
in the 0.051-0.059 range, reported by De Winter and Dodou (2015, e.g., Figures
9 and 10). To conclude, these observed differences provide no indication for a
surge of p-values between 0.041-0.049
over the years due to an increase in questionable research practices.
How changes in average
power over the years affect ratios of p-values
below 0.05
The title
of the article, “A surge of p-values
between 0.041-0.049” is based on the observation that the ratio of p-values between 0.041-0.049 increases
more than the ratio of p-values
between 0.031-0.039, 0.021-0.029, and 0.011-0.019. There are no statistics
reported to indicate whether these differences in ratios are statistically
significant, nor are effect sizes reported to indicate whether the differences
are practically significant (or justify the term ‘surge’), but the ratios do
increase as you move from bins of low p-values
between 0.001-0.009 to bins of high p-values
between 0.041-0.049. Figure 23 reports the ratios of percentages of p-values in 1990 and 2013 for a range of
search terms. Most interesting for the current purpose are the p-values between 0.001 and 0.049.
The first
thing to understand is why these ratios are not close to 1. The reason is that
there is a massive increase in the percentage of papers in which p-values are reported over the years. As
De Winter & Dodou (2015, p. 15) note: “In
1990, 0.019% of papers (106 out of 563,023 papers) reported a p-value between
0.051 and 0.059. This increased 3.6-fold to 0.067% (1,549 out of 2,317,062
papers) in 2013. Positive results increased 10.3-fold in the same period: from
0.030% (171 out of 563,023 papers) in 1990 to 0.314% (7,266 out of 2,317,062
papers) in 2013.” This is not just an increase in the absolute number of
reported p-values in abstracts (in
which case the ratios could still be 1) but a relative 10.3-fold increase in
how often p-values end up in
abstracts. De Winter & Dodou (2015) demonstrate p-values are finding their way into more and more abstracts, which
points to a possible increase in the overreliance on null-hypothesis testing in
empirical articles. This is an important contribution to the literature, even
when other claims about an increase in questionable research practices would
not hold (also, the huge increase in the term 'paradigm shift' in abstracts over time is quite telling).
How can
these differences between the ratios across the 5 bins below 0.05 be explained
by a model of p-value distributions
that consists of the ratio of true to false effects examined, power, the Type 1
error rate, and publication bias? We can only explain the relative differences
between the ratios over the different bins of p-values if we allow at least one of the parameters of the model to
the change over time. We can ignore publication bias, assuming all disciplines
that report p-values in abstracts use
α = 0.05 (this is not true, but we can assume it applies to the majority of
articles that are analyzed). The two remaining possibilities are a change in
the average power of studies over time, and an inflated Type 1 error rate over
time, such as an increase in questionable research practices in the
literature.
If we
ignore Type 1 errors, we can relatively easily reconstruct the observed data
purely based on differences in the average power across the years. I’m not
arguing the numbers in this re-construction reflect the truth. However, they show
it is possible to model the ratios observed by De Winter & Dodou (2015) under
the assumption that power differs from 1990 to 2013. For example, if we assume
average power was 55% in 1990, and 42% in 2013, we can expect to observe the p-value distribution across the 5 bins
as detailed in the table below, with 29.855% of the p-values falling between 0.001 and 0.009 in 1990, but only 19.926%
of p-values falling between
0.001-0.009 n 2013 (which most likely explains the large differences in ratios
between 0.001-0.009 discussed earlier). This is just the p-value distribution as a function of the power of the tests.
Table 1:
Expected percentage of p-values
between 0.001-0.0049 page on 42% and 55% power.
1990 55% power
|
2013 42% power
|
|
p0.001-p0.009
|
0.299
|
0.199
|
p0.011-p0.019
|
0.085
|
0.072
|
p0.021-p0.029
|
0.056
|
0.051
|
p0.031-p0.039
|
0.042
|
0.040
|
p0.041-p0.049
|
0.034
|
0.033
|
If we
incorporate the fact that the percentage of p-values
reported in the abstract has increased by 10% over the years (column 2 and 3 in
Table 2 below), and use as total studies in 1990 563023, and as total studies
in 2013 2317062 (taken from De Winter & Dodou, 2015) then we should expect
the total number of observed p-values
in 1990 and 2013 as displayed in columns 4 and 5 below. These numbers mirror
the observed frequencies (columns 4 and 6) by De Winter and Dodou (2015).
Table 2. Absolute
number of reconstructed and observed p-values
between 0.001-0.049 from 1990 to 2013.
% p-values in
abstract
|
% p-values in
abstract
|
reconstructed p-values
1990
|
reconstructed p-values
2013
|
observed p-values
1990
|
observed p-values
2013
|
|
p0.001-p0.009
|
0.01
|
0.1
|
1681
|
46170
|
1770
|
44970
|
p0.011-p0.019
|
0.01
|
0.1
|
481
|
16728
|
462
|
14885
|
p0.021-p0.029
|
0.01
|
0.1
|
316
|
11725
|
268
|
10630
|
p0.031-p0.039
|
0.01
|
0.1
|
238
|
9210
|
240
|
9108
|
p0.041-p0.049
|
0.01
|
0.1
|
191
|
7646
|
178
|
8250
|
When we
calculate the ratios of the observed p-values,
we see in Table 3 they approach the general pattern of the ratios observed by
De Winter and Dodou (2015). The reconstruction is not perfect, for a number of
reasons. First of all, there is very little data from 1990, which will lead to
substantial variation between expected and observed frequencies for any model
(the fit of the model increases for comparisons between years where there is
more data available). For example, the fact that the difference in the
percentage of p-values in the
0.021-0.029 bin from 1990 to 2013 is larger than for p-values in the 0.031-0.039 bin is only true in 1990 and 2008, but
is reversed (as predicted by a model of p-value distributions where power
changes over time) in the remaining 21 comparisons of 2013 with each preceding
year.
Table 3.
Ratios of reconstructed and observed p-values
between 0.001-0.049 from 1990 to 2013.
reconstructed ratio N/T 1990
|
reconstructed ratio N/T 2013
|
reconstructed 1990/2013 Ratio
|
observed ratio N/T 1990
|
observed ratio N/T 2013
|
observed 1990/2013 Ratio
|
|
p0.001-p0.009
|
0.306
|
1.993
|
6.674
|
0.315
|
1.945
|
6.17
|
p0.011-p0.019
|
0.085
|
0.722
|
8.454
|
0.082
|
0.644
|
7.83
|
p0.021-p0.029
|
0.056
|
0.506
|
9.017
|
0.048
|
0.460
|
9.63
|
p0.031-p0.039
|
0.042
|
0.398
|
9.417
|
0.043
|
0.394
|
9.21
|
p0.041-p0.049
|
0.034
|
0.330
|
9.740
|
0.032
|
0.367
|
11.28
|
Similarly,
when comparing 2013 to each of the 23 preceding years, the ratio is higher for p-values between 0.041-0.049 than for
0.031-0.039 in 12 out of 23 comparisons – only just more than 50% of the time,
which can hardly be called a ‘surge’. The model based on power differences
predicts that ratios for p-values
between 0.031-0.039 should be very similar to those between 0.041-0.049. Given
the small percentages of articles that report p-values and the variation inherent in observed p-value distributions, it is not
surprising the ratios for 0.041-0.049 are only just more than 50% likely to be
higher than those for p-values
between 0.031-0.039. This observation is more difficult to explain based on the
idea that questionable research practices have increased, which typically
assumes p-values between 0.041-0.049
increase more strongly than p-values
between 0.031-0.039 (e.g., Leggett et al., 2013; Masicampo & Lalande,
2012).
Obviously
this model is too simplistic. It does not include any Type 1 errors, and it
assumes homogeneity in the power of the performed tests. We can be certain
power varies substantially across studies and research disciplines, and we can
be certain there are a number of Type 1 errors in the literature. For the
current purpose, which is to demonstrate the observed pattern can be
reconstructed by assuming the average power has changed over time, a more advanced
model is not required, but future attempts to provide support for an increase
in Type 1 errors, or attempts to calculate average effect sizes based on p-value distributions (e.g., Simonsohn,
Nelson, & Simmons, 2014) need to develop more detailed models of p-value distributions.
Let’s assume
the average power has not changed over time, and try to reconstruct the
observed ratios by changing the Type 1 error rates. As long as the Type 1 error
rates are the same for each bin of p-values,
the ratios equal the overall increase in p-values
reported in abstracts over time. To reconstruct the ratios as observed by De
Winter and Dodou (2015), we need to assume p-hacking
leads to a stronger increase in higher p-values
than in lower p-values. Although this
is a reasonable assumption under many types of p-hacking, it turns out to that the specific pattern of inflated
Type 1 error rates required to reconstruct the observed ratios in not very
likely to emerge in real life.
To simulate
the impact of questionable research practices, we need to decide upon the ratio
of studies where H0 is true and studies where H1 is true, and the exact
increase in Type 1 error rates for each bin of p-values below 0.05. Type 1 errors come exclusively from analyzing
results of studies where H0 is true (p-hacking
when H1 is true inflates the effect size estimate, and thus can be seen as an
incorrect way to increase the power of a test). In the calculations below,
power is kept constant, but p-hacking
is introduced. This is the equivalent of the true power of studies reducing
over the years, which is exactly compensated by an inflated Type 1 error rate.
The
observed ratios by De Winter & Dodou (2015) show the ratio is the smallest
for p-values between 0.001-0.009, and
substantially higher for p-values
between 0.011 and 0.049, with a relatively small increase in these 4 bins. This
pattern can be reproduced just based on inflated Type 1 errors, but the required
increase in Type 1 error rates over the 5 bins is very unlikely to occur when p-hacking.
The higher
the average power of statistical tests, the more frequently small p-values will be observed if there is a
true effect. This means there are more p-values
between 0.021-0.029 than between 0.041-0.049 whenever the power is larger than
0. Without p-hacking, the number of
Type 1 errors in each bin (e.g., between 0.001 and 0.009) should be 0.8% (it is
1% between 0 and 0.01). If we assume this was the situation in 1990 (which is a
conservative, albeit unlikely, estimate), the Type 1 error rates need to be
increased to higher levels to reproduce the observed ratios, after selecting
the average power of the studies, and the ratio of studies where H0 is true and
H1 is true. It becomes extremely difficult to reconstruct the observed absolute
numbers and ratios.
One attempt
to model to reconstruct the ratios (but not the absolute values) is presented
in Table 4. The ratio of studies where H0 is true to studies where H1 is true
is set to 1, and the average power is assumed to be 57.5%. The Type 1 error
rate inflation over time is substantial, and the difference in the increase
over the bins is not very typical, with a practically equal increase between
0.021-0.049. To achieve the ratios observed by De Winter & Dodou (2015) for
comparisons between 2013 and later years than 1990, the Type 1 error rate even needs
to be inflated more strongly for p-values
between 0.021-0.029 than for p-values
between 0.041-0.049. Such a pattern of Type 1 error rate inflation is
practically difficult to achieve, because questionable research practices (such
as performing multiple analyses on the same data with different outlier
criteria) produce a p-value
distribution where higher p-values
are observed more frequently than smaller p-values.
Thus, although it is not impossible to achieve the observed ratios purely by p-hacking (although it is very
challenging to reconstruct both ratio’s and absolute numbers), the required Type
1 error rate inflation over the 5 bins of p-values
is unlikely to occur in real life.
Table 4. Absolute
number of reconstructed Type 1 errors between 0.001-0.049 from 1990 to 2013.
1990 true effects
|
2013 true effects
|
Type 1 error rate 1990
|
1990 Type 1 errors
|
Type 1 error rate 2013s
|
2013 Type 1 errors
|
Reconstructed 1990/2013 Ratio
|
|
p0.001-p0.009
|
1814
|
47784
|
0.008
|
90
|
0.015
|
4449
|
6.66
|
p0.011-p0.019
|
492
|
12959
|
0.008
|
90
|
0.020
|
5932
|
7.89
|
p0.021-p0.029
|
319
|
8399
|
0.008
|
90
|
0.025
|
7415
|
9.40
|
p0.031-p0.039
|
238
|
6260
|
0.008
|
90
|
0.025
|
7415
|
10.14
|
p0.041-p0.049
|
189
|
4988
|
0.008
|
90
|
0.027
|
8008
|
11.30
|
To
summarize, we can easily reconstruct the observed ratios by assuming a
relatively small decrease in power over the years (e.g., from 55% to 42%). Such
an assumption could be reasonable, as long as new research areas, or strongly
growing research areas, have lower power than average. One example of such a
research area is neuroscience, with a median power estimated to be as low as
21% (Button et al., 2013). On the other hand, while increases in Type 1 error
rates can be used to reconstruct the observed ratios, the pattern of inflated
Type 1 errors across the 5 bins of p-values
is unlikely to emerge in real life.
Therefore,
I conclude it is not true that there is a ‘surge of p-values between 0.041-0.049’, nor that these data suggest there is
an increase in questionable research practices over the last 25 years. The search for evidence of an increase in questionable
research practices is starting to mirror the search for the ether. After
repeatedly claiming to observe a rise in p-values
just below 0.05 without providing substantial evidence for such a rise (De
Winter & Dodou, 2015; Leggett et al., 2013; Masicampo & LaLande, 2012),
it is time researchers investigating inflated Type 1 errors use better models,
make better predictions, and collect better data. Analyzing huge numbers of p-values, which come from studies with
huge heterogeneity, will not be able to provide any indication of the
prevalence of questionable research practices, not even when changes of p-value
distributions are analyzed over time. All these papers are evidence of is a
peculiar prevalence of incorrect conclusions about p-value distributions.
References
- Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365-376.
- Kühberger, A., Fritz, A., & Scherndl, T. (2014). Publication bias in psychology: A diagnosis based on the correlation between effect size and sample size. PLoS ONE 9(9): e105825. doi:10.1371/journal.pone.0105825
- Leggett, N. C., Thomas, N. A., Loetscher, T., & Nicholls, M. E. (2013). The life of p: “Just significant” results are on the rise. The Quarterly Journal of Experimental Psychology, 66, 2303-2309. doi: 10.1080/17470218.2013.863371
- Lakens, D. (2014). What p-hacking really looks like: A comment on Masicampo and LaLande (2012). The Quarterly Journal of Experimental Psychology, (ahead-of-print), 1-4.
- Masicampo, E. J., & Lalande, D. R. (2012). A peculiar prevalence of p values just below. 05. The Quarterly Journal of Experimental Psychology, 65(11), 2271-2279.
- Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve and effect size: correcting for publication bias using only significant results. Perspectives on Psychological Science, 9(6), 666-681.
- de Winter, J. C., & Dodou, D. (2015). A surge of p-values between 0.041 and 0.049 in recent decades (but negative results are increasing rapidly too). PeerJ,3, e733.
