tag:blogger.com,1999:blog-987850932434001559.post2706847314662877878..comments2021-09-22T08:59:18.879+02:00Comments on The 20% Statistician: What p-hacking really looks like: A comment on Masicampo & LaLande (2012)Daniel Lakenshttp://www.blogger.com/profile/18143834258497875354noreply@blogger.comBlogger8125tag:blogger.com,1999:blog-987850932434001559.post-77426187198278461582017-02-21T18:46:05.369+01:002017-02-21T18:46:05.369+01:00I sat in my lounge room in Australia watching a do...I sat in my lounge room in Australia watching a documentary that was telling the story of a young Rohingya girl. https://www.fiverr.com/sabakhan695<br />john smithhttps://www.blogger.com/profile/07655303417027257101noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-32423520954576373102014-10-17T14:27:54.224+02:002014-10-17T14:27:54.224+02:00Thanks for the reference to Leggett et al. 2013, i...Thanks for the reference to Leggett et al. 2013, it's a nice one!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-60873686892301103512014-10-17T06:28:32.549+02:002014-10-17T06:28:32.549+02:00The p-values were re-calculated from the test scor...The p-values were re-calculated from the test scores. People are not supposed to round a p-value of 0.054 down to p = .05 (but they do, see http://www.tandfonline.com/doi/abs/10.1080/17470218.2013.863371)Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-18064522159307334752014-10-16T17:00:34.298+02:002014-10-16T17:00:34.298+02:00In sociology, p-values are almost always presented...In sociology, p-values are almost always presented to two decimal places. Thus p = .05 covers the range .045 - .055. Yet we see in published articles that every instance of p = .05 is accompanied by an asterisk to denote p < .05! Roughly half of results where p (rounded to 2d.p.) = .05 are actually *not* statistically significant at the .05 level. This anomaly--which shows in a trivial way how much gamesmanship is involved in statistical analysis--should be easily proved by looking at a year's worth of articles in a top journal.<br /><br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-24315691674503470212014-09-30T19:28:34.964+02:002014-09-30T19:28:34.964+02:00Hi Joost - yes, testing after every participant ca...Hi Joost - yes, testing after every participant can really inflate the Type 1 error rate and create a peak. And if you assume 5000 studies do that, and only 1000 studies examine true effects, you can get a peak. But the observed data (by M&L and Kuhberger in my previous post) don't show such a peak just below .05. The question whether it is possible is less relevant than the question whether it happened. <br /><br />About your paper: I have not looked at it carefully. If you want to do these analyses right, you should at least find a clear skewed p-value distribution, and only above this, and increase in a specific range over time, compared to other times. I don't know if your commentary is showing this - the ratio measure is very confusing. In absolute numbers, I think I see a p-curve that is as you'd expect. The ratio measure you focus on looks like it has a high risk of some artefact. But I'd have to look at it more carefully.Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-34758148409610484352014-09-30T13:51:05.662+02:002014-09-30T13:51:05.662+02:00Thanks for this valuable post!
Two discussion ite...Thanks for this valuable post!<br /><br />Two discussion items:<br />1) I tried to repeat your simulations of Figure 1, right pane (in Matlab) and found that (logically) a steeper increase occurs across the range from 0 to 0.05 when not adding 10 participants each try, but when adding only 1 participant (e.g., as in Simmons, Nelson, & Simonsohn., 2011). Equivalently, increasing the sample size from 50 to 500 (but keeping 10 additions each try) also yields a sharp peak for p values just below 0.05. Trying more than 5 times also increases the peak of p-values just below 0.05. In other words, there might be p-hacking mechanisms which DO yield a preponderance of p-values just below 0.05.<br /><br />2) Another way to identify p-hacking may be to look at longitudinal trends. For example, we found that p-values in the 0.041-0.049 range increased more steeply in the past 20 years than p-values in other ranges. I wonder what your opinion is on thisâ€¦ https://sites.google.com/site/jcfdewinter/DeRuiter_reply.pdf?attredirects=0<br /><br />Kind regardsJoost de Winternoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-68867971878934287072014-09-30T13:50:44.307+02:002014-09-30T13:50:44.307+02:00I couldn't agree more. This is now the third b...I couldn't agree more. This is now the third blog post in which I take a close look at results by other researchers, and every time people have been extremely helpful sharing data, insights, and comments. The scientific spirit is strong with this one!Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-44745890845451079512014-09-30T12:08:13.822+02:002014-09-30T12:08:13.822+02:00Big, big kudos to Masicampo and Lalande for /a/ sh...Big, big kudos to Masicampo and Lalande for /a/ sharing their data with you so readily, and /b/ allowing everyone else to see it. :-)Nick Brownhttps://www.blogger.com/profile/18266307287741345798noreply@blogger.com