tag:blogger.com,1999:blog-987850932434001559.post3091831764030077273..comments2024-03-29T11:00:11.612+01:00Comments on The 20% Statistician: Not All Flexibility P-Hacking Is, Young PadawanDaniel Lakenshttp://www.blogger.com/profile/18143834258497875354noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-987850932434001559.post-46838235065593268132021-11-01T08:44:06.109+01:002021-11-01T08:44:06.109+01:00Your first part about sequential analysis is not r...Your first part about sequential analysis is not really a solid analysis - the problems you mentioned are all easily solved, see https://psyarxiv.com/x4azm/. <br /><br />About the second part: Preregistration is more than just being 'open' about a process. It is about allowing others to evaluate the severity of a test. This means you need to provide very specific information in the preregistration - a problem is many are now too vague to evaluate the severity of a test. Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-87563481308494467512021-10-31T16:51:42.298+01:002021-10-31T16:51:42.298+01:00This is all fine except for the advice on testing ...This is all fine except for the advice on testing - running - testing cycle. While it's true that you can plan for this and avoid inflated Type I error rates, it still has problems. The first is that, in the long run, it is not more efficient. Sometimes you'll need to run more participants and sometimes fewer; but it doesn't make things more efficient in the long run. The second is that it generates a literature where all of the small studies have exaggerated effect sizes and the large ones underestimated effect sizes. Consider the situation, you're going to run until you find an effect. If your initial sample was an underestimates of that effect, even in the wrong direction, you'll need to run many participants in order to eventually find an effect, and the under estimate bias will never be eliminated. If you start with an over estimate in your initial sample you'll be done collecting data quickly and, again, not have eliminated the over estimation bias in your sample. <br /><br />Every other bit of overregularization mentioned here is spot on though. I especially often run into the preregistration issue. I never explain it to my students as a way to avoid Type I errors. I only describe it as a way to be open about your process. With that mindset, of doing open science, they don't worry about being able to solve every analysis problem prior to do it.Unknownhttps://www.blogger.com/profile/00227235335343168838noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-84388695405938253282021-10-31T11:52:58.997+01:002021-10-31T11:52:58.997+01:00Thanks for clarifying the issues around when p-hac...Thanks for clarifying the issues around when p-hacking is/is not p-hacking Daniël. I'm now considering how your take applies to our entry in the Catalogue of Bias: https://catalogofbias.org/biases/data-dredging-bias/<br /><br />I wonder if you'd be kind enough to provide your view of our entry and where, if any, edits can be made to improve the accuracy of its content?<br /><br />DavidAnonymoushttps://www.blogger.com/profile/06349977360386625986noreply@blogger.com