New paper: The benefits of preregistration and Registered Reports.

 With my PhD students Cristian Mesquida and Sajedeh Rasti, and former lab visitor Max Ditroilo we published a new paper on preregistration and Registered Reports. We aim to provide a state-of-the-art overview of the idea behind and metascience on preregistration and Registered Reports. https://www.tandfonline.com/doi/full/10.1080/2833373X.2024.2376046

We explain the link between preregistration and severe testing, and how systematic bias might reduce the severity of tests. Preregistration is a tool to allow others to evaluate the severity of tests.

We provide and defend a more narrow use-case of preregistration. In essence, we argue you can only preregister level 6 and 5 studies from this table from the Peer Community In guide for authors https://rr.peercommunityin.org/help/guide_for_authors

We deviate from current consensus, but in the conviction that our use of the term preregistration is more principled, and will become the default in the future (just as how the Preregistration+ badge would be seen as the only valid preregistration today. As our understanding changes, so do our definitions.

We summarize 18 surveys on research practices that reduce the severity of tests. You might have seen previous version of this Figure – this is the final published version, in case you want to re-use or cite this. More details on the studies in this figure is available from https://osf.io/sxg7q. 

We carefully point out: “It is important to point out that the percentages presented here do not directly translate into the percentage of researchers who are engaging in these practices.” We wish we knew, but we just don’t know. 

We discuss cost-benefit analyses of preregistration, and conclude there are too many unknowns to determine if preregistration is beneficial. We also say it does not really matter, because the main reason to preregister is based on a normative argument.

We say: “researchers who test hypotheses from a methodological falsificationist approach to science should preregister their studies if they want a science that has intersubjectively established severely tested claims.” As always, we believe it is essential to be clear about your philosophy on scientific knowledge generation - not being clear about it can lead to a lot of discussion that will go nowhere (see Lakens, 2019).  

That means we also do not expect people who have different epistemological philosophies to preregister – nor is it a logical solution for exploratory research, or certain types of secondary data analysis. We feel it is important to point this out, because there are alternative approaches to argue a test is severe that are better suited for those studies: open lab notebooks, sensitivity analyses, robustness checks, independent replication. It is always important to use the right tool for the job - we do not want preregistration to be mindlessly overused. 

We discuss meta-scientific evidence that shows preregistration makes it possible to evaluate the severity of tests (and we cite some anecdotal examples). Of course, not all preregistrations are equally good yet – people need more training. 

We also engage with the most important criticism on preregistration. Beyond the valid concern that the mere presence of a preregistration may be mindlessly used as a proxy for high quality, we identify conflicting viewpoints, several misunderstandings, and a general lack of empirical support for the criticisms that have been raised. I personally feel critics need to raise the bar if they want to be taken seriously. They should at the very least resolve the contradicting criticisms among each other. They should also collect empirical data to test their claims. 

I strongly expect this fourth paper (following Nosek & Lakens, 2014, Lakens, 2019, and Lakens, 2024) to be my last contribution to this topic. I have said all I want to say, and contributed all I can with this final paper.