Which Data Repository Should You Use?

The Center for Open Science has announced that from November 16, 2026, no new projects can be created on the Open Science Framework. After February 19 all projects will become read-only, and eventually, they anticipate that all private projects will be deleted. The OSF has raised a generation of scientists that is able to practice open science. These scientists will need to take these skills to a new data repository in the future.

 

Choosing a New Data Repository

 

There are many data repositories, each with their own strengths and weaknesses. Before I compare them, I want to make it clear that the fact that organizations are willing to store your data for free for a very long time is an amazing service. You should be incredibly grateful that any and all of these services exist. As someone who had a free Hotmail email account in 1997 with 2 mb of free storage, the fact that we can now store up to 50GB for every project we create in a data repository blows my mind.

Some common data repositories used in psychology are Zenodo, GitHub, ResearchBox, Dataverse, and PsyArchives. I will compare these services on properties that I think are important.

1. Longevity

We are all grains of sand in the desert of eternity. Nothing lasts forever. But it would be nice if our data repositories will keep going for a while. Zenodo is hosted by CERN and commissioned by the European Commission through the OpenAIRE project. CERN has been around since 1954, and it will be around well until anyone who will ever read this is dead. GitHub is too big to fail, and was bought by Microsoft in 2018 for 7.5 US billion. It will be around for a while. PsychArchives is hosted by ZPID, which started in 1971, and which I expect to exist many years from now. The first Dataverse repository was hosted by Harvard in 2006. Dataverse is not a single repository. For example, Harvard Dataverse and DataverseNL are different instantiations of the service. The Open Science Framework started in 2013 and is hosted by the Center for Open Science. ResearchBox started in 2020, and is hosted by the Wharton Credibility Lab, and funded by the Wharton School of the University of Pennsylvania.

Predicting the future is notoriously difficult. I think the order in which I discussed the platforms is also the order in which I would rank their longevity, but I would not dare to give a number on how long each service will last. As the team behind ResearchBox explains, they set up the system to minimize misuse, but their costs will continue to grow, especially if more researchers start to use the platform. I would guess that for the next decade, all these services are likely to continue to exist, and the ones earlier in the list will continue to exist for a long time. Note that data hosted on these platforms will continue to exist in any case. ResearchBox uses Zenodo as one of its long term back-up solutions.

2. API

Although you might think that data re-use will mean that a person will browse through your data, data and code sharing is done as much for re-use by people, as re-use by machines. To enable re-use by machines, platforms need a good API (Application Programming Interface). GitHub, Zenodo, and Dataverse have good APIs. The OSF has a usable API, but it is not well-designed, and requires some effort to use. PsychArchives has an API, but it is hidden – it is not documented, so if you want to use it you have to discover it yourself. This is suboptimal. There is an additional challenge for PsychArchives: While public files on PsychArchives can be downloaded automatically, files shared under more restrictive licenses can be downloaded manually, but not by software.

ResearchBox has no API. This is the biggest weakness of ResearchBox, and it is serious enough such that it prevents me from recommending ResearchBox as the main alternative for people looking for a new data repository. ResearchBox can make an API, and if it does (and it does it well) the platform is an excellent alternative to the OSF. But currently, software can’t automatically interact with (e.g., download from or upload to) ResearchBox, and this is a problem for re-use. It is possible to scrape the ResearchBox website, but this breaks too easily and is not a long-term solution.

Dataverse is open-source repository software rather than a single service: it is run independently by many institutions (e.g., Harvard Dataverse, DataverseNL, and various university-hosted installations), each on its own domain. They all have the same API, but a tool first needs to recognize which of the many independent Dataverse installations a given link belongs to. It is possible to work around this if one has a list of all Dataverse repositories, but it adds complexity that is not present for all other services.

3. Metadata

Metadata might be a bit boring, but it is how re-users will find your data and code when they are not approaching it through a link in a paper. That makes it very important (but boring).

On the OSF, a project (or component) carries a title, a description, a controlled `category` (e.g., project, data, hypothesis, methods and measures, software), an optional custom citation, free-form tags, a license, a discipline classification from OSF's own subject taxonomy, and contributors. This is rather limited as we will see compared to other repositories. When it comes to metadata, there is a lot to gain by using other repositories.

Zenodo's metadata model is richer and more structured for citation purposes: alongside title, description, and creators (name, affiliation, ORCID, GND), a deposit requires an `upload_type` (e.g., publication, dataset, software, poster) and, for publications, a `publication_type` (e.g., journal article, preprint, report, thesis). It also supports free-form keywords, and a controlled `access_right` (open, embargoed, restricted, closed) with an associated license or embargo date. It also has `related_identifiers` linking the deposit to other DOIs with a typed relationship (e.g., isSupplementTo, cites, isNewVersionOf). You are not as excited about this as you should be. Links between files matter a lot in understanding what files are, and how they can be re-used. There are also `contributors` (e.g., DataCurator, Supervisor, RightsHolder) distinct from creators, and `grants`, which link the deposit to a funder-registered grant ID so it appears in that funder's own reporting.

Dataverse is even better when it comes to metadata, and it places strong emphasis on structured, repository-level metadata that supports discovery, citation, and interoperability. Each dataset is described through a citation metadata block containing fields such as title, author, contact, description, subject classification, keywords, producer, distributor, publication date, and related publications. Dataverse repositories can also enable domain-specific metadata schemas, including social science, geospatial, life science, and astronomy extensions, allowing much richer description than a generic file upload. In addition, datasets receive persistent identifiers (typically DOIs), versioning metadata, and machine-readable metadata exports in standards such as Dublin Core and DataCite, facilitating integration with external discovery services and repository networks.

PsychArchives follows a metadata model designed around the needs of psychological science and the FAIR principles. Deposits are assigned DOIs and can represent a wide variety of digital research objects, including articles, preprints, datasets, code, preregistrations, tests, and multimedia content. Metadata record fields support detailed descriptions of authorship, affiliations, object type, publication status, licensing, keywords, and links between related research outputs from the same study. Because PsychArchives is embedded in the ZPID ecosystem, metadata are also optimized for indexing by services such as PsychPorta, Google Scholar, and other scholarly discovery systems. The repository's emphasis is not only on describing objects individually, but also on connecting related outputs into a coherent research record. Once again, you are not as excited about this as you should be, but knowing a dataset belongs to a supplement that belongs to a paper is something we all want even if we do not realize it.

ResearchBox takes a somewhat different approach to metadata, prioritizing usability and transparency of research materials by people who browse the website over highly formalized archival description. Public deposits are organized into standardized "Bingo Tables" that indicate which research components are available, such as preregistrations, materials, datasets, code, and manuscripts. ResearchBox automatically generates dataset codebooks and supports searching within datasets, variable descriptions, and code files, making metadata partly file-derived rather than solely author-supplied (this is very good, and something other repositories can learn from – I strongly believe that if we ever have FAIR metadata, it is because most of it is automatically generated). As there is no API, this metadata can only be searched manually on the website, which is the biggest limitation of ResearchBox.

4. Anonymous Peer Review

Many journals still value and require anonymous peer review. PsychArchives, ResearchBox, and the OSF make it possible to share an anonymous link to the datafiles for peer review. GitHub repositories can be made anonymous for blind peer review through https://anonymous.4open.science. Zenodo allows users to create a new upload in Zenodo, keep it private, fill in the metadata while not disclosing any personal information, and without publishing the record, click Share, Links, and create a secret link with the permission that people with the link "Can preview draft". This allows reviewers to access the unpublished draft, while the files remain invisible to the public. Dataverse offers a similar but more automated solution: from an unpublished draft dataset, selecting Edit, then Preview URL, offers a choice between a "General Preview URL" (a secret link giving access to the draft, comparable to Zenodo's) and an "Anonymous Preview URL". The anonymous option automatically strips author names and other identifying information from the citation, version history, and metadata fields before showing the page to anyone holding the link, rather than relying on the depositor to avoid entering identifying information in the first place.

All platforms allow anonymous access to files, but ResearchBox, OSF, and PsychArchives make it really easy. Dataverse accommodates it well, it takes some getting used to on Zenodo, and it is hacky but possible on GitHub.

5. Usability

When it comes to data sharing platforms, they need to excel at two things. First, it should be easy to upload and share data, and second, others should be able to find and download the data. One of the continuing complaints about the Open Science Framework was that the user experience got in the way of data sharing and re-use. When it comes to sharing your data, ResearchBox and PsychArchives offer a much smoother experience for users who want to manually upload files, and especially ResearchBox cares a lot about ease of use. Dataverse and Zenodo are also quite intuitive, but a bit more bare-bones and it has some peculiarities we will need to get used to. The biggest is probably that Zenodo does not support folders – all files are uploaded in a flat structure – so users need to learn how to use file names or zip files to organize the files they upload. Using GitHub probably requires a short course, and the biggest difference with other platforms is that you are not supposed to interact with it directly. Instead, you use software (or, if you are a hardcore programmer, the terminal) to push and pull files from the platform.

When it comes to finding files, ResearchBox has the most useful search, as it allows you to search not just for filenames, but also inside data and code files. However, the lack of an API means you can only perform these searches manually, and machines can’t access the file content. This is a major limitation. Dataverse also allows you to search inside tabular data. This means you can search for the word ‘PANAS’ and try to find datasets that have data from the positive and negative affect schedule measure. Dataverse can be searched through an API. All other platforms only allow users to search variable names and metadata.

Personally, although I appreciate ‘first-order’ direct manual usability of a platform, I care a lot more about ‘second-order’ indirect programmatic access to a platform. We will always have different data repositories. If I want to search for datasets that contain the ‘PANAS’ I do not expect that I will visit all repositories one by one, but I will use a search engine that covers all existing databases. Indeed, I am pretty sure that these search databases will have downloaded and processed all files on data platforms using algorithms that extract all data they can from the files. Searches will in the future be performed on a custom database. I also think that we will rely less on the user interface of a website. One thing my master students have recently taught me is that nowadays you ask an AI to create the user interface you want, instead of manually using a website. An example of this for the OSF is BetterOSF by Gilad Feldman. Making a custom interface on top of an existing data repository (or on top of multiple data repositories) is something AI can code for you relatively easily, but it requires the data repository to have a good API.

6. GDPR

As a scientist in the EU, I have to follow the General Data Protection Regulation (GDPR). For fully anonymized data, the GDPR does not apply, and I can store the data anywhere in the world. However, most of my data is pseudonymous. There is a participant ID in the data I share that links the participant to a datafile that is not public, but that is stored in the data archive our department uses to store research data. Even if the shared data has no personally identifiable information, the fact that it is possible to link it to an email address or IP address through the participant number makes the data pseudonymized, and I need to follow the GDPR.

As far as I understand the GDPR, it suggests (as they say, #NotALawyer) that there are routes to sharing pseudonymized data outside of the EU, but either the data repository needs to be Data Privacy Framework certified, or there should be a Standard Contractual Clauses agreement, or I would need to document the procedure for pseudonymization, and formally ask for approval from national supervisory authority, and inform participants about it. Sharing anonymized data on Harvard Dataverse or ResearchBox is easy – sharing pseudonymized data is not easy.

Zenodo is an interesting case, and I learned something new and important while preparing this blog. Cern is located in Geneva, so you might think this makes data storage on Zenodo simple from a GDPR standpoint. But Cern is an international organization with special legal status under international law. It is not subject to the GDPR. CERN instead processes personal data under its own internal framework, which it and Zenodo describe as built to the same standard as GDPR, without being a legal GDPR obligation. This means it is legally not possible to share pseudonymized data with them under the GDPR. They explicitly state they can’t sign data processing agreements. They explicitly say other repositories might be better for data that originated from or contains/ed sensitive personal data. Even though I was not aware of this, and I think I have often heard it said the Zenodo is GDPR compliant, it can’t be, as the GDPR does not apply. This is a weird legal loophole that applies to all International Organizations (this is an interesting discussion).

The easiest way to comply with the GDPR for pseudonymized data is to use PsychArchives or DataverseNL (squarely GDPR compliant, in the EU) – or other local repositories you might have access to (for my own universities, there is the 4TU.Research.Data repository). The downside of this is that research data might become distributed over a large number of local repositories, each with their own rules and standards, which will make it more difficult to automatically find and integrate all data in a field.

Conclusion

We live in a time where you get to pick which service to use to freely store your research data. This is truly amazing, and I am immensely grateful to all the people who have built and maintained these infrastructures. It also means there is something to pick, and that platforms can learn from each other. This is also why ResearchBox gets so many things right – as the newest addition to the data repository landscape, it could learn from and improve on existing repositories. However, they really need an API. Without one, it is difficult for me to recommend the platform, even though it scores at least as well as other platforms on the other dimensions. PsychArchives similarly needs to document their API, and allow programmatic access to files shared under restrictions (some files are publicly available under license restrictions, but can only be downloaded after manually clicking a box).

Dataverse and PsychArchives support metadata in the most systematic manner. Although very few people want to learn about DublinCore metadata standards, people who create data repositories should, and repositories that support these metadata standards will turn out to be more useful a decade from now than platforms that do not. However, as I personally prefer metadata to live as a standalone .JSON file that is shared alongside data (similar to how we now recommend a readme file) I think it is not that problematic if repositories do not follow a metadata standard.

The best choice for a repository also depends on your computer skills. If you interact with the internet by pointing and clicking, you will feel more comfortable on ResearchBox and PsychArchives than on GitHub. But if you interact with the internet through R or Python, you will need to use Zenodo, Dataverse, or GitHub.

If you have pseudonymized data, formally Zenodo can’t be GDPR compliant (you either make data anonymous, or switch to another repository). Either make the data fully anonymous (e.g., delete any identifying information, but this makes data provenance more difficult, and is undesirable) or use GDPR compliant data repositories (e.g., DataverseNL, PsychArchives).

It is clear to me that everyone will need to make their own decisions about the repository that is the best fit for their use-case. In my case, I will likely use Zenodo for anonymized data and projects that have code (probably by creating a GitHub release and linking it to Zenodo), and – surprisingly, as I did not expect it when I started this blog post – the local 4TU.Research.Data repository for personal data that needs to adhere to the General Data Protection Regulation.

Coda

When we compare the Open Science Framework against the alternatives, it is clear we are losing an important player in the data repository landscape. It does well in terms of GDPR, has a decent API, and facilitated anonymous peer review. The usability could be better, but for small projects, it worked well enough for me (at least until the last redesign). Big alternative platforms such as Zenodo, GitHub, and Dataverse were around at the same time, but it was the Open Science Framework that managed to convince researchers in the social sciences to share data, code, and materials. That is a huge accomplishment, and if users move on to these other platforms, it is largely because the Open Science Framework got people started in practicing open science.

Why I care about replication studies

In 2009 I attended a European Social Cognition Network meeting in Poland. I only remember one talk from that meeting: A short presentation in a nearly empty room. The presenter was a young PhD student - Stephane Doyen. He discussed two studies where he tried to replicate a well-known finding in social cognition research related to elderly priming, which had shown that people walked more slowly after being subliminally primed with elderly related words, compared to a control condition.

His presentation blew my mind. But it wasn’t because the studies failed to replicate – it was widely known in 2009 that these studies couldn’t be replicated. Indeed, around 2007, I had overheard two professors in a corridor discussing the problem that there were studies in the literature everyone knew would not replicate. And they used this exact study on elderly priming as one example. The best solution the two professors came up with to correct the scientific record was to establish an independent committee of experts that would have the explicit task of replicating studies and sharing their conclusions with the rest of the world. To me, this sounded like a great idea.

And yet, in this small conference room in Poland, there was this young PhD student, acting as if we didn’t need specially convened institutions of experts to inform the scientific community that a study could not be replicated. He just got up, told us about how he wasn’t able to replicate this study, and sat down.

It was heroic.

If you're struggling to understand why on earth I thought this was heroic, then this post is for you. You might have entered science in a different time. The results of replication studies are no longer communicated only face to face when running into a colleague in the corridor, or at a conference. But I was impressed in 2009. I had never seen anyone give a talk in which the only message was that an original effect didn’t stand up to scrutiny. People sometimes presented successful replications. They presented null effects in lines of research where the absence of an effect was predicted in some (but not all) tests. But I’d never seen a talk where the main conclusion was just: “This doesn’t seem to be a thing”.

On 12 September 2011 I sent Stephane Doyen an email. “Did you ever manage to publish some of that work? I wondered what has happened to it.” Honestly, I didn’t really expect that he would manage to publish these studies. After all, I couldn’t remember ever having seen a paper in the literature that was just a replication. So I asked, even though I did not expect he would have been able to publish his findings.

Surprisingly enough, he responded that the study would soon appear in press. I wasn’t fully aware of new developments in the publication landscape, where Open Access journals such as PlosOne published articles as long as the work was methodologically solid, and the conclusions followed from the data. I shared this news with colleagues, and many people couldn’t wait to read the paper: An article, in print, reporting the failed replication of a study many people knew to be not replicable. The excitement was not about learning something new. The excitement was about seeing replication studies with a null effect appear in print.

Regrettably, not everyone was equally excited. The publication also led to extremely harsh online comments from the original researcher about the expertise of the authors (e.g., suggesting that findings can fail to replicate due to “Incompetent or ill-informed researchers”), and the quality of PlosOne (“which quite obviously does not receive the usual high scientific journal standards of peer-review scrutiny”). This type of response happened again, and again, and again. Another failed replication led to a letter by the original authors that circulated over email among eminent researchers in the area, was addressed to the original authors, and ended with “do yourself, your junior co-authors, and the rest of the scientific community a favor. Retract your paper.”

Some of the historical record on discussions between researchers around between 2012-2015 survives online, in Twitter and Facebook discussions, and blogs. But recently, I started to realize that most early career researchers don’t read about the replication crisis through these original materials, but through summaries, which don’t give the same impression as having lived through these times. It was weird to see established researchers argue that people performing replications lacked expertise. That null results were never informative. That thanks to dozens of conceptual replications, the original theoretical point would still hold up even if direct replications failed. As time went by, it became even weirder to see that none of the researchers whose work was not corroborated in replication studies ever published a preregistered replication study to silence the critics. And why were there even two sides to this debate? Although most people agreed there was room for improvement and that replications should play some role in improving psychological science, there was no agreement on how this should work. I remember being surprised that a field was only now thinking about how to perform and interpret replication studies if we had been doing psychological research for more than a century.
 

I wanted to share this autobiographical memory, not just because I am getting old and nostalgic, but also because young researchers are most likely to learn about the replication crisis through summaries and high-level overviews. Summaries of history aren’t very good at communicating how confusing this time was when we lived through it. There was a lot of uncertainty, diversity in opinions, and lack of knowledge. And there were a lot of feelings involved. Most of those things don't make it into written histories. This can make historical developments look cleaner and simpler than they actually were.

It might be difficult to understand why people got so upset about replication studies. After all, we live in a time where it is possible to publish a null result (e.g., in journals that only evaluate methodological rigor, but not novelty, journals that explicitly invite replication studies, and in Registered Reports). Don't get me wrong: We still have a long way to go when it comes to funding, performing, and publishing replication studies, given their important role in establishing regularities, especially in fields that desire a reliable knowledge base. But perceptions about replication studies have changed in the last decade. Today, it is difficult to feel how unimaginable it used to be that researchers in psychology would share their results at a conference or in a scientific journal when they were not able to replicate the work by another researcher. I am sure it sometimes happened. But there was clearly a reason those professors I overheard in 2007 were suggesting to establish an independent committee to perform and publish studies of effects that were widely known to be not replicable.

As people started to talk about their experiences trying to replicate the work of others, the floodgates opened, and the shells fell off peoples' eyes. Let me tell you that, from my personal experience, we didn't call it a replication crisis for nothing. All of a sudden, many researchers who thought it was their own fault when they couldn't replicate a finding started to realize this problem was systemic. It didn't help that in those days it was difficult to communicate with people you didn't already know. Twitter (which is most likely the medium through which you learned about this blog post) launched in 2006, but up to 2010 hardly any academics used this platform. Back then, it wasn't easy to get information outside of the published literature. It's difficult to express how it feels when you realize 'it's not me - it's all of us'. Our environment influences which phenotypic traits express themselves. These experiences made me care about replication studies.

If you started in science when replications were at least somewhat more rewarded, it might be difficult to understand what people were making a fuss about in the past. It's difficult to go back in time, but you can listen to the stories by people who lived through those times. Some highly relevant stories were shared after the recent multi-lab failed replication of ego-depletion (see tweets by Tom Carpenter and Dan Quintana). You can ask any older researcher at your department for similar stories, but do remember that it will be a lot more difficult to hear the stories of the people who left academia because most of their PhD consisted of failures to build on existing work.

If you want to try to feel what living through those times must have been like, consider this thought experiment. You attend a conference organized by a scientific society where all society members get to vote on who will be a board member next year. Before the votes are cast, the president of the society informs you that one of the candidates has been disqualified. The reason is that it has come to the society’s attention that this candidate selectively reported results from their research lines: The candidate submitted only those studies for publication that confirmed their predictions, and did not share studies with null results, even though these null results were well designed studies that tested sensible predictions. Most people in the audience, including yourself, were already aware of the fact that this person selectively reported their results. You knew publication bias was problematic from the moment you started to work in science, and the field knew it was problematic for centuries. Yet here you are, in a room at a conference, where this status quo is not accepted. All of a sudden, it feels like it is possible to actually do something about a problem that has made you feel uneasy ever since you started to work in academia.

You might live through a time where publication bias is no longer silently accepted as an unavoidable aspect of how scientists work, and if this happens, the field will likely have a very similar discussion as it did when it started to publish failed replication studies. And ten years later, a new generation will have been raised under different scientific norms and practices, where extreme publication bias is a thing of the past. It will be difficult to explain to them why this topic was a big deal a decade ago. But since you’re getting old and nostalgic yourself, you think that it’s useful to remind them, and you just might try to explain it to them in a 2 minute TikTok video.

History merely repeats itself. It has all been done before. Nothing under the sun is truly new.
Ecclesiastes 1:9



Thanks to Farid Anvari, Ruben Arslan, Noah van Dongen, Patrick Forscher, Peder Isager, Andrea Kis, Max Maier, Anne Scheel, Leonid Tiokhin, and Duygu Uygun for discussing this blog post with me (and in general for providing such a stimulating social and academic environment in times of a pandemic).

The Value of Preregistration for Psychological Science: A Conceptual Analysis

This blog is an excerpt of an invited journal article for a special issue of Japanese Psychological Review, that I am currently one week overdue with (but that I hope to complete soon). I hope this paper will raise the bar in the ongoing discussion about the value of preregistration in psychological science. If you have any feedback on what I wrote here, I would be very grateful to hear it, as it would allow me to improve the paper I am working on. If we want to fruitfully discuss preregistration, researchers need to provide a clear conceptual definition of preregistration, anchored in their philosophy of science.

For as long as data has been used to support scientific claims, people have tried to selectively present data in line with what they wish to be true. In his treatise ‘On the Decline of Science in England: And on Some of its Cases’ Babbage (1830) discusses what he calls cooking: “One of its numerous processes is to make multitudes of observations, and out of these to select those only which agree or very nearly agree. If a hundred observations are made, the cook must be very unlucky if he can not pick out fifteen or twenty that will do up for serving.” In the past researchers have proposed solutions to prevent bias in the literature. With the rise of the internet it has become feasible to create online registries that ask researchers to specify their research design and the planned analyses. Scientific communities have started to make use of this opportunity (for a historical overview, see Wiseman, Watt, & Kornbrot, 2019).

Preregistration in psychology has been a good example of ‘learning by doing’. Best practices are continuously updated as we learn from practical challenges and early meta-scientific investigations into how preregistrations are performed. At the same time, discussions have emerged about what the goal of preregistration is, whether preregistration is desirable, and what preregistration should look like across different research areas. Every practice comes with costs and benefits, and it is useful to evaluate whether and when preregistration is worth it. Finally, it is important to evaluate how preregistration relates to different philosophies of science, and when it facilitates or distracts from goals scientists might have. The discussion about benefits and costs of preregistration has not been productive up to now because there is a general lack of a conceptual analysis of what preregistration entails and aims to accomplish, which leads to disagreements that are easily resolved when a conceptual definition would be available. Any conceptual definition about a tool that scientists use, such as preregistration, must examine the goals it achieves, and thus requires a clearly specified view on philosophy of science, which provides an analysis of different goals scientists might have. Discussing preregistration without discussing philosophy of science is a waste of time.

What is Preregistration For?

Preregistration has the goal to transparently prevent bias due to selectively reporting analyses. Since bias in estimates only occurs in relation to a true population parameter, preregistration as discussed here is limited to scientific questions that involve estimates of population values from samples. Researchers can have many different goals when collecting data, perhaps most notably theory development, as opposed to tests of statistical predictions derived from theories. When testing predictions, researchers might want a specific analysis to yield a null effect, for example to show that including a possible confound in an analysis does not change their main results. More often perhaps, they want an analysis to yield a statistically significant result, for example so that they can argue the results support their prediction, based on a p-value below 0.05. Both examples are sources of bias in the estimate of a population effect size. In this paper I will assume researchers use frequentist statistics, but all arguments can be generalized to Bayesian statistics (Gelman & Shalizi, 2013). When effect size estimates are biased, for example due to the desire to obtain a statistically significant result, hypothesis tests performed on these estimates have inflated Type 1 error rates, and when bias emerges due to the desire to obtain a non-significant test result, hypothesis tests have reduced statistical power. In line with the general tendency to weigh Type 1 error rates (the probability of obtaining a statistically significant result when there is no true effect) as more serious than Type 2 error rates (the probability of obtaining a non-significant result when there is a true effect), publications that discuss preregistration have been more concerned with inflated Type 1 error rates than with low power. However, one can easily think of situations where the latter is a bigger concern.

If the only goal of a researcher is to prevent bias it suffices to make a mental note of the planned analyses, or to verbally agree upon the planned analysis with collaborators, assuming we will perfectly remember our plans when analyzing the data. The reason to write down an analysis plan is not to prevent bias, but to transparently prevent bias. By including transparency in the definition of preregistration it becomes clear that the main goal of preregistration is to convince others that the reported analysis tested a clearly specified prediction. Not all approaches to knowledge generation value prediction, and it is important to evaluate if your philosophy of science values prediction to be able to decide if preregistration is a useful tool in your research. Mayo (2018) presents an overview of different arguments for the role prediction plays in science and arrives at a severity requirement: We can build on claims that passed tests that were highly capable of demonstrating the claim was false, but supported the prediction nevertheless. This requires that researchers who read about claims are able to evaluate the severity of a test. Preregistration facilitates this.

Although falsifying theories is a complex issue, falsifying statistical predictions is straightforward. Researchers can specify when they will interpret data as support for their claim based on the result of a statistical test, and when not. An example is a directional (or one-sided) t-test testing whether an observed mean is larger than zero. Observing a value statistically smaller or equal to zero would falsify this statistical prediction (as long as statistical assumptions of the test hold, and with some error rate in frequentist approaches to statistics). In practice, only range predictions can be statistically falsified. Because resources and measurement accuracy are not infinitely large, there is always a value close enough to zero that is statistically impossible to distinguish from zero. Therefore, researchers will need to specify at least some possible outcomes that would not be considered support for their prediction that statistical tests can pick up on. How such bounds are determined is a massively understudied problem in psychology, but it is essential to have falsifiable predictions.

Where bounds of a range prediction enable statistical falsification, the specification of these bounds is not enough to evaluate how highly capable a test was to demonstrate a claim was wrong. Meehl (1990) argues that we are increasingly impressed by a prediction, the more ways a prediction could have been wrong.  He writes (1990, p. 128): “The working scientist is often more impressed when a theory predicts something within, or close to, a narrow interval than when it predicts something correctly within a wide one.” Imagine making a prediction about where a dart will land if I throw it at a dartboard. You will be more impressed with my darts skills if I predict I will hit the bullseye, and I hit the bullseye, than when I predict to hit the dartboard, and I hit the dartboard. Making very narrow range predictions is a way to make it statistically likely to falsify your prediction, if it is wrong. It is also possible to make theoretically risky predictions, for example by predicting you will only observe a statistically significant difference from zero in a hypothesis test if a very specific set of experimental conditions is met that all follow from a single theory. Regardless of how researchers increase the capability of a test to be wrong, the approach to scientific progress described here places more faith in claims based on predictions that have a higher capability of being falsified, but where data nevertheless supports the prediction. Anyone is free to choose a different philosophy of science, and create a coherent analysis of the goals of preregistration in that framework, but as far as I am aware, Mayo’s severity argument currently provides one of the few philosophies of science that allows for a coherent conceptual analysis of the value of preregistration.

Researchers admit to research practices that make their predictions, or the empirical support for their prediction, look more impressive than it is. One example of such a practice is optional stopping, where researchers collect a number of datapoints, perform statistical analyses, and continue the data collection if the result is not statistically significant. In theory, a researcher who is willing to continue collecting data indefinitely will always find a statistically significant result. By repeatedly looking at the data, the Type 1 error rate can inflate to 100%. Even though in practice the inflation will be smaller, optional stopping strongly increases the probability that a researcher can interpret their result as support for their prediction. In the extreme case, where a researcher is 100% certain that they will observe a statistically significant result when they perform their statistical test, their prediction will never be falsified. Providing support for a claim by relying on optional stopping should not increase our faith in the claim by much, or even at all. As Mayo (2018, p. 222) writes: “The good scientist deliberately arranges inquiries so as to capitalize on pushback, on effects that will not go away, on strategies to get errors to ramify quickly and force us to pay attention to them. The ability to register how hunting, optional stopping, and cherry picking alter their error-probing capacities is a crucial part of a method’s objectivity.” If researchers were to transparently register their data collection strategy, readers could evaluate the capability of the test to falsify their prediction, conclude this capability is very small, and be relatively unimpressed by the study. If the stopping rule keeps the probability of finding a non-significant result when the prediction is incorrect high, and the data nevertheless support the prediction, we can choose to act as if the claim is correct because it has been severely tested. Preregistration thus functions as a tool to allow other researchers te transparently evaluate the severity with which a claim has been tested.

The severity of a test can also be compromised by selecting a hypothesis based on the observed results. In this practice, known as Hypothesizing After the Results are Known (HARKing, Kerr, 1998) researchers look at their data, and then select a prediction. This reversal of the typical hypothesis testing procedure makes the test incapable of demonstrating the claim was false. Mayo (2018) refers to this as ‘bad evidence, no test’. If we choose a prediction from among the options that yield a significant result, the claims we make base on these ‘predictions’ will never be wrong. In philosophies of science that value predictions, such claims do not increase our confidence that the claim is true, because it has not yet been tested. By preregistering our predictions, we transparently communicate to readers that our predictions predated looking at data, and therefore that the data we present as support of our prediction could have falsified our hypothesis. We have not made our test look more severe by narrowing the range of our predictions after looking at the data (like the Texas sharpshooter who draws the circles of the bullseye after shooting at the wall of the barn). A reader can transparently evaluate how severely our claim was tested.

As a final example of the value of preregistration to transparently allow readers to evaluate the capability of our prediction to be falsified, think about the scenario described by Babbage at the beginning of this article, where a researchers makes multitudes of observations, and selects out of all these tests only those that support their prediction. The larger the number of observations to choose from, the higher the probability that one of the possible tests could be presented as support for the hypothesis. Therefore, from a perspective on scientific knowledge generation where severe tests are valued, choosing to selectively report tests from among many tests that were performed strongly reduces the capability of a test to demonstrate the claim was false. This can be prevented by correcting for multiple testing by lowering the alpha level depending on the number of tests.

The fact that preregistration is about specifying ways in which your claim could be false is not generally appreciated. Preregistrations should carefully specify not just the analysis researchers plan to perform, but also when they would infer from the analyses that their prediction was wrong. As the preceding section explains, successful predictions impress us more when the data that was collected was capable of falsifying the prediction. Therefore, a preregistration document should give us all the required information that allows us to evaluate the severity of the test. Specifying exactly which test will be performed on the data is important, but not enough. Researchers should also specify when they will conclude the prediction was not supported. Beyond specifying the analysis plan in detail, the severity of a test can be increased by narrowing the range of values that are predicted (without increasing the Type 1 and Type 2 error rate), or making the theoretical prediction more specific by specifying detailed circumstances under which the effect will be observed, and when it will not be observed.

When is preregistration valuable?

If one agrees with the conceptual analysis above, it follows that preregistration adds value for people who choose to increase their faith in claims that are supported by severe tests and predictive successes. Whether this seems reasonable depends on your philosophy of science. Preregistration itself does not make a study better or worse compared to a non-preregistered study. Sometimes, being able to transparently evaluate a study (and its capability to demonstrate claims were false) will reveal a study was completely uninformative. Other times we might be able to evaluate the capability of a study to demonstrate a claim was false even if the study is not transparently preregistered. Examples are studies where there is no room for bias, because the analyses are perfectly constrained by theory, or because it is not possible to analyze the data in any other way than was reported. Although the severity of a test is in principle unrelated to whether it is pre-registered or not, in practice there will be a positive correlation that is caused by the studies where the ability to evaluate how capable these studies were to demonstrate a claim was false is improved by transparently preregistering, such as studies with multiple dependent variables to choose from, which do not use standardized measurement scale so that the dependent variable can be calculated in different ways, or where additional data is easily collected, to name a few.

We can apply our conceptual analysis of preregistration to hypothetical real-life situations to gain a better insight into when preregistration is a valuable tool, and when not. For example, imagine a researcher who preregisters an experiment where the main analysis tests a linear relationship between two variables. This test yields a non-significant result, thereby failing to support the prediction. In an exploratory analysis the authors find that fitting a polynomial model yields a significant test result with a low p-value. A reviewer of their manuscript has studied the same relationship, albeit in a slightly different context and with another measure, and has unpublished data from multiple studies that also yielded polynomial relationships. The reviewer also has a tentative idea about the underlying mechanism that causes not a linear, but a polynomial, relationship. The original authors will be of the opinion that the claim of a polynomial relationship has passed a less severe test than their original prediction of a linear prediction would have passed (had it been supported). However, the reviewer would never have preregistered a linear relationship to begin with, and therefore does not evaluate the switch to a polynomial test in the exploratory result section as something that reduces the severity of the test. Given that the experiment was well-designed, the test for a polynomial relationship will be judged as having greater severity by the reviewer than by the authors. In this hypothetical example the reviewer has additional data that would have changed the hypothesis they would have preregistered in the original study. It is also possible that the difference in evaluation of the exploratory test for a polynomial relationship is based purely on a subjective prior belief, or on the basis of knowledge about an existing well-supported theory that would predict a polynomial, but not a linear, relationship.

Now imagine that our reviewer asks for the raw data to test whether their assumed underlying mechanism is supported. They receive the dataset, and looking through the data and the preregistration, the reviewer realizes that the original authors didn’t adhere to their preregistered analysis plan. They violated their stopping rule, analyzing the data in batches of four and stopping earlier than planned. They did not carefully specify how to compute their dependent variable in the preregistration, and although the reviewer has no experience with the measure that has been used, the dataset contains eight ways in which the dependent variable was calculated. Only one of the eight ways in which the dependent variable yields a significant effect for the polynomial relationship. Faced with this additional information, the reviewer believes it is much more likely that the analysis testing the claim was the result of selective reporting, and now is of the opinion the polynomial relationship was not severely tested.

Both of these evaluations of how severely a hypothesis was tested were perfectly reasonable, given the information reviewer had available. It reveals how sometimes switching from a preregistered analysis to an exploratory analysis does not impact the evaluation of the severity of the test by a reviewer, while in other cases a selectively reported result does reduce the perceived severity with which a claim has been tested. Preregistration makes more information available to readers that can be used to evaluate the severity of a test, but readers might not always evaluate the information in a preregistration in the same way. Whether a design or analytic choice increases or decreases the capability of a claim to be falsified depends on statistical theory, as well as on prior beliefs about the theory that is tested. Some practices are known to reduce the severity of tests, such as optional stopping and selective reporting analyses that yield desired results, and therefore it is easier to evaluate how statistical practices impact the severity with which a claim is tested. If a preregistration is followed through exactly as planned then the tests that are performed have desired error rates in the long run, as long as the test assumptions are met. Note that because long run error rates are based on assumptions about the data generating process, which are never known, true error rates are unknown, and thus preregistration makes it relatively more likely that tests have desired long run error rates. The severity of a tests also depends on assumptions about the underlying theory, and how the theoretical hypothesis is translated into a statistical hypothesis. There will rarely be unanimous agreement on whether a specific operationalization is a better or worse test of a hypothesis, and thus researchers will differ in their evaluation of how severely specific design choices tests a claim. This once more highlights how preregistration does not automatically increase the severity of a test. When it prevents practices that are known to reduce the severity of tests, such as optional stopping, preregistration leads to a relative increase in the severity of a test compared a non-preregistered study. But when there is no objective evaluation of the severity of a test, as is often the case when we try to judge how severe a test was based on theoretical grounds, preregistration merely enables a transparent evaluation of the capability of a claim to be falsified.

Can you explain why you did not share data and materials when publishing your article?

I recently signed the Peer Reviewers’ Openness Initiative. At its core, it boils down to one very simple thing: As a reviewer, I will from 2017 onwards ask authors to explain why they can not share their data and materials. Without an explanation, I will choose not to review this specific article. 

In Peter Singer’s ‘The Life You Can Save’ (2009) he describes a simple situation. You walk past a shallow pond where you see a small child who is in danger of drowning. No one else is around, but you can easily save the child if you act immediately. You won’t have time to take off your shoes, and the shoes you are wearing will be ruined, and no one will refund them. Will you save the child at the expense of your shoes?

The answer many people give is: “Yes, sure”. Peter Singer goes on to argue that the same amount of money you would be willing to spend in this situation, could be used right now to save the life of children somewhere else in the world.

Why this story stuck with me, because it forces you to explain your behavior. Why don’t I give more to charity?

I personally think it is important to be able to rationalize some important behaviors I perform. When it comes to my work, which is paid for by taxpayers, I feel I need to give them optimal value for their money. When I share my data, stimuli, and materials, science will become more transparent and efficient. If I don’t adhere to these open science principles, I think I need to give an explanation. That’s why from 2013, most of the data, materials, and scripts of papers I was a first author or co-author on are publically available.

As in Peter Singer’s scenario, the rationalization not to do something is sometimes difficult, and sometimes easy. If you don’t have enough money as it is, you don’t have any money to donate to others. Similarly, if you can not share materials, such as the IAPS pictures I used in Lakens, Fockenberg, Lemmens, Ham, & Midden, 2013, the justification is easy. At other times, such as when you are considering spending money on gadgets you don’t really need, or when the materials and data have no copyright or privacy issues, you might be affectively inclined to come up with an excuse, only to realize they don’t hold up after careful deliberation.

It’s this latter category we aim to address with the Peer Reviewers’ Openness Initiative. It is so easy to just ignore this rational justification process when you are a little busy. The goal is to make people ask themselves: Could I share the data, materials, and stimuli? Would doing so make science more transparent and efficient?

I’ve started send out tweets to let you know how many papers I review share all data and materials, or explain why this was not possible. So far, I’m at 3/3. After all, journals like PLOS already ask authors to specify the reasons for restrictions on public data deposition in line with the PRO initiative (they just don’t ask authors to include stimuli or materials whenever possible). I have a strong conviction that researchers want to do what is best for science. Every now and then, we just need someone who asks us to reflect upon, and explain, our behavior. 

If you want to help remind researchers they need to rationalize why they are not sharing data, materials, and stimuli, you can sign the PRO initiative here.

For other views related to the Initiative, see blog posts by Richard Morey, Candice Morey, and Rolf Zwaan.

[Read the paper -- Sign the Initiative -- More resources for open science]