When authors share their data and analysis code, the idea is that anyone should be able to run that code on that data and get the results reported in the paper. But sharing data and code in a way that results are reproducible is difficult, and researchers have often not received training in how to share data and code. As a consequence, there are often basic mistakes in public repositories that stand in the way of computationally reproducible results. Automated checks can help researchers identify and fix these mistakes, such as not sharing all files that are loaded in code, using absolute instead of relative file-paths, using values that are not computed in the code that is shared, changing file names so they no longer match the code, or not loading all libraries.
Checking whether data and code is reproducible is tedious. It also often requires repeated back and forths between the person performing the checks, and the researchers sharing data and code. Ideally, checks do not just focus on whether code runs, but also whether all results in a manuscript are reproduced by the code, but this takes a lot of time.
Metacheck’s reproducibility_check module attempts to perform all these checks automatically. When it succeeds, researchers and/or reviewers, data managers, or editors know that data and code is shared in a reproducible manner, and they do not need to perform manual checks. This saves everyone time, and leads to data and code repositories that are more useful for science. Our reproducibility check module is completely code-based. It uses no AI (although modules it builds on, such as data_check, can optionally use a large language model, by setting llm_use(TRUE), to better understand the files in a repository, such as which files belong to which study), and it can run locally on your computer without sending information to an external server.
If I say so myself, it is pretty damn cool. It is one of the rare experiences where I honestly thought something could not be done, but where I surprised myself because it actually works.
This post shows the module at work on two versions of the same small study: one where every reported result can be reproduced, and one where several cannot.
Computationally reproducing a simple Stroop experiment
Both papers analyse the same classroom data from my introduction to psychology and technology course. Students named the ink color of words that matched the color (the word “red” printed in red) and of words that did not (the word “red” printed in green), and reported the time they needed in seconds. Both papers report four results: the mean and standard deviation of the reaction times in each condition, the correlation between the reaction times in the two conditions, a paired t-test for the Stroop effect, and a comparison of the Stroop effect between the students tested in 2013 and in 2014.
- The good paper, https://github.com/Lakens/stroop_effect_paper, computes every reported number in its analysis code. The manuscript is a Quarto document, so the text prints these numbers directly from the analysis.
- The bad paper, https://github.com/Lakens/stroop_effect_paper_bad, has several common mistakes deliberately built in. The results were typed into the text by hand, the shared data contain a value the authors removed before their own analysis without saying so, some reported results are not computed by the code at all, and the code stops with an error before it reaches the last analysis.
Running the code locally, or using our server on SURF
The reproducibility check runs code that other people wrote. That is not without risk. A script can delete files, read files that have nothing to do with the project, or send information over the internet, by accident or on purpose. By default, execute = TRUE runs each script in a separate R process on your own computer. This prevents a script that crashes from taking your own R session down with it, but the script can still reach every file on your computer, and the internet. That is fine for the two repositories in this post, which I wrote myself, but not for code from a repository you have not created, as a bad actor might insert problematic code.
For such code, add sandbox = "docker". Docker is a program that runs software in a container: a closed environment on your computer, separated from the rest of the system. Metacheck runs every script in its own container, and has set up these containers so that the code can do as little harm as possible:
- The container has no internet connection while the code runs. Only the step before it, which installs the packages the code needs, is connected to the internet, and in that step none of the paper’s own code runs.
- The code runs on a copy of the repository’s files, in a temporary folder. That folder is the only place on your computer it can write to, and the files on the rest of your computer are not visible inside the container.
- The code runs as an ordinary user without administrator rights, cannot obtain extra rights, and can start only a limited number of processes.
- Each script is stopped after a time limit, 10 minutes by default.
No sandbox makes running unknown code completely safe: a flaw in Docker itself could in principle still be exploited. But these settings close off the usual ways in which a script can do damage. The Reproducibility Check chapter lists the exact settings, and compares them with running code without Docker.
Running the check in Docker takes some preparation, and disk space. You need Docker Desktop (on Windows and macOS) or Docker Engine (on Linux), and the space Docker Desktop itself needs. The first time you run the check, Docker downloads Metacheck’s image for R code: R, Quarto, and the roughly 750 R packages that occurred in a scan of thousands of R files shared with psychology papers, together about 4.9 GB. Python code runs in a second image of about 2.7 GB. Because most packages a paper needs are already in the image, they do not have to be installed again for every paper, which makes the check much faster (but at the expense of a large docker image that you install once). The chapter explains how the image was built.
If you do not use R, the Docker installation runs Metacheck’s website on your own computer, with the same checks. It needs about 1.7 GB of disk space and a computer with 8 GB of memory, plus Docker. There, the code of a repository runs inside Metacheck’s own container: it cannot reach the files on your computer, but unlike in the sandbox described above, it can use the internet. The packages the code needs are downloaded while the check runs. Running the check on your own computer also means that an unpublished manuscript never leaves it, which matters for editors and reviewers who may not upload a submission to an external server.
The easiest way to try the check is our server on SURF Research Cloud, which runs on the servers of SURF, the cooperative of the Dutch universities, in the Netherlands. You need nothing but a web browser. On Metacheck’s website, upload a PDF under Run Metacheck on a Paper and tick Also run the code in the paper’s repository, or, under Run Metacheck on a Repository, choose a folder or a .zip file, or paste a link to an online repository such as GitHub or OSF. Uploaded files are deleted as soon as they have been read, and the report is kept on the server for one hour, so save it if you want to keep it.
The server’s capacity is shared by everyone who uses it, so it is meant for checking a paper or a repository now and then. If you want to check many papers, run Metacheck on your own computer, or build your own server on SURF Research Cloud, which researchers at Dutch institutions can do through their institution or with a free NWO grant. The Setting up Metacheck chapter describes all these options.
Running the reproducibility check in R.
I will illustrate the reproducibility check module in R. Metacheck reads a paper from its PDF. We download both PDFs from GitHub, and convert() turns each into a structured file that read() loads into R. If you want to follow along in the Metacheck website, download the two papers from here and here.
download.file(
"https://raw.githubusercontent.com/Lakens/stroop_effect_paper/main/output/analysis.pdf",
destfile = "good_paper.pdf", mode = "wb"
)
download.file(
"https://raw.githubusercontent.com/Lakens/stroop_effect_paper_bad/main/analysis.pdf",
destfile = "bad_paper.pdf", mode = "wb"
)
paper <- read(convert(file_path = "good_paper.pdf", save_path = "."))
bad_paper <- read(convert(file_path = "bad_paper.pdf", save_path = "."))Both papers link to their repository. The repo_check module in Metacheck supports over 270 data repositories, and can automatically download files from these repositories.
Running the module is a single line. By default, reproducibility_check only inspects the code without running it. execute = TRUE is needed to actually run the full computational reproducibility check:
mo_good <- module_run(paper, "reproducibility_check", execute = TRUE)
mo_good$traffic_light#> [1] "green"cat(mo_good$summary_text)#>
#> - We assessed AND ran 2 R code files for reproducibility (each in an isolated subprocess).
#> - 2 files appear runnable so far (parses, inputs resolve, placeable in the run order).
#> - 2 of 2 scripts ran without error when executed.
#> - 5 installable dependencies detected.
#> - 39 statistics stored from the extracted output; 11 of 11 reported tests matched (100%).The light is green: every script ran without an error, and every result reported in the manuscript was among the results the code produced when it ran. mo_good$table lists each script and what happened when it was run:
mo_good$table[, c("file_name", "outcome")]#> file_name outcome
#> analysis.qmd analysis.qmd ran_ok
#> clean_data.R clean_data.R ran_okclean_data.R turns the raw data file into the processed data file, and analysis.qmd is the manuscript, whose code chunks contain the analysis. The repository also contains renv/activate.R, a script the renv package adds to a project to set up its package library. The module does not check the contents of this folder, because it knows it is not part of the analysis.
mo_good$match_table shows, for each result reported in the paper, whether it was found in the output of the code, and in which analysis:
mo_good$match_table[, c("reported", "found", "confidence", "analysis")] |>
knitr::kable()| reported | found | confidence | analysis |
|---|---|---|---|
| M=15.1 | TRUE | full | mean by group |
| M=23 | TRUE | full | mean by group |
| SD=4.1 | TRUE | full | sd by group |
| SD=4.78 | TRUE | full | sd by group |
| r=0.37 t=4.53 df=129 p=<0.001 | TRUE | full | Pearson’s product-moment correlation |
| t=18.04 df=130 p=<0.001 | TRUE | full | Paired t-test |
| Cohen’s dz=1.58 | TRUE | full | d_z |
| M=7.77 | TRUE | full | mean by group |
| M=8.13 | TRUE | full | mean by group |
| t=-0.42 df=120.37 p=0.673 | TRUE | full | Welch Two Sample t-test |
| Cohen’s d=-0.07 | TRUE | full | Cohens_d |
A result is not compared as single numbers, but as a whole: the t-value, the degrees of freedom, and the p-value of one test have to appear together in the output of one analysis. Any single number will sooner or later turn up somewhere in a large output by coincidence, but several numbers of one test appearing together in one analysis will not.
Making your own results findable
The module can only find a result if the code produces it while it runs, in a form the module recognises. In the good paper, every reported number is computed in a code chunk, and the text of the manuscript only rounds these values. A number that is computed only inside the text, by an inline R expression in Quarto or R Markdown, is not part of the code that is run, so it cannot be found.
The module recognises the results of test functions such as
t.test()andcor.test(), of summaries such assapply(stroop_data[, c("congruent", "incongruent")], mean), and of effect sizes from theeffectsizepackage, also when they are stored in a variable (ttest_result <- t.test(...)). A number computed by hand, such as a Cohen’s d computed asmean(diff) / sd(diff), is found when the code prints it, but not when it is only stored in a variable.
A green light means that the numbers in the paper follow from the shared code and data. It does not mean that the analysis was the right one to perform.
Now let’s see what the module returns if we run it on the bad paper:
mo_bad <- module_run(bad_paper, "reproducibility_check", execute = TRUE)
mo_bad$traffic_light#> [1] "red"cat(mo_bad$summary_text)#>
#> - We assessed AND ran 1 R code file for reproducibility (each in an isolated subprocess).
#> - 1 file appears runnable so far (parses, inputs resolve, placeable in the run order).
#> - 0 of 1 script ran without error when executed.
#> - 3 installable dependencies detected.
#> - 10 statistics stored from the extracted output; 2 of 6 reported tests matched (33.3%).The light is red, because the analysis code stopped with an error:
mo_bad$table[, c("file_name", "outcome")]#> file_name outcome
#> analysis.qmd analysis.qmd erroredcat(mo_bad$run_results$error)#> ! in callr subprocess.
#> Caused by error in `eval(predvars, data, env)`:
#> ! object 'stroop_effect' not foundThe code compares the Stroop effect between the two years using a column stroop_effect, but no line of the code creates that column. The authors presumably created it in their R console while they worked, so the code ran on their own computer, but it cannot run for anyone else. This is one of the most common reasons shared code fails.
Before the code was run, the module also changed one line of it. mo_bad$modifications lists every change:
mods <- mo_bad$modifications
mods[mods$change_type == "setwd_removed", c("change_type", "detail")]#> change_type detail
#> 2 setwd_removed C:/Users/dlakens/Desktop/stroop_studyThe code starts with setwd("C:/Users/dlakens/Desktop/stroop_study"), which sets the folder R works in to a folder that only exists on the author’s computer. The module skips this line, and reports that it did, because it would stop the code on any other computer.
Now the comparison between the paper and the output:
mo_bad$match_table[, c("reported", "found", "confidence", "analysis")] |>
knitr::kable()| reported | found | confidence | analysis |
|---|---|---|---|
| t=18.04 df=130 p=0.25 Cohen’s dz=0.25 | FALSE | none | NA |
| t=-0.42 df=120.37 p=0.673 | FALSE | none | NA |
| M=15.1 | TRUE | full | mean by group |
| SD=4.1 | TRUE | full | sd by group |
| M=23 SD=4.78 | FALSE | none | NA |
| r=0.37 | FALSE | none | NA |
Only the mean and standard deviation of the congruent condition were found. Each of the other results is missing for a different reason:
The incongruent mean and standard deviation (M = 23.0, SD = 4.78) are not in the output. The output contains the statistics the code did compute:
mo_bad$stat_output[[1]]$long[, c("analysis", "statistic", "value")]#> analysis statistic value #> 1 mean by group mean 15.1102878787879 #> 2 mean by group mean 22.080446969697 #> 3 sd by group sd 4.08488216267072 #> 4 sd by group sd 11.63934788467 #> 5 Paired t-test t 6.77356838604279 #> 6 Paired t-test df 131 #> 7 Paired t-test p 3.84360296105763e-10 #> 8 Paired t-test mean difference 6.97015909090909 #> 9 Paired t-test conf.low 4.93450571703671 #> 10 Paired t-test conf.high 9.00581246478147The incongruent mean is 22.08, with a standard deviation of 11.64. The reason is in the data. One participant has a reaction time of -99 in the incongruent condition:
bad_data <- read.csv("https://raw.githubusercontent.com/Lakens/stroop_effect_paper_bad/main/stroop.csv") summary(bad_data$Incongruent)#> Min. 1st Qu. Median Mean 3rd Qu. Max. #> -99.00 19.27 22.64 22.08 26.10 35.13-99 is a code for a missing value, not a reaction time. The authors removed this participant before computing the results in their paper, but the code they shared does not, so it analyses -99 as if it were a reaction time.
The paired t-test, t(130) = 18.04, is not found for the same reason: the shared code analyses all 132 participants, including the -99, and gets t(131) = 6.77. The reported p = .25 and Cohen’s dz = 0.25 would not have matched in any case, because neither is consistent with a t-value of 18.04 (the StatCheck and Effect Sizes modules flag both from the text alone).
The correlation, r = 0.37, is reported, but no line of the code computes it.
The comparison between the years, t(120.37) = -0.42, p = .673, has exactly the numbers the good paper reports, so it is probably correct. But the code stopped with an error before it reached this test, so it cannot be verified from what the authors shared.
The three p-values about accuracy (p = .07, p = .34, and p = .000) are not in the table at all. The module only checks a p-value that is reported together with its test statistic. Had they been checked, they would not have been found either: the repository contains no accuracy data.
Open for feedback
Making this software was a challenge. We have validated the code by running it on more than 3000 scientific papers, with hundreds of folders of data and code, across dozens of data repositories. But there will be bugs. Software this complex, that tries to process the almost infinite ways in which researchers can share data and code, is bound to run into bugs. When thousands of people try this out, for some it will fail on their very first try (while for many others, it will run fine). If you find a bug, please raise an issue on the Metacheck github page. You need to share the data or code files that lead to the problem with us, or we can’t fix the problem. If you can’t share the data and code, you can ask an AI agent to analyze the problem, which might be enough to solve it as well. If the tool lacks features you would like to have, let us know, and we will do our best to build them in.
No comments:
Post a Comment