Replications (Van Bergen, 1963)

‘The last couple of years it has been repeatedly pointed out that it is essential to perform replication studies.’ This is the first line of a Dutch article by Annie van Bergen, written in 1963 for the Nederlands Tijdschrift voor de Psychologie en Haar Grensgebieden (Dutch Journal for Psychology and her Border Areas). I first heard about it one month ago, when a group of researchers were giving short presentations about the value of replication research, and Ruud Abma cited some lines from this article. I was able to order a copy of the journal from an antique bookstore (I’ve scanned in the article for those of you who read Dutch, or want to check the references, and you can download it here). Allow me to continue translating the article for you – you will find it surprisingly familiar, even though it was written more than 50 years ago.

‘Duijker (1955) even went as far as to call psychology a true science only when studies are replicated regularly. De Groot (1961) focused on the bad habit – íf they are performed in the first place – not to publish replication studies. Sterling (1959) and – following him – Tullock (1959) denounced the habit of editors of American psychological journals to primarily accept research reports which describe significant results. This policy provides, in the absence of replications, a reasonable likelihood that Type 1 errors (wrongly rejecting the null hypothesis) appear in print (Sterling, 1959, p 34). One could make a doleful sound in response to the independent tests collected by Hanson (1958) of propositions and hypotheses previously published – even though these were not direct replications. The number of non-verified hypotheses was 36 (out of a total of 99 tests).

Why are replication studies performed only rarely, now that the necessity to perform such studies is widely known? Are they related to special difficulties? What are the difficulties that a researcher faces who wants to do something as ostensibly straightforward as repeating an experiment? First of all, he is confronted with the (whether or not perceived) criticism of his colleague, who charges him with a lack of innovativeness. It is likely the over-appreciation of creating something new is so strong, that this is the primary reason for the small number of replications. A reason that is not valid, by the way, because one can start ‘own’ research lines by repeating a related study performed by another researcher (for example like Mulder, 1958, did).’

She then continues by discussing how, when repeating an experiment, one might encounter some unclear or even erroneous issues in the experiment that need to be corrected – however, this no longer makes the replication study a direct replication. Furthermore, there are cultural challenges when replicating an experiment in a different culture. Finally, she highlights the difficulty of drawing strong conclusions from a replication study, especially when it had low statistical power.

She then writes: ‘Finally the researcher needs to be aware of the experimental design and the results of the original study. Most publications are insufficient as a guideline for a well performed replication. This applies especially to recent publications, which seem to primarily have to adhere to the demand of brevity. One thus contacts the author of the original manuscript and asks for the exact instructions and experimental set-up, the scoring procedure, the basic results, etc.. Regrettably, establishing this contact is not that easy. Sometimes the replicator will receive a meaningless short scribble from a secretary, but oftentimes the original researchers seems to belong to the Willem Parel-fans judging by their ‘don’t respond to it Lena’ policy. (Willem Parel was a character created by the Dutch stand-up comedian Wim Sonneveld).

Especially shocking in this respect are the experiences reported by Wolins (1962). Of 37 authors of articles, published between 1959 and 1961, the original materials were requested (not for replication purposes, by the way). 32 authors replied, but the reply from 21 researchers consisted of the notification that the original materials were lost or thrown away. Of the remaining 11, 2 researchers sent the materials months later (too late), and 2 authors requested a say and influence in any publication of a re-analysis. Finally, in 3 of the remaining 7 studies flagrant statistical and calculation errors were discovered.’

She the continues with: ‘This for science fatal (or laughable) situation would not exists to this extent if the habit would exist to perform replication studies. Every researcher would than keep in mind that his work is repeated at a certain time. This would first of all mean that the basic materials would be stored for other researchers, and furthermore this tradition would have as a consequence that analyses are performed with more care, even if only to avoid the risk of being exposed later on.

It is understandable that a researcher who is too busy with commissioned research cannot make time available for replications. What would be possible, is that PhD students and students who have to write a thesis are stimulated to perform replication studies.’ She ends with some examples on how people have incorporated replication research in teaching or PhD projects, and ends with the statement she hopes more people will follow.

I think all of this sounds eerily familiar. The fact that someone working at my university more than 50 years ago wrote something so very similar as I’ve written recently (e.g., Koole & Lakens, 2012) makes me a little proud and a little sad at the same time. However, as Alison Ledgerwood writes in her introduction to the special section on Advancing Our Methods and Practices, this time, ‘online communication, media attention, and a series of conference symposia and journal issues kept the critiques and concerns front and center.' We are finally seeing some changes, such as the Special Issue of Social Psychology on Replications of Important Results in Social Psychology that will appear online May 19th that Brian Nosek and I edited. I’m sure Annie van Bergen would have been thrilled.

P.S. After repeatedly being asked whether it was possible to leave comments at my previous blog on my site, I've moved my blog to Blogger so you can comment. Please be nice.