Imagine that an architect were tasked with building a house, came up with a design, had a team of builders construct the house and then asked the prospective owner whether anything needed to be changed. The prospective owner tells the architect that he doesn’t like a part of the house and would rather it were done differently. To please the owner, the architect knocks down that section of the house, rebuilds as directed and then repeats the process until the prospective owner is satisfied.
To any sane person, this situation would seem completely ridiculous. It would be stupid to go through all the effort of constructing a building without first confirming whether everything was acceptable at the planning stage. Strangely enough, this is not far off from the common practice in science. We design and perform experiments, collate and narrate the results and then send our write-ups to journal editors, who in turn distribute the manuscript among a coterie of our peers. These peer reviewers supposedly check the manuscript for publication-worthiness, advise us of errors in execution or interpretation, and recommend various changes and additions. Frequently, this turns out to be a prolonged exercise that incurs significant expense, but does not add much value to the central theme of the manuscript. Without a doubt, the scientific process can benefit from well-performed peer review. However, extant peer review processes are more wasteful than beneficial as in the analogy above.
Preregistration will save resources
If thorough reviews of research plans would be performed at the logical, pre-experimental stage, most of the justifiable demands for additional data points, modifications of ingredients and alterations of direction can be made before any resources are expended. This presents a marked improvement over the current situation because researchers can then start with optimized protocols that best address their needs. In contrast, when reviews happen after the fact, the calls for alternative experiments and repetition of old ones, which are all too common, cause significant wastage of the time and money which were already spent. Despite unending commentaries and discussions, there is no clear estimate of the monetary toll of such experimental wastage. Scientists, funders and policy makers have begun expressing grave concerns.
Another important consideration is that the current system of post-submission peer review remains unable to mitigate the famed reproducibility crisis, estimated to cost over US$28 billion each year in the United States alone. Some might even argue that wayward peer reviews, which ask for major directional alterations, might be contributing to this costly global problem. Perhaps pre-registered and pre-planned projects with detailed reviews that correct predictable deficiencies might allay some of these staggering costs; avoidable costs that the taxpayers, who do not have unlimited funds, needlessly bear.
It is also worth noting that recommendations from laboratory members, collaborators or core facility service providers are sometimes set aside with the intention of saving time and money whereas similar requests made by reviewers will be followed, sometimes at an even higher cost than if they had been performed earlier. This might be because the authors feel, as Ron Vale has suggested, that they are held hostage by the peer review process and have no choice but to accede to the reviewers’ demands if their paper is to move forward. If this is true, it is further reason to involve reviewers earlier in the publication process.
Preregistration will save time
The current publication system is glacially slow, taking weeks or months before a paper is available online. For that duration, the work is not easily accessible, forcing researchers to operate without information that could potentially point them in the right directions or aid in troubleshooting. Delays in the availability of information are particularly troublesome in fast-moving research fields, in which the data may be outdated by the time it is published. Preprints mitigate some problems associated with the slow publication process, but are not seen as proper publications themselves; thus the work usually ends up being submitted later to the more mainstream venues and going through the aforementioned expenses and delays.
These delays in publication can be costly both to individual scientists and to the scientific enterprise itself. For example, a recent paper, authored by one of us, had six authors and took about eight months from submission through revisions to acceptance and then several more weeks before final publication. Even before the paper had completed the review process, half of those authors had already finished their contracts and two other authors had relocated to a new institution. This obviously presents major hurdles for for making the suggested revisions. This experience is unlikely to be an isolated event, considering that most research contracts are of short durations, and papers are typically submitted towards the end of a research project when employment contracts are often ending. This means that scientists either have to work without remuneration after their contracts expire, rely on another scientist who might not have been involved or familiar with the project, or see their work go unpublished – meaning the money and time would have been spent with no tangible benefit to the scientific community or the public at large.
Preregistration offers a way to significantly speed up the publication process. When research has been preregistered and reviewed before experimentation, review of the publication only needs to serve a quality control function. There would be no need for extensive changes because the work that needed to be done to meaningfully address the research question would have been decided beforehand and performed according to plan. A single person would be able to verify that the manuscript submitted is in accordance with the preregistered plans. At most, there would be minor changes pertaining to the structure and narrative of the paper and changes should no longer take several weeks or months.
The above discussion highlights just two ways in which preregistration and pre-experimental peer review would benefit science. They have the potential to substantially decrease the amount of time, money and other resources wasted on unnecessary experiments and in the performance of work which may never be published. Additionally, we can also increase the pace at which experimental results are made available, allowing researchers to make decisions on the best available data as soon as it is known. Finally, we can also aid early career scientists on short term contracts by shortening the time required for publication and thus lowering the risk of having to work out of contract.
Doesn’t most good science require meticulous planning? Can experimental plans not be reviewed by peers or other designated experts ahead of time to eliminate “reviewer wastage” and increase efficiency? We think so and, given the potential benefits to the scientific endeavour, it is high time that the scientific community ask itself why this solution is not being proactively explored at all levels.
Jason Bosch earned a PhD through the Gregor Mendel Institute, part of the Vienna BioCenter, and the University of Vienna and is currently a postdoc at the University of Pretoria. Anoop Kavirayani is a pathologist at the Vienna BioCenter Campus Facilities. This commentary represents their own views which are neither endorsed by, nor necessarily representative of the views of their host institutions.
The above article was a collaboration between myself and Anoop Kavirayani that started when he suggested writing a combined article after my SAJS commentary. Although originally conceived as a much larger project, the idea was reworked many times until it became this smaller article on the potential benefits of preregistration. We had hoped to publish in a mainstream venue where it could be read by scientists, politicians and other decision makers but after several months of submissions and silence, we decided it was better to at least send it out into the world.
As a further note from my side, I would like to dedicate my portion of this article to Jon Tennant who recently passed away in a motorcycle accident. I do not recall how I came to learn about him and his own work promoting open science but he did serve as a bit of a role model and the scientific community is poorer without him. Earlier this year, he published a collection of his writings about open science as an ebook, The (R)evolution of Open Science, which is available free on Zenodo.