Why We Should Showcase Research Without Significant Results
POLITICAL SCIENCES |

Why We Should Showcase Research Without Significant Results

PUBLISHING PRACTICES TEND TO HIDE STUDIES THAT LACK STATISTICALLY SIGNIFICANT RESULTS, WITH NEGATIVE SIDE EFFECTS ON THE SPREAD OF FALSE POSITIVES, REPLICABILITY AND INEFFICIENT RESEARCH FUNDING. SCOTT WILLIAMSON SUGGESTS HOW TO OVERCOME THE ISSUE

Academic journals and peer reviewers across different fields currently favour the visibility and publication of those manuscripts which highlight the statistical significance of their results. In turn, this creates an incentive for researchers to produce strongly significant effects with often outdated research designs. In his latest PNAS paper, Bocconi Professor Scott Williamson joins a consortium of authors in the social sciences arguing that the persistence of such publication bias increases the risk of  “false positives”, discourages scientific dissemination, hinders replicability and contributes to inefficient spending of research funds. To mitigate these problems, Professor Williamson and hi co-authors propose a new framework called “RARE”, or Report All Results Efficiently, which demands better practices by all of the different agents involved in scientific research, from grant agencies to investigators and journals.
 
A significant part of the problem highlighted by Prof. Williamson and a team of co-authors is the systematic removal of “null results”, which are those results that do not display statistical significance. As many journals in the social sciences are more likely to publish results with statistical significance and established causalities, researchers tend to omit those less successful results from their publication. This contributes to the so-called “file drawer” problem, where studies with less evident effects are systematically rejected or withdrawn from publication.
 
The “file drawer” problem and the omission of “null results” raise several crucial debates. Firstly, excessive emphasis on the statistical significance of results shadows what is perhaps more meaningful in scientific research, which is the emergence of more advanced and appropriate research designs that may produce null results in the earlier stages of their development. Secondly, researchers need to be aware of studies conducted in the past even if they delivered null results to improve them or perhaps even to avoid unnecessary replications. Thirdly, funding agencies would benefit greatly from larger access to unpublished results to avoid inefficient spending.
 
The “RARE” framework proposed by Prof. Williamson and his co-authors would largely amend these inefficiencies if various actors involved in the research process start to abide by a series of best practices As an example, scientific registries should abide by higher standards for the reporting of null results. Likewise, investigators should commit to publicly reporting results for all pre-registered hypotheses regardless of the findings. Grant agencies should also require researchers to identify registrations from previous grants and to report whether all results have been disseminated in the past, independently from their statistical significance. Finally, journals can play an extremely important role in this paradigm shift by reinforcing incentives on open reporting and by selecting research on the basis of good questions and design rather than on the significance of results.
 
To conclude, the article calls for a new paradigm for scientific dissemination in the social sciences and beyond. This requires larger transparency from all actors involved in research regarding the significance of results in order to overcome the “file drawer” problem and other publication biases. Abiding by the new “RARE” framework is likely to require extra costs on various actors in the short term, as they are required to change existing procedures on the reporting of null results. However,  it is expected to yield large benefits in the future both in terms of efficiency and scientific reputation.
 
David D. Laitin et al. “Reporting All Results Efficiently: A RARE Proposal to Open up the File Drawer.” Proceedings of the National Academy of Sciences Dec 2021, 118 (52) e2106178118; DOI: 10.1073/pnas.2111611118. DOI: https://dx.doi.org/10.1073/pnas.2106178118.
 

by Umberto Platini
Bocconi Knowledge newsletter

News

  • Caselli, Ventoruzzo and Mosca Join the Committee for the Reform of Capital Markets

    The activities will lead to the new Consolidated Text  

  • Bocconi Research Excellence Awards 2024

    Faculty whose publications have been accepted by the most prestigious journals or publishers honored  

Seminars

  March 2024  
Mon Tue Wed Thu Fri Sat Sun
        1 2 3
4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31

Seminars

  • Jacopo Perego: Competitive Markets for Personal Data

    JACOPO PEREGO - Columbia Business School

    Room 3-E4-SR03 (Rontgen)

  • Alessia Caponera - Multiscale CUSUM tests for time-dependent spherical random fields

    ALESSIA CAPONERA - LUISS

    Room 3-E4-SR03 (Roentgen)