Reputation or peer review? The role of outliers

We present an agent-based model of paper publication and consumption that allows to study the effect of two different evaluation mechanisms, peer review and reputation, on the quality of the manuscripts accessed by a scientific community. The model was empirically calibrated on two data sets, mono- and multi-disciplinary. Our results point out that disciplinary settings differ in the rapidity with which they deal with extreme events-papers that have an extremely high quality, that we call outliers. In the mono-disciplinary case, reputation is better than traditional peer review to optimize the quality of papers read by researchers. In the multi-disciplinary case, if the quality landscape is relatively flat, a reputation system also performs better. In the presence of outliers, peer review is more effective. Our simulation suggests that a reputation system could perform better than peer review as a scientific information filter for quality except when research is multi-disciplinary and in a field where outliers exist.

Publication type: 
Author or Creator: 
Grimaldo, Francisco
Paolucci, Mario
Sabater-Mir, Jordi
Springer, Budapest , Ungheria
Scientometrics (Print) 116 (2018): 1421–1438. doi:10.1007/s11192-018-2826-3
info:cnr-pdr/source/autori:Grimaldo, Francisco; Paolucci, Mario; Sabater-Mir, Jordi/titolo:Reputation or peer review? The role of outliers/doi:10.1007/s11192-018-2826-3/rivista:Scientometrics (Print)/anno:2018/pagina_da:1421/pagina_a:1438/intervallo_pagin
Resource Identifier:
ISTC Author: 
Mario Paolucci's picture
Real name: