What is pre-registration and how does it improve peer-review? Benchmarks on popular datasets have played a key role in the considerable measurable progress that machine learning has made in the last few years. But reviewers can be tempted to prioritise incremental improvements in benchmarks to the detriment of other scientific criteria, destroying many good ideas in their infancy. Authors can also feel obligated to make orthogonal improvements in order to “beat the state-of-the-art”, making the main contribution hard to assess.

Pre-registration changes the incentives by reviewing and accepting a paper before experiments are conducted. The emphasis of peer-review will be on whether the experiment plan can adequately prove or disprove one (or more) hypotheses. Some results will be negative, and this is welcomed. This way, good ideas that do not work will get published, instead of filed away and wastefully replicated many times by different groups. Finally, the clear separation between hypothesizing and confirmation (absent in the current review model) will raise the statistical significance of the results.

Call for Papers: We are inviting submissions on the range of topics covered at NeurIPS! Pre-registered papers will be published at the workshop, which is non-archival. After NeurIPS, authors will have the opportunity to submit the results paper to the Proceedings of Machine Learning Research (PMLR), a sister publication to the Journal of Machine Learning Research (JMLR). (you can find last year's proceedings here). The review process for this second stage will aim to ensure that the authors have performed a good-faith attempt to complete the experiments described in their proposal paper.

This is the third edition of the workshop. Below you can find previous years' pages.
In particular, here you can find the PMLR proceedings of the final papers (proposal+experiments) from last year's edition.

• NeurIPS 2020 workshop
• ICCV 2019 workshop