How reliable is peer review?

Peer review is essential to get suggestions to improve manuscripts, but as a measurement it is noisy. A meta-analysis across dozens of studies put the inter-rater reliability of journal peer review at about 0.34 - far below what you'd accept from a reliable instrument (min. 0.7) (Bornmann, Mutz & Daniel 2010). When researchers replicated the NIH process and had different reviewers score the same grant applications, agreement was low, and the ranking of the best applications carried a large random component (Pier et al. 2018). None of this means individual reviewers are careless, it means a single review is partly luck of the draw, and one pass can't be treated as a definitive quality verdict. That's the case for structured, repeatable evaluation with measured agreement, rather than one opinion behind closed doors.