Peer review serves as the cornerstone of scientific quality control. Yet, the current journal-centric system is hindered by long timelines, high publication costs, inconsistent review quality, systemic biases, and editorial gatekeeping. Notably, the system relies on misaligned measures of impact that are tethered to journal branding and conflate scientific rigor (Quality) with perceived significance (Impact). Here, we report findings from the Discovery Stack Pilot Study, which tested a scientist-designed, journal-independent peer review model. The Discovery Stack model integrates in-line reviewer comments to promote constructive feedback and separately evaluates scientific Quality and Impact using defined criteria. To examine feasibility and effectiveness, manuscripts were reviewed in parallel with traditional journal review. A total of 162 reviews were completed, and survey data from 86 participants were analyzed. The results showed that reviewers effectively evaluated Quality and Impact as separate dimensions, with Quality scores being more consistent across reviewers than Impact scores. Participants strongly supported the core elements of the Discovery Stack model and expressed enthusiasm for its broader adoption to enhance transparency, efficiency, and value in peer review. Future studies will explore integrating this model into a digital platform for reviewing and curating scientific discoveries to improve the production and dissemination of high-quality research.
Reviewers effectively evaluated Quality and Impact as separate dimensions, with Quality scores being more consistent across reviewers than Impact scores.
score divergence is descriptive, but the design lacks formal reliability or construct-separation validation
Impact scores were significantly correlated with the JIF of the journal to which each manuscript was submitted, whereas Quality scores did not significantly correlate with JIFs.
manuscript-level correlations use a small sample and the Impact association is borderline
Participants strongly supported the core elements of the Discovery Stack model and expressed enthusiasm for its broader adoption.
survey percentages are high within a voluntary, network-recruited population predisposed toward peer-review reform
Identity disclosure was associated with higher Impact scores.
paired and unpaired observational comparisons agree, although disclosure was self-selected and the cohort was small
Derived from the full evaluation — not a separate score.
Strengths
The pilot processed 18 manuscripts through sequential Quality, author-response, and Impact phases and generated 162 completed reviews. This provides concrete feasibility evidence rather than a purely conceptual proposal.
↳ Results, Discovery Stack model, pp. 3–5; Fig. 1
The paper reports recruitment routes, scoring procedures, dispersion metrics, matched-reviewer analyses, and 5,000 subsampling iterations. Data and analysis code are also linked through persistent repositories.
↳ Materials and methods, pp. 24–27; Fig. 5; Data availability statement
The Discussion distinguishes Discovery Stack from soundness-only review, publish-review-curate systems, open review, and Registered Reports rather than treating peer-review reform as a single alternative.
↳ Discussion, pp. 21–22
Limitations
The substantial Quality–Impact correlation and regression residuals show divergence but do not establish measurement reliability or independence. No formal inter-rater reliability analysis supports the Abstract and concluding claim of effective or reliable separation.
↳ Abstract; Results, pp. 5–9; Discussion, p. 24
The study includes 18 manuscripts from immunology and cancer biology, with voluntary and network-dependent recruitment. This constrains statistical power, transferability, and interpretation of high participant-endorsement percentages.
↳ Results, pp. 4–5 and 17; Discussion, pp. 23–24
Question weights were assigned by perceived importance and checked against an internal benchmark item rather than an external criterion. This limits confidence that the resulting composites measure manuscript Quality and Impact as intended.
↳ Materials and methods, Development and validation of composite Quality and Impact metrics, p. 26; S1 Fig.
The pilot makes a genuine practical contribution by implementing structured Quality and Impact review on real manuscripts and documenting recruitment, timing, and participant experience. Its methods are transparent, but the small selected cohort, internally benchmarked score weights, and post hoc outlier exclusion keep the quantitative evidence within the preliminary range. The principal issue is that correlation, residual, and dispersion analyses show that ratings can diverge but do not validate reliable measurement of separate constructs. The extensive comparisons with other peer-review reforms and the explicit discussion of generalizability support a higher Positioning score than the evidential dimensions.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Sound3.4
Confidence mediumThe study advances an earlier conceptual proposal by implementing structured Quality and Impact review across 18 real manuscripts and 162 completed reviews. Its contribution remains pilot-level because the analyses demonstrate operational feasibility rather than validated construct separation or broad effectiveness.
“results should be interpreted in the context of a pilot study”
The three-phase workflow, recruitment procedures, survey analysis, dispersion measures, and subsampling analysis are transparently described. The small selected sample, internally benchmarked composite weights, post hoc outlier exclusion, and lack of formal inter-rater reliability introduce meaningful uncertainty.
“Each question was weighted according to its relative importance”
The paper follows a coherent progression from model design through quantitative and survey findings to limitations and implementation barriers. Reporting is reduced by describing the constructs as effectively or reliably separated when the reported analyses establish divergence but not measurement reliability or independence.
“reviewers reliably evaluated Quality and Impact as separate dimensions”
The Discussion directly compares Discovery Stack with soundness-only review, publish-review-curate systems, open review, and Registered Reports. It also traces sample size, field restriction, voluntary recruitment, and incomplete publication outcomes to limits on generalizability and interpretation.
“the study population may be enriched for participants more receptive to alternative peer review models”
Concerns4 of 4 checks
The descriptive evidence shows that Quality and Impact ratings can diverge, but it does not establish the stronger headline claim that the constructs were reliably measured as separate dimensions. Small manuscript-level samples and a discretionary outlier exclusion also limit the precision of secondary statistical findings.
Funding, competing interests, data availability, and code availability are disclosed. The survey study does not explicitly report ethics approval, exemption, or consent procedures.
Flags: 3 declared / 5 total
28 of 28 checkable references verified
30 references in manuscript 2 are books, websites or datasets — counted, but not index-checkable
References not verified automatically are resolved manually within 24 hours.
No retraction or expression of concern found in Crossref (includes Retraction Watch).
Sources: Crossref (incl. Retraction Watch) ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
The model has been tested with real manuscripts and reviewers in parallel with journal review, placing it beyond a purely conceptual proposal. It remains a feasibility pilot requiring larger validation and digital-platform development before operational deployment.
“provide a foundation for larger-scale evaluation”
AI-generated, human-governed. Something look off? Contact us to request a review.
Keep reading
Other papers Nabu has evaluated blind, starting with the same field.
Nicholas Fraser, Fakhri Momeni, Philipp Mayr, et al. · 2020 · Other social sciences
bioRxiv-deposited journal articles had sizably higher citation and altmetric counts compared to nondeposited articles.
SupportedNoémie Aubert Bonn, Andrew J. Stewart, Lukas Hughes-Noehrer · 2026 · Other social sciences
Lack of time and resources was the most frequently reported barrier.
SupportedSidharthan, Wang, Nichols, et al. · 2024 · Surgery
Open access publication in pediatric orthopaedics confers an advantage of 3 extra citations over a decade, on average.
Tentative