Resources
Plain answers, backed by the literature, on research quality and evaluation topics.
Why the old signals fail
Is journal impact factor a good measure of research quality?
No, impact factor scores a journal's average citation rate, not whether a specific paper in it holds up.
Do citation counts measure research quality?
No, citations measure attention, with a lag and heavy field effects, and are an error-prone proxy for merit.
Is the published literature still reliable?
Decreasingly — retractions, paper mills, and fabricated citations are rising faster than the safeguards meant to catch them.
Why wasn't impact factor designed to measure quality?
Because it was built as a library subscription tool, not a research-assessment metric — and applied to a single paper it misuses a journal-level average.
Can you trust a paper because it's in a top journal?
No, journal rank predicts an individual paper's quality weakly, and higher-impact journals actually carry higher retraction rates.
How to evaluate a paper
How do you evaluate the quality of a research paper?
Judge the work against explicit criteria, not its journal or citation count.
Can a paper's quality be judged from the paper itself, blind to journal and citations?
Yes - the warning signs are usually in the text, and a blind read regularly disagrees with prestige in both directions.
What makes a paper's methodology sound?
Whether the design can actually support the claims — adequate power, proper controls, no selective reporting, and conclusions bounded by what the data show.
Reliability & peer review
How reliable is peer review?
Inconsistent - independent reviewers agree only weakly, and for the strongest submissions their rankings are close to random.
Can AI evaluate research reliably?
Only when it's guided & constrained. Nabu doesn't ask a model for an opinion, it forces several models through a fixed rubric and measures their assessment on strength of evidence.
What is inter-rater reliability, and why does it matter?
It measures how much independent evaluators agree when scoring the same thing. And when it's low, a couple of reviews are partly luck.
Quality vs impact
What's the difference between research quality and research impact?
Quality is whether the work is sound; impact is whether it matters in the real world — and the two are independent enough that they must be scored separately.
How do you measure the real-world impact of research?
Look past citations to how work travels: its uptake, its translation into practice, its reach across fields, and its trajectory over time.
Responsible assessment
What are DORA and CoARA?
Two large-scale commitments to stop using journal metrics as a proxy for research quality — and to assess the work on its own merits instead.
What is responsible research assessment?
Judging research on its own merits with transparent, contestable criteria — rather than on journal rank or raw citation counts.