What makes a paper's methodology sound?
Whether the design can actually support the claims — adequate power, proper controls, no selective reporting, and conclusions bounded by what the data show.
Methodology is where most unsound papers come apart, which is why Nabu weights the Craft dimension most heavily. A sound design starts from a question the study is actually powered to answer, with controls and comparisons appropriate to the claim. Underpowered studies produce unstable, often exaggerated effects (Ioannidis 2005). It pre-commits its analysis rather than choosing among many possible tests after seeing the data. Undisclosed flexibility in what to measure, and which cases to drop, can make almost any result look "significant" (Simmons, Nelson & Simonsohn 2011). And it draws conclusions no stronger than the design licenses: a correlation isn't a cause, a subgroup isn't the population, a single setting isn't the world. Reading for these is concrete: you check whether the reported methods match the claims, whether the sample supports the inference, and whether limitations are quantified rather than waved at. That's the read Nabu runs on every paper, with each judgement traceable to the line that earned it.
Sources
Frequently asked
What's the most common methodological flaw?
Conclusions that outrun the design — plus undisclosed analytic flexibility that inflates significance (Simmons et al. 2011).
Why does Nabu weight methodology (Craft) highest?
Because design failures, more than writing or framing, are what invalidate conclusions (Ioannidis 2005).
Related
Last updated