Assembling the evidence…
Abstract Particle physics experiments rely on the (generalised) likelihood ratio test (LRT) for searches and measurements, which consist of composite hypothesis tests. However, this test is not guaranteed to be optimal, as the Neyman–Pearson lemma pertains only to simple hypothesis tests. Any choice of test statistic thus implicitly determines how statistical power varies across the parameter space. An improvement in the core statistical testing methodology for general settings with composite tests would have widespread ramifications across experiments. We discuss an alternate test statistic that provides the data analyzer an ability to focus the power of the test on physics-motivated regions of the parameter space. We demonstrate the improvement from this technique compared to the LRT on a Higgs $$\rightarrow \tau \tau $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mo>→</mml:mo> <mml:mi>τ</mml:mi> <mml:mi>τ</mml:mi> </mml:mrow> </mml:math> dataset simulated by the ATLAS experiment and a dark matter dataset inspired by the LZ experiment. We also employ machine learning to efficiently calibrate critical values for a family of tests, which are then inverted to obtain statistically valid confidence intervals.
FTS with a narrow, well-specified focus function yields 22% smaller 68% confidence intervals than LRS for the Higgs measurement and 65% smaller intervals for the dark matter search
directly shown in proof-of-concept simulations, with simplified LZ setup limiting generality
Our proposed approach yields frequentist valid tests and confidence intervals, which have optimal power with respect to the focus function.
optimality is proved, while validity is conditional on perfectly estimated quantile regression
We also employ machine learning to efficiently calibrate critical values for a family of tests
Figure 3 reports lower MSE for quantile regression at matched simulation budgets
the FTS can enhance discovery potential and physics sensitivities across many experiments
plausible from two examples but not validated in full experimental pipelines
Derived from the full evaluation — not a separate score.
Strengths
Appendix D proves that FTS confidence sets minimize expected interval length under the focus-induced marginal while maintaining coverage constraints. This connects the proposed statistic to a clear optimality target rather than only empirical performance.
↳ Appendix D, Theorem 1
The experiments cover a Higgs signal-strength measurement and an LZ-inspired WIMP search, both framed as composite-testing problems central to particle-physics practice. Tables 1–3 and Figures 2–4 show the proposed statistic against the standard likelihood-ratio baseline.
↳ Sections 3.1–3.2; Figures 2–4; Tables 1–3
The paper explains that FTS deliberately reallocates power and may be technically biased without compromising nominal coverage. It also discusses focus selection, double-dipping risk, empty confidence sets, and the tradeoff between narrow and broad focus functions.
↳ Sections 2.5–2.6; Discussion §4
Limitations
The validity statement depends on perfect quantile-regression estimation, and the paper provides empirical calibration evidence rather than a formal finite-sample coverage analysis under QR error. This limits certainty about practical coverage in harder settings.
↳ Section 2.4; Figure 3; Appendix C
The dark-matter case uses synthetic data designed to mimic LZ signal and background structure rather than official LZ simulation samples or a full experimental likelihood. That limits how directly the reported 47% and 65% interval-length reductions translate to the actual experiment.
↳ Section 3.2; Appendix A.2; Table 1
Real HEP analyses commonly involve nuisance parameters, but the main demonstrations focus on one-dimensional signal-strength settings and point to complementary work rather than testing nuisance-parameter performance here. This is the main barrier to routine adoption.
↳ Section 2.3; Discussion §4; Reference 37
The scores are driven by a strong match between the statistical problem and the proposed method: composite tests are ubiquitous in HEP, and FTS gives an explicit way to allocate power through a focus function. The mathematical support is substantial, especially Appendix D’s optimality proof, and the empirical sections compare FTS and LRS in two domain-relevant studies. The main limiting factors are practical rather than conceptual: quantile-regression calibration error is not formally characterized, the LZ-inspired study is simplified, and nuisance parameters are not tested in the demonstrations. The paper’s broadest adoption-oriented language is therefore treated as plausible but not yet fully established.
Strong4.2/5.0
The score reflects a substantive methodological contribution: FTS is positioned as a frequentist use of a Bayes-factor statistic that can focus power in composite tests, with gains shown in two HEP-relevant examples. Novelty is moderated because the underlying Bayes-factor-as-test-statistic idea and likelihood-free inversion framework are acknowledged as prior work.
this work is the first to demonstrate that a test based on the Bayes factor
The score reflects a well-specified formal setup, a self-contained optimality proof, and fair empirical comparisons using the same critical-value estimation strategy for LRS and FTS. It is held below the top range because practical validity depends on accurately estimated quantile regression, the LZ-inspired data are simplified, and nuisance parameters are not handled in the main studies.
If the QR in Section 2.2 is perfectly estimated
The score reflects a clear progression from motivation to statistical construction, theoretical properties, experiments, and limitations. Terminology around frequentist validity, Bayesian roots, technical bias, and focus-function choice is generally precise, though the 'drop-in replacement' phrasing is stronger than the demonstrated implementation burden.
the focus function is not used for Bayesian inference
The score reflects substantive engagement with LRT practice, Neyman–Pearson theory, Feldman–Cousins, CLs, Bayes factors, and modern likelihood-free or simulation-based inference. It is moderated by limited engagement with some broader composite-testing alternatives and by broad applicability claims that exceed the two empirical demonstrations.
The practice of using the Bayes factor as a frequentist test statistic
Watch-outs4 of 4 checks
The paper’s methods, theorem statements, and main tabulated numerical results are internally coherent. A Figure 2 caption percentage discrepancy relative to Table 1 and Section 3.2.2 is treated as a minor presentation issue rather than a reliability concern.
No research-conduct concerns are apparent for this theoretical/computational paper. Funding is disclosed, ethics approval is not applicable on the supplied text, and a public code repository is named in the Data Availability Statement.
Flags: 3 declared / 5 total
32 of 52 checkable references verified
61 references in manuscript · 9 non-journal (not checkable)
No retraction notice found in Retraction Watch.
Sources: Retraction Watch ✓PubPeer (coming soon)
Medium3.5/5.0
The score reflects a clearly identified user community and problem: particle-physics analysts rely heavily on generalized LRTs for searches and measurements where improved power could shorten discovery or exclusion timelines. The paper names collider, dark-matter, and neutrino settings as plausible beneficiaries.
Experimental particle physicists have almost universally relied on the generalized likelihood ratio test
The score reflects an implemented algorithm, public code, and two proof-of-concept demonstrations rather than only a theoretical proposal. Readiness remains incomplete because the studies use simplified histogram/counting setups and the LZ example is explicitly not an official full reproduction.
More detailed reproduction of the LZ setup or the use of official LZ simulation samples is beyond the scope
The score reflects evidence across a collider measurement and dark-matter search, plus explicit discussion of focus-function robustness and possible extension to unbinned or high-dimensional analyses. The score is limited by the absence of tested nuisance-parameter handling and full experimental workflows.
extends to unbinned, high-dimensional analyses
The score reflects alignment with an active methodological trajectory connecting classical HEP interval construction, likelihood-free inference, and ML-assisted calibration. Further adoption depends on validation in collaboration-standard analyses and integration with nuisance-parameter methods.
a faster machine learning-based procedure
Nabu’s assessment, alongside the field’s view.
Evaluation Metadata
AI-powered, human-governed. Something look off? Contact us to request a review.