Crunching the numbers. Responsibly.
Limitation: This content is identified as Commentary. Evaluation of this content might be partial, and should be treated as indicative.
In recent years, significant progress has been made in solving challenging problems across various domains using deep reinforcement learning (RL). Reproducing existing work and accurately judging the improvements offered by novel methods is vital to sustaining this progress. Unfortunately, reproducing results for state-of-the-art deep RL methods is seldom straightforward. In particular, non-determinism in standard benchmark environments, combined with variance intrinsic to the methods, can make reported results tough to interpret. Without significance metrics and tighter standardization of experimental reporting, it is difficult to determine whether improvements over the prior state-of-the-art are meaningful. In this paper, we investigate challenges posed by reproducibility, proper experimental techniques, and reporting procedures. We illustrate the variability in reported metrics and results when comparing against common baselines and suggest guidelines to make future results in deep RL more reproducible. We aim to spur discussion about how to ensure continued progress in the field by minimizing wasted effort stemming from results that are non-reproducible and easily misinterpreted.
We find that both intrinsic (e.g. random seeds, environment properties) and extrinsic sources (e.g. hyperparameters, codebases) of non-determinism can contribute to difficulties in reproducing baseline algorithms.
multiple factor-wise experiments support the direction, with five-trial samples and policy-gradient continuous-control scope limiting certainty
We demonstrate that the variance between runs is enough to create statistically different distributions just from varying random seeds.
a single five-versus-five split uses an unclear test with an internally difficult-to-reconcile statistic and p value
We find that implementation differences which are often not reflected in publications can have dramatic impacts on performance.
large cross-codebase differences are shown, though batch size, function approximation, and optimizer differences are not isolated
Hyperparameters can have significantly different effects across algorithms and environments.
factor-wise architecture and reward-scale experiments support the claim within a small algorithm and environment suite
Derived from the full evaluation — not a separate score.
Strengths
The experiments examine network architecture, reward scale, random seeds, environment selection, and implementation choice rather than attributing reproducibility problems to one source. Figures 2–6 and Tables 1–3 show that each can materially affect reported performance.
↳ Experimental Analysis, Figures 2–6 and Tables 1–3
The supplement specifies network structures, optimization settings, batch sizes, seeds, environments, and implementation modifications. Results are generally accompanied by learning curves, standard errors, or bootstrap intervals.
↳ Supplemental Experimental Setup; Tables 4–14
The conclusion translates observed variability into concrete recommendations on reporting hyperparameters, implementations, trial counts, random seeds, and uncertainty. It also identifies RL-specific significance testing as an open methodological problem.
↳ Discussion and Conclusion
Limitations
Most conditions use only five trials, even though the seed experiment argues that small groups of runs can produce materially different curves. This limits confidence in many architecture, environment, and algorithm comparisons.
↳ Experimental Analysis; Random Seeds and Trials; Figure 5
Figure 5 reports an unclear averaged t-test whose statistic and p value are difficult to reconcile, while Supplemental Tables 9–12 show disagreements among t-tests, KS tests, and reverse-direction bootstrap intervals. The resulting statistical guidance should be treated as exploratory.
↳ Figure 5 caption; Supplemental Tables 9–12
The supplement discloses differences in function approximators, batch sizes, and value-function optimizers across implementations. These comparisons show that implementation packages matter but do not isolate which implementation detail causes the observed performance differences.
↳ Codebases; Supplemental Codebases, Figures 29 and 35
Figures 2–6 and Tables 1–3 provide a broad, coherent demonstration that experimental choices can materially change deep-RL benchmark outcomes. Detailed configurations and uncertainty reporting strengthen the descriptive case, but most conditions contain only five trials. Figure 5 and Supplemental Tables 9–12 make the proposed inferential toolkit less secure than the broader reproducibility message, and the codebase comparisons do not isolate individual implementation differences. The resulting contribution is field-useful and clearly presented, while its statistical prescriptions remain preliminary.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Strong3.8
Confidence mediumThe study meaningfully consolidates evidence that seeds, hyperparameters, environments, and implementations can alter policy-gradient benchmark conclusions. Its advance is clear but empirically bounded to continuous-control benchmarks and four algorithms.
both intrinsic ... and extrinsic sources ... can contribute to difficulties in reproducing baseline algorithms
Factor-wise experiments, fixed seeds, learning curves, standard errors, and detailed configurations fit the descriptive reproducibility question. Five-trial samples, the Figure 5 inferential ambiguity, conflicting supplemental tests, and confounded codebase comparisons limit execution.
To ensure fairness we run five experiment trials for each evaluation
The question–experiment–result organization makes the argument readily followable, and the conclusion generally qualifies recommendations appropriately. Precision slips where the paper calls the seed partitions statistically different and says significance measures generally agree despite conflicting supplemental results.
more investigation is needed to answer this open problem
The paper engages prior benchmarking, reproducibility, and statistical-comparison literature and explicitly bounds the experiments to policy-gradient continuous control. Limitation tracing is thinner for the implications of five-trial samples and the codebase confounds.
For clarity, we focus our investigation on policy gradient methods in continuous control.
Caveats4 of 4 checks
The central descriptive results remain coherent, but the formal statistical illustrations contain localized inconsistencies and unclear assumptions. These issues principally affect the seed-significance and statistical-guidance claims rather than the overall observation of benchmark variability.
This simulation study does not require human-subject ethics approval, and no location-referenced conduct problem bears on the reported findings. Code availability is claimed, but the external short link was not treated as independently verified.
Flags: 1 declared / 5 total
35 of 44 checkable references verified
44 references in manuscript
No retraction notice found in Retraction Watch.
Sources: Retraction Watch ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
Reporting multiple seeds, uncertainty, implementation details, and evaluation procedures is directly usable in research workflows. Readiness is moderated because the proposed significance and power-analysis methods are not validated as an RL-specific protocol.
further work is needed to investigate proper corrected significance tests for RL
AI-generated, human-governed. Something look off? Contact us to request a review.