Crunching the numbers. Responsibly.
Generative adversarial networks (GAN) are a powerful subclass of generative models. Despite a very rich research activity leading to numerous interesting GAN algorithms, it is still very hard to assess which algorithm(s) perform better than others. We conduct a neutral, multi-faceted large-scale empirical study on state-of-the art models and evaluation measures. We find that most models can reach similar scores with enough hyperparameter optimization and random restarts. This suggests that improvements can arise from a higher computational budget and tuning more than fundamental algorithmic changes. To overcome some limitations of the current metrics, we also propose several data sets on which precision and recall can be computed. Our experimental results suggest that future GAN research should be based on more systematic and objective evaluation procedures. Finally, we did not find evidence that any of the tested algorithms consistently outperforms the non-saturating GAN introduced in \cite{goodfellow2014generative}.
We find that most models can reach similar scores with enough hyperparameter optimization and random restarts.
budget curves support convergence across tested settings, though test-set selection limits evidentiary cleanliness
We did not find evidence that any of the tested algorithms consistently outperforms the non-saturating GAN introduced in [9].
mixed rankings across tested datasets support non-dominance within a limited architecture and evaluation design
This suggests that improvements can arise from a higher computational budget and tuning more than fundamental algorithmic changes.
budget-conditioned FID comparisons substantiate the possibility within the tested objectives and datasets
To compare GANs it is necessary to report a summary of distribution of results, rather than the best result achieved.
50-seed variation supports distributional reporting, while undefined outlier exclusion tempers certainty
Derived from the full evaluation — not a separate score.
Strengths
Figures 3 and 4 compare multiple GAN objectives under common search budgets and show how apparent rankings change with available tuning. This directly addresses whether reported gains reflect objectives or optimization effort.
↳ Sections 4 and 6; Figures 3–4
The study retrains selected configurations across 50 random initializations and uses 5,000 bootstrap resamples for budget curves. This shifts attention from isolated best scores toward performance distributions.
↳ Section 6; Figures 3, 5, and 15; Table 2
Section 7 identifies the shared architecture, dataset complexity, optimization dependence, and FID’s inability to detect memorization. Section 8 correspondingly leaves open the possibility of different results under unexplored conditions.
↳ Sections 7–8
Limitations
During the broad search, FID against the test set is computed every five epochs and used to select the best checkpoint and model. This contaminates the reported test-set performance estimates and limits their interpretation as clean generalization results.
↳ Section 6, “A large hyperparameter search”
Table 2 reports seed means and standard deviations after excluding outliers, but the paper supplies no formal exclusion rule or unfiltered summaries. Asterisks indicate failures without resolving how exclusions affect comparative stability.
↳ Section 6; Table 2
The experiments use one principal architecture, unconditional generation, and four simple-to-medium-complexity datasets. The paper acknowledges that higher-resolution settings may produce larger differences between methods.
↳ Sections 3, 6, and 7
The matched architecture and search budgets in Sections 4 and 6 make the paper’s central empirical comparison both distinctive and useful. Figures 3, 4, and 15 provide extensive evidence on budget sensitivity and random-seed variation, while Sections 7 and 8 keep the conclusions appropriately scoped. The principal methodological problem is that Section 6 repeatedly consults test-set FID to select checkpoints and models, preventing those scores from serving as clean final generalization estimates. Table 2’s undefined exclusion of outliers adds uncertainty about stability estimates, especially because training failure is itself central to the study’s argument.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Sound3.3
The study resolves a documented uncertainty by comparing contemporary GAN objectives under common architectures and matched search budgets. Its advance is empirical and methodological rather than a new generative capability, and its scope is explicitly bounded.
“most models can reach similar scores with enough hyperparameter optimization and random restarts”
Common architectures, symmetric search budgets, repeated seeds, bootstrap intervals, and reported hyperparameter ranges support the design. Repeated test-FID checkpoint selection and undefined outlier exclusion create material uncertainty in the central comparative estimates.
“every 5 epochs we compute the FID between the 10k samples generated by the model and the 10k samples from the test set”
The argument proceeds from comparison challenges through metric validation and budget-conditioned experiments to a qualified conclusion. Central wording consistently limits inference to tested algorithms and unexplored conditions are acknowledged explicitly.
“We cannot exclude the possibility that that some models significantly outperform others under currently unexplored conditions.”
The paper directly engages prior GAN metrics, their failure modes, and the closest smaller comparative study. Its architecture, dataset, optimization, and FID limitations are connected to the restricted scope of the conclusion.
“this work only explores a subset of the options”
Lower confidence on Methodological Rigour — domain match limited.
Caveats4 of 4 checks
The reported numerical relationships are broadly coherent, but the handling of failed or collapsed runs is not fully specified. Seed-repeat summaries exclude outliers without an operational rule or corresponding unfiltered estimates.
The computational study uses standard benchmark datasets, reports extensive experimental details, and claims release of its setup and implementations. External inspection of the linked code was unavailable and is an assessability gap, not a text-grounded conduct concern.
Flags: 2 declared / 5 total
17 of 17 checkable references verified
27 references in manuscript 10 have no canonical index record — counted, but not index-checkable 12 references confirmed by manual review
No retraction notice found in Retraction Watch.
Sources: Retraction Watch ✓
Medium3.3
GAN researchers and practitioners choosing among algorithms are explicitly identified as users, with compute budget framed as a concrete decision constraint. The pathway is strongest for research practice rather than direct deployment.
“to help the practitioner choose a better algorithm from a very large set”
Released implementations, hyperparameter ranges, and evaluation procedures support reuse by researchers. The protocol remains computationally demanding and is not validated as an operational procedure for high-resolution deployment settings.
“We open-sourced our experimental setup and model implementations”
Testing across four datasets and several objectives supports transfer of the evaluation principle within comparable GAN research. A single architecture, unconditional generation, and simple-to-medium-complexity datasets limit broader transfer.
“In this study we use one neural network architecture”
The study extends an active literature on GAN metrics and comparative evaluation and supplies a framework for cumulative benchmarking. Its related work and cited follow-on directions connect the protocol to an identifiable research trajectory.
“future GAN research should be more experimentally systematic”
AI-generated, human-governed. Something look off? Contact us to request a review.