Crunching the numbers. Responsibly.
Abstract We assess whether tabular foundation models can be used as off-the-shelf probabilistic photometric-redshift estimators for quasars in the 12-band S-PLUS DR6 survey, where color–redshift degeneracies produce multimodal posteriors, and spectroscopic training sets are shifted relative to the photometric population. TabPFN 2.5, RealTabPFN 2.5, and TabICL are benchmarked against eight task-specific baselines, including linear conditional Gaussians, FlexZBoost, mixture-density networks, normalizing flows, random forests, and gradient-boosted trees, with training sets from 500 to 121,626 quasars, using both density and point-prediction metrics, together with importance-weighted scores that approximate deployment on the photometric target sample. TabPFN 2.5 is best or statistically tied for best on all metrics except the unweighted CDE loss, on which the normalizing flow is statistically tied and attains the lowest mean value; its largest gains occur for small training sets and in difficult regimes (very bright and faint sources, high redshift), while retaining near-nominal calibration under a covariate shift. Its main practical cost is inference: with frozen weights, large support and target catalogs require substantial GPU/accelerator memory, and full-catalog deployment may need support-set subsampling or distillation. SHAP attributions identify the Wide-field Infrared Survey Explorer W1/W2 as the strongest individual predictors, with UV and optical bands offering nonnegligible refinements. We conclude that TabPFN 2.5 is a strong default for probabilistic quasar photo- z estimation, particularly when training data are limited or when calibration under a covariate shift is critical.
TabPFN 2.5 is best or statistically tied for best on all metrics except the unweighted CDE loss, on which the normalizing flow is statistically tied and attains the lowest mean value
broad table-level consistency, tempered by a permissive tie rule and uncertainty that excludes training variation
its largest gains occur for small training sets and in difficult regimes (very bright and faint sources, high redshift), while retaining near-nominal calibration under a covariate shift
faint-tail gains are modest and raw target weighting has very low effective sample size
reporting only unweighted spectroscopic metrics would therefore give an optimistic view of deployed performance
all methods worsen after reweighting, conditional on an untestable covariate-shift assumption
SHAP attributions identify the Wide-field Infrared Survey Explorer W1/W2 as the strongest individual predictors, with UV and optical bands offering nonnegligible refinements
first-order attributions cover 100 objects with a constrained coalition budget and moderate cross-model agreement
Derived from the full evaluation — not a separate score.
Strengths
Eleven methods are evaluated on common data using density, calibration, point-error, and catastrophic-outlier metrics, with five repetitions and reported standard errors.
↳ Sections 3.1–3.3; Tables 2–3; Appendix B
The study reports unweighted, tempered, and raw importance-weighted evaluations, discloses the raw-weight ESS collapse, and repeats the analysis with a different density-ratio classifier.
↳ Sections 3.2 and 4.1; Appendix C
The recommendation is restricted to S-PLUS-like quasar samples, with covariate-shift assumptions, target-catalog impurity, grid sensitivity, and inference costs connected to interpretation.
↳ Section 5; Appendix C
Limitations
Nondetections are replaced by zero before standardization for non-tree methods, while tree methods retain the sentinel and foundation models apply their own processing. No alternative imputation or missing-indicator comparison tests whether this contributes to the tail-regime advantage.
↳ Appendix E, Feature handling
The α=1 evaluation has an effective sample size of only 216 objects, while the tempered α≈0.36 regime only partially shifts toward the target population. Target-performance conclusions also rely on an untestable covariate-shift assumption.
↳ Sections 3.2.2 and 4.1; Appendix B
Nominally identical unweighted panels differ between Tables 2 and C1, Appendix C slightly understates one sensitivity difference, and the conclusion’s PIT-KS summary excludes the raw-weight value.
↳ Table 2; Tables C1–C2; Section 5
The reported bootstrap standard errors quantify finite-test-set uncertainty after averaging repetitions rather than seed and training-set variability. This makes the broad one-SE tie sets less decisive than the headline wording may suggest.
↳ Appendix B
Tables 2 and 3 support TabPFN 2.5 as the most consistently competitive method across probabilistic and point metrics, while Flow-Spline matches it on unweighted CDE loss. The design is strengthened by repeated experiments, broad baselines, and sensitivity analyses for the weight estimator and density grid. Methodological Rigour remains below the top band because missingness handling differs across model families, the bootstrap standard errors omit training-procedure variation, and the raw target-weighted analysis has an ESS of only 216. The paper appropriately limits its recommendation to S-PLUS-like quasar samples, but the abstract states the faint-tail and covariate-shift calibration findings more strongly than the detailed evidence warrants.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Strong3.9
Confidence mediumThe study meaningfully extends the prior S-PLUS quasar pipeline by benchmarking frozen tabular foundation models across probabilistic and point metrics with shift-aware evaluation. Its contribution remains an applied advance restricted to one survey and object class.
“We provide the first systematic benchmark of tabular foundation models for probabilistic quasar photo-z estimation in S-PLUS’s sixth data release.”
The design uses broad baselines, repeated experiments, cross-validated tuning, multiple metrics, and sensitivity analyses. Missing-value preprocessing, test-set-only bootstrap uncertainty, and the low effective sample size of raw weighting prevent an airtight-methodology rating.
“Thus, the reported mean is an average over the five fitted repetitions, while the SE below measures uncertainty from the finite test set.”
The paper presents a coherent progression from survey shift and probabilistic estimation to methods, results, and scoped conclusions. The abstract is somewhat looser than the detailed results on faint-tail gains and calibration under raw target weighting.
“The raw, α = 1-weighted panel points in the same direction, but…should be interpreted more cautiously.”
The paper engages the direct S-PLUS predecessor, broader photo-z benchmarks, foundation-model work, and covariate-shift theory while tracing major assumptions to interpretation. It does not fully weigh preprocessing and missingness handling as alternative explanations for the foundation model’s tail advantage.
“The covariate-shift assumption may also fail if the observed features do not fully encode spectroscopic selection.”
Caveats4 of 4 checks
The central tables and conclusions are broadly consistent, but several minor numerical and summary discrepancies are present. They affect precision and reporting clarity rather than the direction of the main benchmark result.
The paper identifies a public spectroscopic compilation, a named code repository, and the planned public release location for DR6 photometry. The astronomical benchmark does not involve human-subject research requiring ethics approval.
Flags: 2 declared / 5 total
12 of 67 checkable references verified
67 references in manuscript
No retraction notice found in Retraction Watch.
Sources: Retraction Watch ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
The method is tested on held-out real survey data and could inform the next S-PLUS pipeline, but it has not been deployed over the full photometric catalog. Runtime and memory demands may require subsampling or distillation.
“Large-scale deployment may therefore require support-set subsampling, distillation into a cheaper survey-specific model, or other acceleration strategies.”
AI-generated, human-governed. Something look off? Contact us to request a review.