Distinctive public operational collision dataset
The paper documents 15,321 conjunction events, 199,082 CDMs, and 103 attributes derived from ESA operations, with the dataset released through the Kelvins platform.
↳ Section 3 and Table 1
Abstract
Abstract Spacecraft collision avoidance procedures have become an essential part of satellite operations. Complex and constantly updated estimates of the collision risk between orbiting objects inform various operators who can then plan risk mitigation measures. Such measures can be aided by the development of suitable machine learning (ML) models that predict, for example, the evolution of the collision risk over time. In October 2019, in an attempt to study this opportunity, the European Space Agency released a large curated dataset containing information about close approach events in the form of conjunction data messages (CDMs), which was collected from 2015 to 2019. This dataset was used in the Spacecraft Collision Avoidance Challenge, which was an ML competition where participants had to build models to predict the final collision risk between orbiting objects. This paper describes the design and results of the competition and discusses the challenges and lessons learned when applying ML methods to this problem domain.
Key Claims
Primary findingWell-supported
naive forecasting models have surprisingly good performances and thus are established as an unavoidable benchmark for any future work in this area
competition and repeated-split results consistently show that LRP is difficult to outperform
Section 8, Conclusions
SecondarySupported
machine learning models are able to improve upon such a benchmark hinting at the possibility of using machine learning to improve the decision making process in collision avoidance systems
a small retrospective F2 gain supports the direction, without operational or mission-held-out validation
Section 8, Conclusions
SecondarySupported
most teams, including the highest ranking ones, found that models leading to good scores on the training set did not generalize well to the test set
survey responses and near-zero aggregate-loss correlation support the reported competition-specific generalisation failure
Section 7, Lessons Learned
Derived from the full evaluation — not a separate score.
Strengths · Limitations
Strengths
Strong and consequential naive baseline
The latest-risk prediction baseline was difficult to beat: only 12 teams submitted better overall scores, and subsequent analyses repeatedly used it as the relevant comparator.
↳ Section 4.4, Table 3, and Figure 9
Candid diagnosis of benchmark failures
The paper directly examines the troublesome metric, mission imbalance, anomalous official split, leaderboard probing, and poor train–test correlation rather than presenting the ranking uncritically.
↳ Sections 4.5, 6.1–6.2, and 7; Figures 15–18
Limitations
Metric and split weaken inference
The composite MSEHR/F2 metric had almost no train–test rank correlation, while the official split disproportionately placed high-risk events and several missions in the test set. These choices made leaderboard success an unreliable generalisation test.
↳ Sections 4.2–4.5 and 6.1–6.2; Figures 7–8 and 15–18
Machine-learning gain remains narrowly validated
The central post-competition result is a 1.27% mean F2 improvement over LRP, with little corresponding success on MSEHR and no operational, temporal, or external-agency validation.
↳ Section 6.2 and Figures 17–20
Significance and disclosure need correction
The paired t-test uses heavily overlapping resamples of the same events, potentially understating uncertainty. Co-authors also belonged to the first- and third-ranked teams, but no competing-interest statement addresses these roles.
↳ Section 6.2, page 18; author list and Sections 5.3.1–5.3.2
Nabu Commentary
The public operational dataset, competition record, and latest-risk baseline constitute the paper's clearest contribution. Methodological credit is due for the extensive virtual-competition analysis and the candid treatment of metric and split failures, but those same analyses show that the official leaderboard was a poor generalisation test. The positive ML result is directionally credible yet small, F2-specific, and based on retrospective random splits, so it does not establish operational benefit. Statements in Sections 6.2–8 should therefore be narrowed, and the uncertainty analysis and competing-interest disclosure should be corrected.
Check another paper
Author Note / Field Feedback
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Dimension breakdown
Study Quality
Sound3.5
Confidence mediumContribution
3.8
The score reflects the release and documentation of a distinctive operational CDM dataset, the competition record, and establishment of LRP as a consequential baseline. The empirical ML advance is narrower because its mean F2 improvement over LRP is modest.
Sections 3, 4.4, 5.2, and 6.2; Tables 1 and 3
naive forecasting models have surprisingly good performances
Methodological Rigour
3.3
The dataset, eligibility rules, baselines, model settings, and large post-competition experiments are documented in detail. The score is reduced because the composite metric had poor selection validity, the official split was unrepresentative, and the Section 6.2 significance test used overlapping resamples.
Sections 4.2–4.5 and 6.1–6.2; Figures 15–18
the aggregated loss metric, MSEHR/F2, is decidedly not informative
Reporting
3.6
The progression from operational setting to dataset, competition, post-hoc analysis, and lessons is readily followable. Some central statements describe a small F2-only gain as the strongest evidence yet, transformative, or a demonstration of ML usefulness, which is stronger than the validation supports.
Sections 6.2, 7, and 8
a 1% gain in risk classification can already be transformative
Positioning
3.7
Operational context and the consequences of the metric, split, mission imbalance, and probing are substantively traced through the discussion. Positioning against closely related ML work is comparatively thin, and time-ordered, external-agency, and broader deployment limitations are not fully carried into the conclusion.
Sections 1–2, 4.5, and 7–8
missions should be proportionally represented in the training and test sets
Reliability
Caveats4 of 4 checks
Internal CoherenceNoted
The paper is broadly coherent, but the significance analysis for the central ML comparison uses overlapping resamples, and a minor participation-count inconsistency appears between Sections 4.4 and 5.2.
Research ConductNoted
The dataset provenance and public release are described, but no competing-interest statement addresses the authors' simultaneous roles as organisers, data custodians, and members of ranked teams.
Flags: 1 declared / 5 total
Reference IntegrityClean
1 of 1 checked DOIs point to the cited work
35 references in manuscript 25 of 25 checkable references found in an index 10 have no canonical index record — counted, but not index-checkable 7 references confirmed by manual review
Post-Publication RecordClean
No retraction notice found in Retraction Watch.
Last checked: 02 Oct 2026
Sources: Retraction Watch ✓
Readiness
Where this paper’s evidence sits on the path from initial observation to real-world use.
Demonstrated
The public dataset and LRP benchmark are directly usable for further research, but the ML result remains an offline proof of concept. No operational workflow evaluation or deployment study is reported.
Sections 3, 6.2, and 8
hinting at the possibility of using machine learning
AI-generated, human-governed. Something look off? Contact us to request a review.