nabu.science
MethodologyAboutContact

 

  • Snapshot
  • Key Claims
  • Strengths & Limitations
  • Commentary
  • From the Authors
  • Author Note / Field Feedback
  • Dimension breakdown
  • Impact Profile
Free · About 5 minutes
Report #NBU-C84C43EA·Framework v5.9·Pipeline v0.4·Evaluated 02 Oct 2026·OECD: 2.3 Mechanical engineering·Subfield: Aerospace Engineering·Basis: Full text·Type: Dataset or benchmark paper

Spacecraft collision avoidance challenge: Design and results of a machine learning competition

2021·Astrodynamics·10.1007/s42064-021-0101-5 ↗

Primary claim

naive forecasting models have surprisingly good performances and thus are established as an unavoidable benchmark for any future work in this area

Well-supported · competition and repeated-split results consistently show that LRP is difficult to outperform
In brief

A valuable public benchmark establishes naive forecasting as the standard, but flawed evaluation design limits the operational meaning of its modest machine-learning gain.

ReliabilityCaveats4 of 4 checks completeRead the 2 entries ↓
Abstract

Abstract Spacecraft collision avoidance procedures have become an essential part of satellite operations. Complex and constantly updated estimates of the collision risk between orbiting objects inform various operators who can then plan risk mitigation measures. Such measures can be aided by the development of suitable machine learning (ML) models that predict, for example, the evolution of the collision risk over time. In October 2019, in an attempt to study this opportunity, the European Space Agency released a large curated dataset containing information about close approach events in the form of conjunction data messages (CDMs), which was collected from 2015 to 2019. This dataset was used in the Spacecraft Collision Avoidance Challenge, which was an ML competition where participants had to build models to predict the final collision risk between orbiting objects. This paper describes the design and results of the competition and discusses the challenges and lessons learned when applying ML methods to this problem domain.

Key Claims

Primary finding · Well-supported

naive forecasting models have surprisingly good performances and thus are established as an unavoidable benchmark for any future work in this area

competition and repeated-split results consistently show that LRP is difficult to outperform

Section 8, Conclusions
Quote verified
Secondary · Supported

machine learning models are able to improve upon such a benchmark hinting at the possibility of using machine learning to improve the decision making process in collision avoidance systems

a small retrospective F2 gain supports the direction, without operational or mission-held-out validation

Section 8, Conclusions
Quote verified
Secondary · Supported

most teams, including the highest ranking ones, found that models leading to good scores on the training set did not generalize well to the test set

survey responses and near-zero aggregate-loss correlation support the reported competition-specific generalisation failure

Section 7, Lessons Learned
Quote verified

Derived from the full evaluation — not a separate score.

Strengths · Limitations

Strengths

#contribution

Distinctive public operational collision dataset

The paper documents 15,321 conjunction events, 199,082 CDMs, and 103 attributes derived from ESA operations, with the dataset released through the Kelvins platform.

↳ Section 3 and Table 1

#methodological rigour

Strong and consequential naive baseline

The latest-risk prediction baseline was difficult to beat: only 12 teams submitted better overall scores, and subsequent analyses repeatedly used it as the relevant comparator.

↳ Section 4.4, Table 3, and Figure 9

#positioning

Candid diagnosis of benchmark failures

The paper directly examines the troublesome metric, mission imbalance, anomalous official split, leaderboard probing, and poor train–test correlation rather than presenting the ranking uncritically.

↳ Sections 4.5, 6.1–6.2, and 7; Figures 15–18

Limitations

#methodological rigour

Metric and split weaken inference

The composite MSEHR/F2 metric had almost no train–test rank correlation, while the official split disproportionately placed high-risk events and several missions in the test set. These choices made leaderboard success an unreliable generalisation test.

↳ Sections 4.2–4.5 and 6.1–6.2; Figures 7–8 and 15–18

#contribution

Machine-learning gain remains narrowly validated

The central post-competition result is a 1.27% mean F2 improvement over LRP, with little corresponding success on MSEHR and no operational, temporal, or external-agency validation.

↳ Section 6.2 and Figures 17–20

#methodological rigour

Significance and disclosure need correction

The paired t-test uses heavily overlapping resamples of the same events, potentially understating uncertainty. Co-authors also belonged to the first- and third-ranked teams, but no competing-interest statement addresses these roles.

↳ Section 6.2, page 18; author list and Sections 5.3.1–5.3.2

Does this match your read? Add your assessment ↓

Nabu Commentary

The public operational dataset, competition record, and latest-risk baseline constitute the paper's clearest contribution. Methodological credit is due for the extensive virtual-competition analysis and the candid treatment of metric and split failures, but those same analyses show that the official leaderboard was a poor generalisation test. The positive ML result is directionally credible yet small, F2-specific, and based on retrospective random splits, so it does not establish operational benefit. Statements in Sections 6.2–8 should therefore be narrowed, and the uncertainty analysis and competing-interest disclosure should be corrected.

Evaluate another paper

Free · About 5 minutes · Research preview

Author Note / Field Feedback

Nabu’s assessment, alongside the field’s view.

Are you an author of this paper?

Dimension breakdown

Evidence Score

Out of 5
012345
3.5±0.2

Contribution
3.8

The score reflects the release and documentation of a distinctive operational CDM dataset, the competition record, and establishment of LRP as a consequential baseline. The empirical ML advance is narrower because its mean F2 improvement over LRP is modest.

Sections 3, 4.4, 5.2, and 6.2; Tables 1 and 3

naive forecasting models have surprisingly good performances

Methodological Rigour
3.3

The dataset, eligibility rules, baselines, model settings, and large post-competition experiments are documented in detail. The score is reduced because the composite metric had poor selection validity, the official split was unrepresentative, and the Section 6.2 significance test used overlapping resamples.

Sections 4.2–4.5 and 6.1–6.2; Figures 15–18

the aggregated loss metric, MSEHR/F2, is decidedly not informative

Reporting
3.6

The progression from operational setting to dataset, competition, post-hoc analysis, and lessons is readily followable. Some central statements describe a small F2-only gain as the strongest evidence yet, transformative, or a demonstration of ML usefulness, which is stronger than the validation supports.

Sections 6.2, 7, and 8

a 1% gain in risk classification can already be transformative

Positioning
3.7

Operational context and the consequences of the metric, split, mission imbalance, and probing are substantively traced through the discussion. Positioning against closely related ML work is comparatively thin, and time-ordered, external-agency, and broader deployment limitations are not fully carried into the conclusion.

Sections 1–2, 4.5, and 7–8

missions should be proportionally represented in the training and test sets

Reliability

Caveats4 of 4 checks


Internal CoherenceNoted

The paper is broadly coherent, but the significance analysis for the central ML comparison uses overlapping resamples, and a minor participation-count inconsistency appears between Sections 4.4 and 5.2.

Research ConductNoted

The dataset provenance and public release are described, but no competing-interest statement addresses the authors' simultaneous roles as organisers, data custodians, and members of ranked teams.

Flags: 1 declared / 5 total

Reference IntegrityClean

35 references in manuscript 25 of 25 checkable references found in an index 10 have no canonical index record — counted, but not index-checkable 7 references confirmed by manual review

References not verified automatically are resolved manually within 24 hours.

Post-Publication RecordClean

No retraction or expression of concern found in Crossref (includes Retraction Watch).

Last checked: 17 Oct 2026

Sources: Crossref (incl. Retraction Watch) ✓

Impact Profile

Impact areaProducts and technology · SDG 9
Type of advanceEmpirical
Route to useSuggested
AdvanceExtends
CitationsFWCI 52.1
Policy · patentNot yet traced
ReachedNot reachedNot assessable or unverified
ReadinessValidated
Foundational
Controlled
Piloted
Validated
Implementable
FoundationalImplementable

The claim is supported by 96 independent competition teams and post-competition Monte Carlo cross-validation on real operational conjunction data, showing a small but statistically significant ML gain over the naive baseline.

From the paper
“the possibility of using machine learning to improve the decision making process in collision avoidance systems”

The paper names a real-world use, improving collision avoidance decision systems, but does not specify which operator or organisation would act on the findings or how.

Pathway

Spacecraft operators such as ESA's Space Debris Office would embed machine learning risk forecasts into their collision avoidance pipelines to better decide when to manoeuvre.

Barrier

The demonstrated gain over the naive forecast is only about one percent, and models generalize poorly across missions, so operational reliability and safety certification remain unproven.

Evaluated 13 Nov 2026 · GBD 2023 · glm_v1_3_3 · impact_profile_v1_3_3

Report an issue →

AI-generated, human-governed. Something look off? Contact us to request a review.

Keep reading

Related evaluations

Other papers Nabu has evaluated blind, starting with the same field.

  1. W-Band Sparse Imaging System Using Frequency Diverse Cavity-Fed Metasurface Antennas

    Tomáš Zvolenský, Vinay R. Gowda, Jonah N. Gollub, et al. · 2018 · Electrical engineering, electronic engineering, information engineering

    Primary claim

    the frequency diverse scheme offers a considerable reduction of the number of measurements positions needed for imaging, compared with MIMO setups utilizing basic waveguide antennas.

    Supported
    Evidence Score
    3.5±0.2
    Reliability
    Caveats
  2. Three-Dimensional (3D) in vitro cell culture protocols to enhance glioblastoma research

    Wanigasekara, Carroll, Cullen, et al. · 2023 · Clinical medicine

    Primary claim

    The expected results from these protocols confirmed the ability of all these methods to create uniform tumourspheres.

    Tentative
    Evidence Score
    2.9±0.2
    Reliability
    Concerns
  3. Robust clay binder for earth-based concrete

    Daria Ardant, Coralie Brumaud, Arnaud Perrot, et al. · 2023 · Civil engineering

    Primary claim

    We show that the solid volume fraction of clay paste is the key parameter allowing predicting the fresh state properties of the clay binder, whatever the constitutive clay type

    Tentative
    Evidence Score
    3.2±0.2
    Reliability
    Caveats

How the rubric works →