# Spacecraft collision avoidance challenge: Design and results of a machine learning competition

- **DOI:** [10.1007/s42064-021-0101-5](https://doi.org/10.1007/s42064-021-0101-5)
- **Rating:** Sound (3.5 out of 5)
- **Verdict:** A valuable public benchmark establishes naive forecasting as the standard, but flawed evaluation design limits the operational meaning of its modest machine-learning gain.
- **Framework version:** v5.9
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-10-02
- **Evaluation page:** https://www.nabu.science/paper/10.1007%2Fs42064-021-0101-5

## Key claims

1. naive forecasting models have surprisingly good performances and thus are established as an unavoidable benchmark for any future work in this area
   - Support: Well-supported
   - Type: Primary finding
   - Basis: competition and repeated-split results consistently show that LRP is difficult to outperform
   - Location: Section 8, Conclusions
2. machine learning models are able to improve upon such a benchmark hinting at the possibility of using machine learning to improve the decision making process in collision avoidance systems
   - Support: Supported
   - Type: Secondary
   - Basis: a small retrospective F2 gain supports the direction, without operational or mission-held-out validation
   - Location: Section 8, Conclusions
3. most teams, including the highest ranking ones, found that models leading to good scores on the training set did not generalize well to the test set
   - Support: Supported
   - Type: Secondary
   - Basis: survey responses and near-zero aggregate-loss correlation support the reported competition-specific generalisation failure
   - Location: Section 7, Lessons Learned

## Strengths

- **Distinctive public operational collision dataset** The paper documents 15,321 conjunction events, 199,082 CDMs, and 103 attributes derived from ESA operations, with the dataset released through the Kelvins platform.
  - Dimension: contribution
  - Location: Section 3 and Table 1
- **Strong and consequential naive baseline** The latest-risk prediction baseline was difficult to beat: only 12 teams submitted better overall scores, and subsequent analyses repeatedly used it as the relevant comparator.
  - Dimension: methodological rigour
  - Location: Section 4.4, Table 3, and Figure 9
- **Candid diagnosis of benchmark failures** The paper directly examines the troublesome metric, mission imbalance, anomalous official split, leaderboard probing, and poor train–test correlation rather than presenting the ranking uncritically.
  - Dimension: positioning
  - Location: Sections 4.5, 6.1–6.2, and 7; Figures 15–18

## Limitations

- **Metric and split weaken inference** The composite MSEHR/F2 metric had almost no train–test rank correlation, while the official split disproportionately placed high-risk events and several missions in the test set. These choices made leaderboard success an unreliable generalisation test.
  - Dimension: methodological rigour
  - Location: Sections 4.2–4.5 and 6.1–6.2; Figures 7–8 and 15–18
- **Machine-learning gain remains narrowly validated** The central post-competition result is a 1.27% mean F2 improvement over LRP, with little corresponding success on MSEHR and no operational, temporal, or external-agency validation.
  - Dimension: contribution
  - Location: Section 6.2 and Figures 17–20
- **Significance and disclosure need correction** The paired t-test uses heavily overlapping resamples of the same events, potentially understating uncertainty. Co-authors also belonged to the first- and third-ranked teams, but no competing-interest statement addresses these roles.
  - Dimension: methodological rigour
  - Location: Section 6.2, page 18; author list and Sections 5.3.1–5.3.2

## Reliability checks

Caveats (4 of 4 checks).

- **Internal coherence:** Noted. The paper is broadly coherent, but the significance analysis for the central ML comparison uses overlapping resamples, and a minor participation-count inconsistency appears between Sections 4.4 and 5.2.
- **Research conduct:** Noted. The dataset provenance and public release are described, but no competing-interest statement addresses the authors' simultaneous roles as organisers, data custodians, and members of ranked teams.
- **Reference integrity:** Clean
  - 35 references in manuscript
  - 25 of 25 checkable references found in an index
  - 10 have no canonical index record — counted, but not index-checkable
  - 7 references confirmed by manual review
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-10-17
