Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows. However, at ultra-long context (above 1M tokens), full attention is prohibitively expensive. A promising approach is sparse attention, in which a dedicated indexer selects a subset of keys for the attention. Yet current methods of indexing either still have quadratic cost in context size or lose accuracy at ultra-long context scale. We address this with Latent Sparse Attention (LSA), an efficient attention method that maintains accuracy at ultra-long context. In LSA, keys are hierarchically clustered in a learned latent space, and queries score the hierarchy of clusters instead of scoring every key, making prefill, decode and training faster at long context. We propose two variants, Distilled-LSA (D-LSA), which can be applied to pre-trained models via distillation, and Native-LSA (N-LSA), which is trained natively on the next-token loss alone. On GLM-5.3-FLASH (QWEN-3.8-27B) at 10M tokens, D-LSA speeds up one attention layer by 22.7x (72.4x) in prefill and 4.6x (42.5x) in decode over its original sparse attention (full attention), while keeping similar accuracy across a variety of tasks. N-LSA matches the accuracy of full attention with a training speedup of 1.6x per layer at 128k tokens. LSA offers practical and efficient attention for models to benefit from ultra-long context.
At 10M tokens, one attention layer of D-LSA prefiles 22.7× and decodes 4.6× faster than sparse attention in GLM-5.3-FLASH, and 72.4× and 42.5× faster than Full Attention in QWEN-3.8-27B.
layer measurements support speedups under last-chunk prefill and single-request decode conditions rather than whole-prefill or concurrent-serving performance
D-LSA distils LSA into pre-trained models and stays close to the accuracy of their baseline attention up to 10M tokens.
small ultra-long-context benchmark subsets do not establish preservation across variants, with substantial Tree and model-specific Flat deficits
N-LSA trains LSA natively on the next-token loss alone, at the accuracy of Full Attention and 1.6× faster per layer at 128k tokens.
one approximately 0.5B-parameter model without run-level uncertainty matches validation loss but scores 32.2 versus 34.7 on 128k RULER
Derived from the full evaluation — not a separate score.
Strengths
The method replaces direct key or block scans with learned semantic-cluster search and demonstrates attention speedups at 10M tokens. Table 1 distinguishes this contribution from token-, block-, and semantic-sparse alternatives.
↳ Table 1; Sections 2.2–3 and 4.3
Parameter tables, kernel descriptions, and parallelism details connect the algorithm to SGLang serving on H200 and B300 systems. End-to-end measurements separately report time to first token and time per output token.
↳ Tables 2–3; Appendices F.1–F.2
PaperTrail separates fabricated facts and insertions from host and background papers. Its controls test questions alone, hosts without insertions, fabricated material, and background material to assess where answers originate.
↳ Appendix D; Table 8
Limitations
On GLM at 10M tokens, Tree scores 55.8 versus the baseline's 66.8 on RULER, while Flat trails by up to eight points at 2M–5M. These results narrow the abstract's general accuracy-preservation claim.
↳ Abstract; Section 4.2
RULER uses 20 prompts per task above 1M tokens, and LongBench uses 60 questions. End-to-end latency reports medians of three requests without dispersion, while native training lacks run-level uncertainty.
↳ Sections 4.1, 4.3, and 4.4; Table 7
Tree's prefill advantage is predicted beyond 10M rather than demonstrated there. Native training covers only an approximately 0.5B-parameter model at up to 128k tokens with Flat search.
↳ Sections 4.3–4.4 and 5
Learned semantic indexing addresses a documented attention bottleneck, and the implementation connects layer-level gains to end-to-end SGLang measurements at 10M tokens. Detailed training, kernel, and parallelism reporting supports the execution assessment, while small benchmark subsets and timing medians without dispersion constrain precision. The abstract's accuracy-preservation language receives less support than the efficiency claim because Section 4.2 reports substantial Tree losses on GLM and Section 4.4 tests native training only at small scale. Table 1 and the Discussion clearly situate the contribution and its boundaries, including linear KV-cache growth and the extrapolated Tree prefill crossover.
Nabu’s assessment, alongside the field’s view.
Are you an author of this paper?
Strong3.8
Confidence highLearned semantic indexing, distilled and native variants, and evaluation to 10M tokens meaningfully extend the sparse-attention approaches distinguished in Table 1. Flat prefill remains quadratic, and the practical prefill advantage of Tree is extrapolated beyond the measured range.
The study specifies training, comparator treatment, hardware, kernels, and serving configurations, with benchmark controls and failure analyses. Small ultra-long-context subsets, absent native-training run uncertainty, and three-request timing medians without dispersion limit precision.
The exposition makes the indexing mechanism and Flat-versus-Tree distinction followable. Headline accuracy-preservation language is broader than the model-specific deficits and small-scale native-training evidence reported in the results.
The related-work taxonomy distinguishes indexing units, search strategies, training regimes, and evaluation scales. Discussion connects KV-cache growth, Tree accuracy loss, and small native-training scale to interpretation, although closest semantic alternatives lack matched empirical comparisons.
Lower confidence on Methodological Rigour — domain match limited.
Caveats4 of 4 checks
The reported numbers and statistics are consistent with the stated results.
Table 4 assigns the same ARC-c label to two commonsense benchmark columns, leaving their identities ambiguous. Named code-release statements contain no identified availability contradiction.
Code availability: declared
96 references in manuscript 58 of 58 checkable references found in an index 38 have no canonical index record — counted, but not index-checkable 26 references confirmed by manual review
References not verified automatically are resolved manually within 24 hours.
No retraction or expression of concern found in Crossref (includes Retraction Watch).
Sources: Crossref (incl. Retraction Watch) ✓
Where this paper’s evidence sits on the path from initial observation to real-world use.
The Validated readiness stop reflects SGLang integration and end-to-end measurements on H200 and B300 hardware with tensor and context parallelism. Single-request serving measurements and limited model coverage leave broader deployment performance unestablished.
AI-generated, human-governed. Something look off? Contact us to request a review.
Keep reading
Other papers Nabu has evaluated blind, starting with the same field.
Liu, Zhao, Xie, et al. · 2026 · Computer and information sciences
CLAIMNOV catches 97% of disguised papers at a 1.5% false-alarm rate
SupportedSon, Han, Koh, et al. · 2026 · Computer and information sciences
Decomposition delivers: peak KV working set of 14.3 MiB per query against 35.5 and 35.3 MiB for single-pass and retrieval-augmented baselines.
SupportedLu, Chen, Wu, et al. · 2026 · Computer and information sciences
Across multiple datasets, task types, and LLMs, the Procedural Graph delivers consistent gains over memory-based baselines
Supported