# Retrofitting language models to operate over bytes

- **DOI:** [10.1038/s41586-026-11111-4](https://doi.org/10.1038/s41586-026-11111-4)
- **Rating:** Strong (3.9 out of 5)
- **Verdict:** Byteification brings pretrained language models to byte-level operation with near-source aggregate performance, but single-run evaluations limit confidence in robustness.
- **Framework version:** v5.13
- **Pipeline version:** v0.4
- **Evaluation date:** 2026-11-20
- **Evaluation page:** https://www.nabu.science/paper/10.1038%2Fs41586-026-11111-4

## Key claims

1. Our byteified models outperformed, on average, all earlier publicly available byte-level LLMs of comparable size.
   - Support: Supported
   - Type: Primary finding
   - Basis: suite averages support capability gains, although inherited pretraining and single runs limit comparison
   - Location: Introduction; Table 1
2. The byteified models substantially surpassed their subword-level counterpart regarding character understanding [CUTE and EXECUTE benchmarks].
   - Support: Supported
   - Type: Secondary
   - Basis: character scores improve, although synthetic training data and controls limited to Olmo constrain architectural attribution
   - Location: Performance after byteification; Table 1
3. Although the Bolmo base model originally performed worse than Olmo 3 on IFEval, merging with task arithmetic lifted its performance to on par with the original post-trained checkpoint.
   - Support: Supported
   - Type: Secondary
   - Basis: the result holds for one checkpoint on IFEval rather than demonstrating general post-training transfer
   - Location: Post-training byteified models through task arithmetic; Fig. 3
4. Our results remove a long-standing performance barrier to end-to-end byte-level language modelling, demonstrating that models operating on raw text encodings can scale competitively
   - Support: Tentative
   - Type: Secondary
   - Basis: two retrofit scales using subword-pretrained backbones do not establish general competitive scaling
   - Location: Abstract, final sentence
5. We also explored the increased flexibility of the byte-level models, such as arbitrarily decreasing token granularity for faster inference.
   - Support: Tentative
   - Type: Secondary
   - Basis: finite compression experiments do not establish unrestricted granularity or efficiency gains
   - Location: Conclusion

## Strengths

- **Practical conversion across three model families** Bwen, Blama and Bolmo approach their source models on aggregate benchmarks while outperforming earlier byte-level models on suite averages. This establishes a practical route for reusing pretrained subword-model capabilities.
  - Dimension: contribution
  - Location: Introduction; Table 1
- **Controls distinguish training and architecture effects** The Olmo continued-training baseline uses the conversion data mix, helping separate data effects from architectural changes. The stage-one ablation also compensates approximately for differing computational costs.
  - Dimension: methodological rigour
  - Location: Table 1; Benefits of training in two stages; Fig. 4
- **Alternative explanations receive explicit interpretive attention** The discussion explains why sampling settings could account for the code diversity trade-off and why task-arithmetic transfer depends on embedding resettability. Related work identifies the architectural and training components inherited from earlier approaches.
  - Dimension: positioning
  - Location: Performance after byteification; Post-training byteified models through task arithmetic; Methods, Related work

## Limitations

- **Scaling and efficiency language exceeds testing** The Abstract describes competitive scaling from evidence at two retrofit scales, while the Conclusion describes arbitrary granularity changes from a finite compression experiment. These statements imply broader validation than the experiments provide.
  - Dimension: reporting
  - Location: Abstract, final sentence; Conclusion; Fig. 2
- **Single runs leave training robustness unresolved** Benchmark bootstrapping measures variation across selected tasks rather than independent training runs. Related benchmarks, including pass@1 and pass@16 versions of code tasks, also limit the independence of the resampled units.
  - Dimension: methodological rigour
  - Location: Table 1; Fig. 2 caption; Methods, Statistical tests
- **Aggregate parity conceals category performance losses** Bwen scores 60.2 versus Qwen3's 68.2 on Math and 39.6 versus 45.8 on Code pass@1. Character gains also combine architectural changes with synthetic training data, with a matched continued-training control available only for Olmo.
  - Dimension: contribution
  - Location: Table 1; Performance after byteification; Methods, Character-level training data

## Reliability checks

No concerns (4 of 4 checks).

- **Results check:** Clean. The reported numbers and statistics are consistent with the stated results.
- **Reporting and transparency:** Clean. Public repositories are named for training data, evaluation data and code, and competing interests are disclosed.
  - Conflict of interest: declared · Data availability: declared · Code availability: declared
- **Reference integrity:** Noted. 45 of 58 checkable references verified
  - 58 references in manuscript
  - Could not be verified automatically — queued for manual review:
    - Huang, H. et al. Over-tokenized transformer: vocabulary is generally worth scaling. In Proc. 42nd International Conference on Machine Learning, Vol. 267 (eds Singh, A. et al.) 26261–26282 (PMLR, 2025). (Crossref bibliographic top candidate ratio 0.35 — no plausible match)
    - Tito Svenstrup, D., Hansen, J. & Winther, O. Hash embeddings for efficient word representations. In Advances in Neural Information Processing Systems, Vol. 30 (eds Guyon, I. et al.) (Curran Associates, 2017). (Crossref bibliographic top candidate ratio 0.49 — no plausible match)
    - Athanasiadis, I., Karmush, A. & Felsberg, M. Grounding functional similarity by invarianceaware model stitching. In Proc. Forty-third International Conference on Machine Learning (2026). (Ambiguous: title ratio 0.50 via crossref_bibliographic sits between the divergence (0.30) and match (0.70) thresholds; too close to call either way, so it is recorded as unresolved rather than judged.)
    - Zhang, Y. & Yang, Q. A survey on multi-task learning. IEEE Trans. Knowl. Data Eng. 34, 5586–5609 (2022). (Crossref bibliographic search unavailable (search could not be performed) — no conclusion drawn about this reference)
    - Hendrycks, D. et al. Measuring massive multitask language understanding. In Proc. International Conference on Learning Representations (ICLR) (eds Mohamed, S. et al.) (International Conference on Learning Representations, 2021). (Ambiguous: title ratio 0.63 via crossref_bibliographic sits between the divergence (0.30) and match (0.70) thresholds; too close to call either way, so it is recorded as unresolved rather than judged.)
    - Lai, Y. et al. DS-1000: a natural and reliable benchmark for data science code generation. In Proc. Fortieth International Conference on Machine Learning, Vol. 202 (eds Krause, A. et al.) 18319–18345 (PMLR, 2023). (Crossref bibliographic top candidate ratio 0.34 — no plausible match)
    - Pal, A., Umapathi, L. K. & Sankarasubbu, M. MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Proc. Conference on Health, Inference, and Learning, Vol. 174 (eds Flores, G. et al.) 248–260 (PMLR, 2022). (Ambiguous: title ratio 0.60 via crossref_bibliographic sits between the divergence (0.30) and match (0.70) thresholds; too close to call either way, so it is recorded as unresolved rather than judged.)
    - Dosovitskiy, A. et al. An image is worth 16 × 16 words: transformers for image recognition at scale. In Proc. International Conference on Learning Representations (eds Mohamed, S. et al.) (International Conference on Learning Representations, 2021). (Crossref bibliographic top candidate ratio 0.42 — no plausible match)
    - Gloeckle, F., Youbi Idrissi, B., Roziere, B., Lopez-Paz, D. & Synnaeve, G. Better & faster large language models via multi-token prediction. In Proc. Forty-first International Conference on Machine Learning, Vol. 235 (eds Salakhutdinov, R. et al.) 15706–15734 (PMLR, 2024). (Ambiguous: title ratio 0.58 via crossref_bibliographic sits between the divergence (0.30) and match (0.70) thresholds; too close to call either way, so it is recorded as unresolved rather than judged.)
    - Rust, P. et al. Language modelling with pixels. In Proc. Eleventh International Conference on Learning Representations (eds Liu, Y. et al.) (International Conference on Learning Representations, 2023). (Crossref bibliographic top candidate ratio 0.35 — no plausible match)
    - Kallini, J., Murty, S., Manning, C. D., Potts, C. & Csordás, R. MrT5: dynamic token merging for efficient byte-level language models. In Proc. Thirteenth International Conference on Learning Representations (eds Yue, Y. et al.) 56646–56669 (International Conference on Learning Representations, 2025). (Ambiguous: title ratio 0.56 via crossref_bibliographic sits between the divergence (0.30) and match (0.70) thresholds; too close to call either way, so it is recorded as unresolved rather than judged.)
    - Dobler, K., Elliott, D. & de Melo, G. Token distillation: attention-aware input embeddings for new tokens. In Proc. Fourteenth International Conference on Learning Representations (eds Vondrick, C. et al.) 140590–140615 (International Conference on Learning Representations, 2026). (Crossref bibliographic top candidate ratio 0.42 — no plausible match)
    - Shenfeld, I., Pari, J. & Agrawal, P. RL's razor: why online reinforcement learning forgets less. In Proc. Fourteenth International Conference on Learning Representations (eds Vondrick, C. et al.) 59839–59864 (International Conference on Learning Representations, 2026). (Ambiguous: title ratio 0.54 via crossref_bibliographic sits between the divergence (0.30) and match (0.70) thresholds; too close to call either way, so it is recorded as unresolved rather than judged.)
  - References not verified automatically are resolved manually within 24 hours.

## Post-publication record

- **Status:** Clean
- **Summary:** No retraction or expression of concern found in Crossref (includes Retraction Watch).
- **Last checked:** 2026-11-20
