Retrofitting language models to operate over bytes.

Byte-level language models, retrofitted from subword systems, achieve performance comparable to subword models with minimal retraining, excelling in character-level reasoning.

• Why it matters: Fine-grained information is crucial for scientific data like code and biological sequences, where meaning depends on individual characters or bytes, but current byte models lag behind in performance.
• What they did: A two-stage byteification method was developed to retrofit existing subword-based models into byte-level models, enabling them to process raw text efficiently with minimal additional training.
• The result: The resulting models outperform earlier byte-level approaches, enabling scalable, high-speed, fine-grained textual understanding, thus bridging a long-standing performance gap in byte-level language modeling.