Imperial researcher helps overcome long-standing barrier in AI language models

Research published in Nature shows a practical way to convert existing AI language models into byte-level systems, improving their ability to process information where character-level precision matters.

Dr Edoardo Maria Ponti, Associate Professor in Natural Language Processing in the Department of Computing at Imperial College London, has co-authored a Nature paper that could help make AI systems better at handling code, multilingual text and biological sequences. The approach could also benefit multilingual AI, as the flexibility of byte-level tokenisation makes it less biased towards high-resource languages. This could enable more efficient processing across a wider range of languages and writing systems.

Published with colleagues from the Allen Institute for AI (Ai2) and partner institutions, the research presents a new method for converting existing large language models into byte-level systems through a process known as "byteification". The approach allows models to work directly with the underlying digital representation of text while retaining much of the capability of today's leading AI systems.

This flexibility could help create AI systems that work more effectively across different languages and scientific data, broadening access to AI technologies and supporting applications from software development to biological research." Dr Edoardo Maria Ponti Associate Professor in Natural Language Processing

Commenting on the publication, Dr Ponti said: "This project aligns the unit in which language models perceive text with the unit in which all digital data is stored: the byte. In current language models, the unit of perception is decided before training even begins, and none of the training that follows can revisit it. Our models instead learn, at the same time, how to predict text and how to merge bytes into larger units.

"This flexibility could help create AI systems that work more effectively across different languages and scientific data, broadening access to AI technologies and supporting applications from software development to biological research."

Large language models underpin many of today's AI tools. Before processing text, they typically divide it into units known as tokens, usually representing words or parts of words. This approach has supported rapid advances in AI by allowing models to process and generate text efficiently.

However, tokenisation also hides information contained within the individual characters that make up words. This can create difficulties in domains where meaning depends on precise sequences of characters, including software code, biological sequences and certain multilingual applications.

Researchers have long explored whether language models could instead operate directly on bytes, the basic units used to store digital information. While this approach offers a more detailed representation of text, byte-level models have historically lagged behind conventional language models in performance.

The new study introduces a general method for converting existing language models into byte-level systems through a two-stage process requiring minimal additional training. Rather than building entirely new models from scratch, the approach enables researchers to reuse existing language models and the wider tools and infrastructure that support them.

The converted models significantly outperformed previous byte-level approaches and excelled at character-level reasoning tasks while maintaining competitive performance on broader language benchmarks. The researchers also demonstrated practical inference speeds, addressing another challenge that has historically limited byte-level models.

The findings suggest a path towards more capable AI systems for domains that depend on fine-grained textual understanding. In software engineering, this could help models better process programming languages where small character changes can alter functionality. In biology, it may support applications involving DNA, RNA and protein sequences, where meaning is encoded in precise strings of symbols.

The approach could also benefit multilingual AI. Because byte-level models are not tied to language-specific vocabularies, they have the potential to work more effectively across different languages and writing systems.

By demonstrating that byte-level language models can scale competitively with conventional approaches, the researchers believe their work removes a major barrier to wider adoption of AI systems that operate directly on raw text encodings.