Tokenizer Evaluation on European Languages

Intro

The tokenizer is a vital component of any LLM, encoding sequences of text into a pre-defined set of tokens. However, the tokenizer is built seperately from the LLM itself and undergoes a seperate training phase with its own training data. Consequently, the tokenizers of most commercial models are heavily optimized for English text with varying performance for non-english languages. Since Occiglot is building LLMs for non-english languages based on existing models and tokenizers, we need to gain a thorough understanding of their inherent performance on the languages we aim to support.

There are a lot of nuances going into tokenizer design, such as the vocabulary size and the algorithm used, but here we will just focus on evaluating the tokenizers of popular LLMs on European Languages. For more detailed information on tokenizers in general I highly recommend to check out the recent lecture of Andrej Karpathy¹.

Setup

To evaluate the performance of a tokenizer on a given langauge we are mainly interested in two metrics: the subword fertility and proportion of continued words. Fertility measures the average number of tokens used per word, whereas proportion of continued words refers to words that were split into multiple tokens. Thus, an ideal tokenizer would have fertility of 1.0 and continuation proportion of 0.0.

These results will vary based on dataset used for evaluation and the specific method to split words. Therefore, we present this comprehensive evaluation that applies the same methodology to multiple tokenizers and languages to facilitate meaningful comparisons.

We loosly follow the approach of Rust et al.² using the fast UDPipe^{3 4} to pre-split documents into words and subsequently run the tokenizer over isolated words. For all languages we use the respective November 2023 snapshot from Wikipedia⁵. Since Wikipedia, by nature, contains significantly more numbers and dates than other text and most tokenizers split those into single digits, we filtered all lone-standing numbers from the documents.

We considered 29 (european) languages (see below) and 4 popular tokenizers: Mistral, Llama-2, Phi-2 & Gemma. The first two both have a vocabulary size of 32k, whereas Phi-2 utilizes 51k and Gemma has a whopping 256k vocabulary.

Results

As a baseline, let us first look at the results for English which, as expected, is the best supported language for all tokenizers. Subsequently, the values in the tables always denote fertility, and the value in brackets is the proportion of continued words.

We can see that Mistral and Llama perform roughly the same, while Gemma’s significantly larger vocab size yields better performance. Surprisingly, Phi achieves the worst fertility despite it having a more than 1.5x larger vocabulary than Mistral/Llama.

With that baseline established, we now look into the majority of European languages, divided by their language family.

Romance Languages

West-Germanic Languages

North-Germanic Languages

Baltic Languages

East-Slavik Languages

South-Slawik Languages

West-Slavik Languages

Uralic Languages

Other Languages

In general, Mistral and Llama perform very similar with the latter achieving slightly better fertility than the former for most languages. Thanks to its very large vocab size, Gemma has the best performance on all languages, while the subpar performance of Phi continues for non-english languages especially on non-latin scripts.

As expected, the tokenizers perform best for languages closely related to English, i.e. Romance and West-Germanic languages. The fertility drops off for other language families, most strongly for Baltic and Uralic languages and those with non-latin (i.e. cyrillic and greek) scripts.

Conclusion

First of all, it is important to re-iterate that tokenizer performance on a specific language does not have to correlate with the capabilities on the model itself on that language. As stated earlier, the tokenizer is trained separately from the LLM and might have more multilingual data in its training set than the LLM and vice versa.

What do these results mean for Occiglot then? The evaluation conducted here is part of our continued commitment to creating dedicated (bilingual) LLMs for all European languages. For each new language we first ask ourselves if the existing tokenizer and its vocabulary is sufficient or if it requires extension. Based on the results above we set that threshold to a fertility of 2.0 that we aim to achieve for every bilingual model. Consequently, we will keep/have kept the tokenizer unchanged for West-Germanic and Romance languages, but will extend the tokenizer for all others. Additionally, we will refer to this benchmark whenever training a new or extended tokenizer for any language.

References