TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

This paper introduces TokSuite, a valuable resource for understanding the impact of tokenization on language models. By training multiple models with identical architectures but different tokenizers, the authors isolate and measure the influence of tokenization. The accompanying benchmark further enhances the study by evaluating model performance under real-world perturbations. This research addresses a critical gap in our understanding of LMs, as tokenization is often overlooked despite its fundamental role. The findings from TokSuite will likely provide insights into optimizing tokenizer selection for specific tasks and improving the robustness of language models. The release of both the models and the benchmark promotes further research in this area.