---
license: apache-2.0
language:
- en
library_name: transformers
datasets:
- allenai/olmo-mix-1124
---

# SuperBPE Tokenizer
SuperBPE extends the BPE algorithm to train tokenizers that include both traditional subword tokens (contained within word boundaries), as well as new **superword** tokens (containing parts of multiple words)! This tokenizer has a vocabulary size of 200k and transitions from learning subword to learning superword tokens at vocabulary size of 160k. It is trained on a random subset of Olmo2's pretraining data.

## Example Usage
```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("alisawuffles/OLMo2-8B-SuperBPE-t180k")
tokenizer.convert_ids_to_tokens(tokenizer.encode("By the way, I am a fan of the Milky Way."))
# ['ByĠtheĠway', ',ĠIĠamĠa', 'ĠfanĠofĠthe', 'ĠMilkyĠWay', '.']
```

# Citation
```
@misc{liu-etal-2025-superbpe,
  title={SuperBPE: Space Travel for Language Models}, 
  author={Alisa Liu and Jonathan Hayase and Valentin Hofmann and Sewoong Oh and Noah A. Smith and Yejin Choi},
  year={2025},
  eprint={2503.13423},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2503.13423}, 
}
```