Five design choices that together push the spread below 0.034.
🔤
Metaspace Pre-tokeniser
Words are marked with ▁ instead of splitting on punctuation. "India's" stays as one unit, allowing BPE to learn India's as a single token — reducing fertility below what whitespace splitting could achieve.
🧹
NFKC Normalisation
Unicode compatibility decomposition followed by canonical composition. Essential for Indic scripts where multiple code-point sequences can represent the same glyph. Ensures consistent character coverage across languages.
⚖️
Corpus Balancing
Each language's corpus is replicated by a tuned weight (en×3, hi×5, te×8, bn×8) so that every language receives proportional BPE merge capacity despite their different article lengths (43K – 201K chars).
🔗
Script Synergy
Bengali and Hindi both use Brahmic-family scripts; Telugu shares many Indic numeral and punctuation patterns. Cross-language merges benefit all four scripts, compressing each more than a monolingual tokenizer could.
🎯
min_frequency = 1
Every pair that appears even once is eligible for merging. This is critical for Indic scripts where character combinations are numerous but often infrequent in a single article. No pair is left behind.
🔄
Faithful Round-trip
decode(encode(text)) preserves all visible characters. The Metaspace decoder restores word-initial spaces, and NFKC normalisation ensures lossless treatment of apostrophes, numbers, and punctuation.