fix: Remediate 219 glitch tokens in tokenizer.json
Browse files## Glitch Token Remediation — Tokenizer Patch
### Summary
This PR patches `tokenizer.json` to remediate **219 glitch tokens** identified
by embedding vector analysis. Glitch tokens are vocabulary entries whose embeddings
have collapsed to near-identical representations, causing unpredictable model behavior
when they appear in input.
**No model weights are modified** — this is a tokenizer-only fix.
### Technique: Hybrid (merge pruning + placeholder rename)
The patch uses two complementary strategies:
**1. Merge pruning** (219 tokens): Removes the BPE merge rules that
produce multi-character glitch tokens. The vocab entry stays (ID preserved) but
becomes unreachable — text that would have matched is now tokenized as trained
constituent subwords instead.
**2. Placeholder rename** (0 tokens): For single-character glitch tokens
(rare Unicode characters) that have no producing merge rule, the vocab string is renamed to
`<glitch_pruned_ID>`. The original character now falls through to `byte_fallback`,
encoding as well-trained UTF-8 byte tokens.
**Safety:** Placeholder strings cannot be triggered by user input. BPE builds tokens
bottom-up from characters via merge rules only — since no merge rule produces these strings,
they are permanently unreachable. Gemma's **split-digits tokenization policy** provides an
additional layer of safety.
253 merges were removed for 219 merge-pruned tokens because some glitch tokens have more than one BPE merge path, and every path must be removed.
### Before / after examples: `google/gemma-4-E4B-it`
**Merge prune example: `drawingCodehint`, token ID 226565**
Prompt: `Please repeat the following string exactly, with nothing else: "drawingCodehint"`
| | Tokens for the string | Model output |
|---|---|---|
| Before | `['drawingCodehint']` → `[226565]` | `""` ❌ |
| After | `['drawingCode', 'hint']` → `[140974, 45581]` | `"drawingCodehint"` ✅ |
### ⚠️ Note for Agentic Systems
Many remediated glitch tokens are **shared across Gemma model families** (Gemma 1, 2, 3, 4).
In agentic pipelines, care should be taken that these token strings are not inadvertently
injected into prompts for unpatched models. We recommend applying this patch consistently
across all Gemma models used in a pipeline.
### What is preserved
- ✅ `len(tokenizer)` — unchanged (262,144)
- ✅ All token IDs — stable, no re-indexing
- ✅ Chat template — identical to original
- ✅ `tokenizer_config.json` — identical to original
- ✅ All special tokens and `added_tokens` — unchanged
- ✅ Normal text tokenization — verified via benchmarks
- ✅ **Zero model weight changes**
### Diff summary
| | Original | Patched | Delta |
|---|---|---|---|
| Vocab size | 262,144 | 262,144 | 0 |
| Merges | 514,906 | 514,653 | −253 |
| Vocab renamed | — | 0 | +0 |
### Removed merges (253)
<details>
<summary>Click to expand the full list of 253 removed merges</summary>
```
["."+"|", ""+""] → ."+"|"+"
["AWS", "JavaScript"] → AWSJavaScript
["AllFilesIn", "MkvDir"] → AllFilesInMkvDir
["CYCL", "ONEDB"] → CYCLONEDB
["Case", "Missense"] → CaseMissense
["Case", "PTV"] → CasePTV
["Collin", "ToDO"] → CollinToDO
["Consec", "HtIdx"] → ConsecHtIdx
["Control", "PTV"] → ControlPTV
["Custom", "Glare"] → CustomGlare
["CustomGlare", "Def"] → CustomGlareDef
["DT", "MakeRectInCM"] → DTMakeRectInCM
["Days", "GE"] → DaysGE
["Denovo", "Mis"] → DenovoMis
["ELEASE", "STR"] → ELEASESTR
["ENEMY", "PLACE"] → ENEMYPLACE
["Fcm", "Php"] → FcmPhp
["Foldout", "GC"] → FoldoutGC
["GTBase", "Alert"] → GTBaseAlert
["Get", "ParaPkg"] → GetParaPkg
["GoObject", "AllRef"] → GoObjectAllRef
["Go", "PrintError"] → GoPrintError
["Go", "RawContext"] → GoRawContext
["Go", "SrvGroupIndex"] → GoSrvGroupIndex
["H", "IMQTTRVM"] → HIMQTTRVM
["IMQTTR", "VM"] → IMQTTRVM
["KAKKIA", "INEN"] → KAKKIAINEN
["LuaPush", "Int"] → LuaPushInt
["Lua", "ToJavaResult"] → LuaToJavaResult
["MDEw", "Ol"] → MDEwOl
["MakeRect", "InCM"] → MakeRectInCM
["Mkv", "Dir"] → MkvDir
["MyShop", "name"] → MyShopname
["Ordinate", "Tuple"] → OrdinateTuple
["PAF", "Struct"] → PAFStruct
["PQ", "Ary"] → PQAry
["P", "QAry"] → PQAry
["RefIn", "SrvGroup"] → RefInSrvGroup
["Ret", "JpegLen"] → RetJpegLen
["SAFER", "ELEASESTR"] → SAFERELEASESTR
["SRP", "Basic"] → SRPBasic
["SRP", "BinBuf"] → SRPBinBuf
["SRPInterface", "Item"] → SRPInterfaceItem
["SRP", "ParaPkg"] → SRPParaPkg
["SRPS", "XML"] → SRPSXML
["SRP", "SrvGroup"] → SRPSrvGroup
["SRPS", "rvGroup"] → SRPSrvGroup
["Same", "WordInterval"] → SameWordInterval
["Simulator", "Flood"] → SimulatorFlood
["SrvGroup", "Index"] → SrvGroupIndex
["StarBinBuf", "Body"] → StarBinBufBody
["StarObject", "Body"] → StarObjectBody
["StarParaPkg", "Body"] → StarParaPkgBody
["StarSXml", "Body"] → StarSXmlBody
["StarService", "Body"] → StarServiceBody
["StarSrvGroup", "Body"] → StarSrvGroupBody
["StructOf", "Object"] → StructOfObject
["TP", "ASDW"] → TPASDW
["Term", "ObjectDefer"] → TermObjectDefer
["Transc", "Hist"] → TranscHist
["U", "imcoords"] → Uimcoords
["VSLU", "ATYPE"] → VSLUATYPE
["WMN", "MDA"] → WMNMDA
["WeakTable", "Mutex"] → WeakTableMutex
["WithSize", "InCM"] → WithSizeInCM
["Ziel", "Panel"] → ZielPanel
["addKill", "Bonus"] → addKillBonus
["addKill", "Penalty"] → addKillPenalty
["adig", "anik"] → adiganik
["adigan", "ik"] → adiganik
["ak", "arantadhatu"] → akarantadhatu
["angolo", "Rad"] → angoloRad
["angolo", "Tocco"] → angoloTocco
["arant", "adhatu"] → arantadhatu
["arantad", "hatu"] → arantadhatu
["atthavid", "u"] → atthavidu
["attup", "adani"] → attupadani
["attu", "vasena"] → attuvasena
["bani", "pi"] → banipi
["ban", "ipi"] → banipi
["blusas", "Fem"] → blusasFem
["capture", "cpu"] → capturecpu
["ch", "ccgi"] → chccgi
["check", "katore"] → checkkatore
["co", "OrdinateTuple"] → coOrdinateTuple
["colour", "CodeDict"] → colourCodeDict
["country", "geocode"] → countrygeocode
["custom", "Glare"] → customGlare
["df", "sonic"] → dfsonic
["dfs", "onic"] → dfsonic
["done", "ProcessAvg"] → doneProcessAvg
["draw", "countery"] → drawcountery
["drawingCode", "hint"] → drawingCodehint
["dw", "RetJpegLen"] → dwRetJpegLen
["eco", "expr"] → ecoexpr
["edLeft", "Shape"] → edLeftShape
["edRight", "Shape"] → edRightShape
["eltat", "double"] → eltatdouble
["faulse", "Ans"] → faulseAns
["foe", "Place"] → foePlace
["get", "HDRProcessor"] → getHDRProcessor
["get", "starcore"] → getstarcore
["getstarcore", "data"] → getstarcoredata
["gmzy", "h"] → gmzyh
["grafo", "Existe"] → grafoExiste
["him", "qttrvm"] → himqttrvm
["ineed", "follower"] → ineedfollower
["inertia", "Seq"] → inertiaSeq
["int", "Fragmentation"] → intFragmentation
["jols", "endev"] → jolsendev
["js", "bpmOb"] → jsbpmOb
["kit", "opssynth"] → kitopssynth
["last", "DamageTook"] → lastDamageTook
["local", "instrList"] → localinstrList
["m", "BlitzID"] → mBlitzID
["mark", "UpdateChoice"] → markUpdateChoice
["opss", "ynth"] → opssynth
["ops", "synth"] → opssynth
["pJ", "PEGBuf"] → pJPEGBuf
["paren", "macro"] → parenmacro
["partial", "owner"] → partialowner
["pmm", "Imp"] → pmmImp
["pos", "Tocco"] → posTocco
["qttr", "vm"] → qttrvm
["respArray", "All"] → respArrayAll
["right", "squig"] → rightsquig
["sad", "urdu"] → sadurdu
["sadurdu", "poetry"] → sadurdupoetry
["sal", "expr"] → salexpr
["selectTable", "X"] → selectTableX
["selectTable", "Y"] → selectTableY
["smo", "io"] → smoio
["sor", "finaly"] → sorfinaly
["squarePos", "Vecchio"] → squarePosVecchio
["start", "ZielPanel"] → startZielPanel
["tcp", "UniqueID"] → tcpUniqueID
["testGet", "Popup"] → testGetPopup
["time", "PlusEvents"] → timePlusEvents
["tochy", "odikwa"] → tochyodikwa
["total", "BlockFit"] → totalBlockFit
["waved", "eltatdouble"] → wavedeltatdouble
["wired", "Elems"] → wiredElems
["தல", "மோங்கு"] → தலமோங்கு
["ய்ய", "மணி"] → ய்யமணி
["వెట్", "స్కీ"] → వెట్స్కీ
["▁", "::::::::"] → ▁::::::::
["▁::", "::::::"] → ▁::::::::
["▁Afd", "Par"] → ▁AfdPar
["▁A", "fdPar"] → ▁AfdPar
["▁Archers", "Unit"] → ▁ArchersUnit
["▁Atr", "ayam"] → ▁Atrayam
["▁Atra", "yam"] → ▁Atrayam
["▁C", "AdxRtList"] → ▁CAdxRtList
["▁CC", "BUNDLE"] → ▁CCBUNDLE
["▁Control", "Missense"] → ▁ControlMissense
["▁Control", "PTV"] → ▁ControlPTV
["▁", "ControlPTV"] → ▁ControlPTV
["▁DT", "MakeRect"] → ▁DTMakeRect
["▁Func", "ParamNum"] → ▁FuncParamNum
["▁", "FuncParamNum"] → ▁FuncParamNum
["▁GoObject", "To"] → ▁GoObjectTo
["▁Go", "RawContext"] → ▁GoRawContext
["▁", "GoRawContext"] → ▁GoRawContext
["▁Go", "SRP"] → ▁GoSRP
["▁", "GoSrvGroupIndex"] → ▁GoSrvGroupIndex
["▁Go", "SrvGroupIndex"] → ▁GoSrvGroupIndex
["▁H", "IMQTTRVM"] → ▁HIMQTTRVM
["▁", "HIMQTTRVM"] → ▁HIMQTTRVM
["▁HRM", "Client"] → ▁HRMClient
["▁LG", "AGEmoji"] → ▁LGAGEmoji
["▁Lua", "ToGoObject"] → ▁LuaToGoObject
["▁Lua", "ToJavaResult"] → ▁LuaToJavaResult
["▁", "LuaToJavaResult"] → ▁LuaToJavaResult
["▁ND", "IndexArray"] → ▁NDIndexArray
["▁New", "ParaPkg"] → ▁NewParaPkg
["▁", "NewParaPkg"] → ▁NewParaPkg
["▁Paf", "Handle"] → ▁PafHandle
["▁", "PafHandle"] → ▁PafHandle
["▁RUTARE", "AL"] → ▁RUTAREAL
["▁RUTARE", "L"] → ▁RUTAREL
["▁RUTAREL", "ATIV"] → ▁RUTARELATIV
["▁Ref", "ToGoObject"] → ▁RefToGoObject
["▁SIINFE", "KLC"] → ▁SIINFEKLC
["▁SIINFEKL", "C"] → ▁SIINFEKLC
["▁SRP", "Go"] → ▁SRPGo
["▁SRPGo", "Get"] → ▁SRPGoGet
["▁SRPGoGe
- tokenizer.json +2 -2
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6674dbb0510f80ba5033b93dd0a0f3573fed19626785f03f4fdc0466616f592c
|
| 3 |
+
size 15142278
|