2026-08-23 · Erdem Hanay

An Empirical Study on Sub-Word Tokenizer Fragmentation via Zero-Width Unicode Injection

Abstract

Byte Pair Encoding (BPE) compresses text by capturing frequent character runs into dedicated vocabulary entries. Introducing an invisible character, specifically U+200C (ZERO WIDTH NON-JOINER, or ZWNJ), into those runs breaks the exact-match lookup. As a result, the tokenizer fragments the sequence into smaller pieces paired with a standalone token for the non-joiner itself, while the visual text remains identical to a human reader.

This article documents a production RSS interleaver that applies that transform at stride 2, and three measurements on a 512-token window of Andersen (1837/n.d.): token inflation ratio (TIR) under cl100k_base, extractive QA exact match on deepset/tinyroberta-squad2, and raw DistilBART summaries (Shleifer, 2020). TIR ranged from +539.1% at stride 1 to +98.4% at stride 4. These two small models failed on the haunted window (QA exact match 0.571 to 0.000 at stride 2; DistilBART shredded). One regular expression restored both. The QA reader truncates at 512 of its own tokens, so zeros at strides 1–2 mix merge-breaking with length. Cosine embedding shift (ΔH) and ROUGE-1 were not measured. Large chat models were not evaluated. This page is not interleaved; only phile bodies in /rss.xml receive the transform.

I. Introduction

BPE does not see a word. It sees the longest byte-run it was trained to merge (Gage, 1994; Sennrich et al., 2016). Break that run with a format character that occupies no ink, and the merge is gone while the glyphs on the page stay put.

That gap now sits on a live ingest path. RSS remains a common way to pull prose into summarizers, classifiers, and retrieval-augmented generation (RAG) pipelines (RSS Advisory Board, 2009). Those systems are billed and bounded by tokens. Invisible-character attacks on NLP models are already documented: Boucher et al. (2022) showed that imperceptible Unicode can flip model behavior, Schulz et al. (2025) showed that token-boundary manipulation can bypass classifiers, and Sarabamoun (2025) surveyed special-character attacks on open-source language models. The present work is narrower. It measures how a stride-aware ZWNJ interleaver fragments cl100k_base, what that costs in token count, and whether two small local models still recover the same window.

The thesis is threefold. First, I describe the production helper that walks a .phile body, copies protected spans, and interleaves only letter-runs longer than two characters. Second, I report TIR for strides 1 through 4 on a public-domain window. Third, I report extractive QA and DistilBART output on that same window, with a sanitizer restore condition. The runner is leafturner/locust-gutenberg-test (Hanay, 2026). The informal origin of the idea is Locust I.

This article leaves several things out on purpose. It does not report embedding cosine (ΔH) or ROUGE-1. It does not test a large chat model. It does not claim a novel attack. It is not a jailbreak kit, not a new tokenizer, and not a security boundary. One regular expression restores the letters. The first live-RSS audit is included only as a negative control: its “clean” side never stripped U+200C, so the +8.0% figure is decoration cost, not husk cost. Husk, used once here, means a letter-run split around a ZWNJ so that BPE can no longer store it as one piece.

II. Fundamental Concepts and Architecture

Terminology

Byte Pair Encoding. A compression-derived merge algorithm that iteratively replaces frequent symbol pairs with a new symbol (Gage, 1994). Neural machine translation adopted it as a subword tokenizer so rare words share pieces with frequent ones (Sennrich et al., 2016). OpenAI’s cl100k_base encoding, exposed through tiktoken, is the BPE table used in this study (OpenAI, 2023).

ZERO WIDTH NON-JOINER (U+200C). A Unicode format character in general category Cf. It is invisible in ordinary rendering. It is not U+200B (ZERO WIDTH SPACE). U+200B is a line-break opportunity; some readers wrap on it and the “invisible” tax becomes a ragged paragraph (Unicode Consortium, 2023). This implementation uses ZWNJ for that reason.

Stride S. The number of letters copied before each inserted ZWNJ. Stride 1 inserts a mark between almost every letter in a long run. Stride 2, the production default, inserts a mark every two letters. Words of length 1 or 2 are left whole.

Token inflation ratio.

TIR(S) = (N_perturbed(S) - N_clean) / N_clean

N_clean is the token count of an unperturbed window. Windows are sliced on the clean encoding so the denominator stays comparable across strides.

Protected spans. Substrings the linear scan copies unchanged: HTML tags, URLs, markdown images and links, ink blocks, CDATA closers, and ANSI wrappers of the form #[role|text]. The wrapper is preserved; only the inner text is interleaved.

Workflow

The production path and this article’s path diverge on purpose.

.phile body
  -> RSS item render
  -> linear scan (copy protected spans)
  -> interleave letter-runs at stride S
  -> RSS description and content:encoded

HTML research pages (/blog/...)
  -> no interleave

Feed items follow RSS 2.0 item content conventions (RSS Advisory Board, 2009). This measurement page stays clean so the sandbox and the quoted tables are not self-poisoned.

Theoretical foundation

BPE lookup is longest-match against a learned merge table. A vocabulary entry for a frequent run such as a common English stem exists only as that exact byte sequence. Inserting U+200C splits the sequence. The table miss produces (a) leftover pieces that may themselves be vocabulary entries and (b) a standalone token for the format character. The reader’s glyph stream is unchanged.

The interleaver is a single left-to-right pass over the input: O(n) in character length. It does not build a suffix array or retrain a vocabulary. TIR is expected to fall as S grows, because each increment of S reduces the number of insertion sites inside a letter-run of length L to about ceil(L / S) - 1 (for L greater than 2). That is a counting argument, not a latency benchmark. No wall-clock or memory figures are reported.

III. Implementation and Practical Application

The production transform lives in src/modules/seo/zero-width.ts. It walks the rendered item body, copies each protected span, and runs interleaveProse on the gaps. interleaveProse tokenizes on non-whitespace, then skips tokens that are already short, already contain format characters, or match digits, hexadecimal literals, or email addresses. Remaining tokens have their Unicode letter-runs passed to interleaveWord.

export function interleaveWord(word: string, stride = 2): string {
  if (word.length <= 2) {
    return word;
  }

  const step = Math.max(1, Math.floor(stride));
  const chunks: string[] = [];

  for (let index = 0; index < word.length; index += step) {
    chunks.push(word.slice(index, index + step));
  }

  return chunks.join(ZWNJ);
}

The Python companion implements the same letter-run rule and the same sanitizer. Anyone who wants the original letters can drop the common zero-width format characters in one pass:

import re

def strip_zero_width(text: str) -> str:
    return re.sub(r"[\u200B-\u200D\uFEFF]", "", text)

The regex package can also strip the entire Unicode Cf class in one substitution. That is not a shield. It is a tax on the lazy ingest path.

The sandbox below uses the same TypeScript interleaveProse helper as the feed, with a live stride control. It is a demonstration, not the 512-token official window.

Trade-offs, edge cases, and pitfalls

ZWNJ over ZWSP is a rendering choice. Stride 2 over stride 1 is a readability choice: per-letter insertion nearly doubles the BPE bill relative to stride 2 on the official window and makes some selection UIs behave poorly. Short words stay whole so articles and prepositions do not become noise. URLs, emails, and hex tokens stay whole so links and identifiers remain copyable.

This article and the rest of /blog/ are not RSS items. Confirming that property is part of the method: a measurement page that interleaves its own prose cannot quote TIR honestly.

The first live-RSS audit is the main pitfall. The “clean” side stripped ASCII banners and #[role|text] markers but did not strip U+200C. The resulting +8.0% TIR is mostly decoration versus raw, not clean prose versus husked prose. That mistake is recorded in Section IV as a negative control, not as a headline finding.

IV. Benchmarks and Performance Analysis

TIR is the cost metric. Extractive QA EM/F1 and raw DistilBART strings are the recovery metrics. The interleaver is O(n) as described above. Latency, memory overhead, and throughput were not measured.

The official TIR run used python benchmark.py from Hanay (2026) on 2026-08-23T00:10:39Z, Python 3.14.7, tiktoken==0.14.0, encoding cl100k_base. QA and DistilBART used the same window via qa.py and summarize.py and --text-file 1597-0.txt (Andersen, 1837/n.d.; Gutenberg cache/epub copy). After whitespace collapse the story body was 297351 characters. The first 512 clean tokens (2403 characters) were the window. Raw stdout is in the supplementary log.

Official window versus stride

N_clean is 512. TIR falls as stride grows, which matches the insertion-site count. Stride 1 is the expensive case. Production stride 2 more than triples the window. Even stride 4 almost doubles it. The script asserted strip_zero_width(perturbed) == window after every stride and exited 0.

These counts will move if Gutenberg edits the file or tiktoken ships a new cl100k_base merge table. The runner, not this table, is the source of truth.

Extractive QA on the same window

The QA reader truncates at 512 of its own tokens. Interleaved windows are longer than that. A zero at stride 1 or 2 is merge-breaking and possible cutoff, not proof that “AI cannot understand” the story.

python qa.py --text-file 1597-0.txt. Model: deepset/tinyroberta-squad2. Fourteen hand-written questions whose gold spans sit in the 512-token window. Predictions are stripped of U+200B–U+200D and U+FEFF before exact match and token F1.

Clean exact match is 8 of 14. That is above chance and not a strong reader. At strides 1 and 2 the score is 0 of 14. At stride 4 it is 5 of 14. Every sanitized row matches clean. That is the causal pattern: the ghosts (and the extra length they create) caused the miss; one strip restored the baseline. This is span recovery on one small model. It is not a hallucination study and not a claim about large chat models.

DistilBART strings on the same window

python summarize.py --text-file 1597-0.txt. Model: DistilBART-CNN-12-6 (Shleifer, 2020; cf. Lewis et al., 2020). Beam search, max_length=80. ROUGE was not computed. These are the raw strings.

clean. An Emperor was so fond of new clothes that he spent all his money in dress. One day, two rogues, calling themselves weavers, made their appearance at the court. They gave out that they knew how to weave stuffs of the most beautiful colors and elaborate patterns. “These must, indeed, be splendid clothes!” thought the Emperor. “Had I

S=1. shredded leftover letters; not a summary of the story.

S=2 (production). starts “Many years ago” as broken two-letter husks; not a usable summary.

S=4. almost the opening sentence, with leftover marks inside words (“Empe’ror”, “clot’hes”).

S=2 sanitized. identical to the clean string.

That is generation collapse and copy-garbage on this summarizer, not an invented alternate plot. After sanitizing, DistilBART tells the story again.

Negative control: contaminated live RSS

First content:encoded item on https://www.erdemhanay.info/rss.xml (childhood post, after the feed transform). The “clean” side stripped banners and ANSI markers and left ZWNJ in place.

XML parse succeeded under ElementTree and strict lxml. DistilBART and MNLI runs on that dirty feed are not findings. The clean-window DistilBART strings above replace them.

Extra corpus: sandbox sentence

Same helper and encoding, one sentence from the sandbox. This is not the official window. A 12-token sentence has less punctuation and fewer already-atomic tokens, so TIR is worse. The two tables must not be averaged.

Clean text: The Emperor walked through the grand hall in his invisible robe. Clean tokens: 12.

V. Real-World Use Cases and Future Outlook

The deployed case is this site. /rss.xml interleaves phile bodies at stride 2. A scraper that concatenates content:encoded and runs cl100k_base pays the official-window tax on that stride: +244.5% on the Gutenberg control, and an unknown TIR on each live item. The same stride zeroed the small QA reader and shredded DistilBART on the control window. Human subscribers see ordinary prose. The research pages, including this one, are excluded from that feed.

Castagnaro et al. (2025) describe attacks on RAG data loaders that treat plain text as trusted. A loader that does not strip Cf will index husked tokens, retrieve them, and spend context on format characters. The defense is the sanitizer in Section III, applied before chunking. That is also why this transform is not digital rights management. Anyone who wants the letters can have them.

Current limitations are sharp. The study uses one encoding, one public-domain window, one production stride, and two small CPU models. The QA reader truncates at 512 of its own tokens. ΔH and ROUGE-1 remain undone. Larger chat models were not run. Other encodings (SentencePiece, older GPT-2 BPE, multilingual tables) were not run. XML validity is not an obstacle: both parsers accepted the poisoned feed.

Ongoing work, stated as unfinished rather than as results: a multi-encoding TIR table, a larger-model QA pair, and a reusable library extracted from zero-width.ts.

VI. Conclusion and Key Takeaways

Four points follow from the measurements.

1. BPE longest-match is brittle against an invisible format character placed inside a letter-run. The human reader is not the tokenizer.
2. On a 512-token cl100k_base window of Andersen, TIR was +539.1%, +244.5%, +137.5%, and +98.4% at strides 1 through 4. Production RSS uses stride 2.
3. These two small models failed; a regex fixed it. tinyroberta-squad2 exact match fell from 0.571 to 0.000 at strides 1 and 2 (that reader also truncates at 512 of its own tokens). DistilBART’s clean summary became shredded leftover letters. Sanitizing restored both.
4. The defense is to strip Cf (or the small zero-width set) before tokenization. That is a tax on lazy ingest, not a claim about large chat models.

To reproduce, clone leafturner/locust-gutenberg-test and run python benchmark.py, python qa.py --text-file 1597-0.txt, and python summarize.py --text-file 1597-0.txt. The raw log is the transcript. Locust I is the informal origin, including the contaminated-baseline mistake. If you ingest RSS into a model, strip format characters first.

© 2026 Erdem Hanay. Text and tables on this page may not be reproduced without attribution. The measurement scripts are MIT (locust-gutenberg-test).

References