2026-08-23 · Erdem Hanay

Raw log: Gutenberg window TIR, QA, DistilBART, 2026-08-23

Lab note supporting the author preprint (https://doi.org/10.5281/zenodo.22064523). Not peer-reviewed. Not a second paper. I copied printer output from the companion scripts. Methodology and the live sandbox live in the other research post. The runner is leafturner/locust-gutenberg-test.

ΔH and ROUGE-1 were not measured. Do not read missing columns as zeros. Do not call the DistilBART strings hallucinations.

Environment

The Gutenberg fetch used User-Agent: locust-gutenberg-test/0.1. First 512 tokens of the collapsed story body (from Many years ago to the Gutenberg end marker). Body after whitespace-collapse: 297351 characters. Window: 2403 characters, 512 tokens.

Command and stdout

$ python benchmark.py
encoding=cl100k_base
window_tokens=512
stride	perturbed	TIR
1	3272	+539.1%
2	1764	+244.5%
3	1216	+137.5%
4	1016	+98.4%
# ΔH and ROUGE-1 are not measured here. They need an embedder and a summarizer.

Exit code 0. The script asserts strip_zero_width(perturbed) == window after every stride. That assert held.

Table

TIR = (N_perturbed - N_clean) / N_clean with N_clean = 512.

TIR falls as stride grows. Stride 2 is what /rss.xml uses. Even stride 4 almost doubles the window.

These counts will move if Gutenberg edits the file or tiktoken ships a new cl100k_base merge table. Re-run the repo instead of treating this page as a frozen constant.

Window edges (so we know what was sliced)

Start:

Many years ago, there was an Emperor, who was so excessively fond of new clothes, that he spent all his money in dress. He did not trouble himself in the least about his soldiers; nor did he care to go either to the theatre or the chase, except for the opportunities then afforded him for displaying his new clothes. He had a different suit for each hour of the day; and as of any other king or emper

End:

the city had heard of the wonderful property the cloth was to possess; and all were anxious to learn how wise, or how ignorant, their neighbors might prove to be. “I will send my faithful old minister

Extra: sandbox sentence (not the official script)

Same helper, same encoding, one sentence from the article sandbox. Not part of benchmark.py. Logged so the island and the Python tree can be compared.

Clean text: The Emperor walked through the grand hall in his invisible robe. Clean tokens: 12.

A 12-token sentence is a worse TIR than a 512-token window. Short windows have less punctuation and fewer already-atomic tokens to dilute the husks. Do not mix these two tables.

QA stdout

$ python qa.py --text-file 1597-0.txt
encoding=cl100k_base
window_tokens=512
reader_max_length=512
n_questions=14
condition  stride n    em    f1
clean      -      14   0.571 0.571
S=1        1      14   0.000 0.000
S=1_san    1      14   0.571 0.571
S=2        2      14   0.000 0.000
S=2_san    2      14   0.571 0.571
S=3        3      14   0.143 0.147
S=3_san    3      14   0.571 0.571
S=4        4      14   0.357 0.357
S=4_san    4      14   0.571 0.571

Every sanitized row matches clean. Strides 1 and 2 are zero. The reader max length is 512 of its own tokens, so part of those zeros can be truncation.

DistilBART stdout

$ python summarize.py --text-file 1597-0.txt
model=sshleifer/distilbart-cnn-12-6
window_tokens=512
reader_max_input=1024

===== clean =====
An Emperor was so fond of new clothes that he spent all his money in dress . One day, two rogues, calling themselves weavers, made their appearance at the court . They gave out that they knew how to weave stuffs of the most beautiful colors and elaborate patterns . “These must, indeed, be splendid clothes!” thought the Emperor. “Had I

===== S=1 =====
M a n’s an E m p’e m r i y y s a n a l h e, “he is s t d a d o f,  s

===== S=2 =====
Ma ny ye ar’s ag o, th er e wa s an Em pe r, wh o wa s so ex ce ss y fo l i d no t tr ou bl

===== S=4 =====
Many year’s ago, ther’e was an Empe’ror, who was so exce’ssiv ely fond of new clot’hes, that he spen’t all his mone’y in dres s . He had a diff eren’d suit for each hour of the day

===== S=2_san =====
An Emperor was so fond of new clothes that he spent all his money in dress . One day, two rogues, calling themselves weavers, made their appearance at the court . They gave out that they knew how to weave stuffs of the most beautiful colors and elaborate patterns . “These must, indeed, be splendid clothes!” thought the Emperor. “Had I

S=2_san matches clean. S=1 and S=2 are shredded leftover pieces. Invisible U+200C marks in the raw stdout are written here as spaces so the page stays readable.

What this log is not

• Not an embedding study.
• Not ROUGE.
• Not the contaminated live-RSS +8.0% figure. That number is a different corpus and a dirty baseline; it stays on the methodology page as a negative control.
• Not a claim that a large chat model cannot recover the letters. One regex did, every stride.
• Not a hallucination study. DistilBART emitted copy-garbage. The QA reader missed spans.

See also

• Methodology and sandbox
• Locust I: init
• Runner: github.com/leafturner/locust-gutenberg-test