A large share of the text used to build language model training sets and retrieval corpora starts life as a PDF: research papers, reports, filings, manuals. Almost none of it is cleaned before it enters a pipeline, because at the scale these corpora are built, manual review of individual documents is not feasible. The result is that a measurable fraction of "text" data fed into modern NLP systems is not the author's prose at all. It is the author's prose plus a layer of extraction artifacts that the model has to learn to work around, and never quite does for free.
Tokenization doesn't forgive broken words
A subword tokenizer, the kind used by essentially every modern language model, builds its vocabulary from statistics over a training corpus: which character sequences appear often enough to deserve their own token. A word split by a PDF hyphenation artifact, "infor-" on one line and "mation" on the next, does not tokenize as the word "information." It tokenizes as two unrelated fragments, each competing for vocabulary space with genuinely common subwords, and each carrying none of the semantic weight the model has learned to associate with the whole word. At small scale this is noise. At the scale of a pretraining corpus, it is millions of wasted tokens and a measurable dilution of how cleanly the model's vocabulary maps onto real words.
The problem compounds during fine-tuning and evaluation, where corpora are smaller and every example matters more. A model fine-tuned on a document set full of hyphen splits learns, in a small but real way, that this is a normal way for words to appear, and can reproduce the same artifact in its own output.
Retrieval and chunking depend on sentence boundaries
Retrieval-augmented systems split documents into chunks, typically by paragraph or by a fixed token window, and embed each chunk for similarity search. This process assumes the document's paragraph and sentence boundaries are real. A PDF extraction that has merged three visually distinct sentences into one hard-wrapped block, or split one sentence into three because of page-boundary noise, hands the chunker a false picture of where ideas begin and end. A chunk boundary landing in the middle of a sentence produces two embeddings, neither of which represents a complete thought, and a query that should retrieve that passage may miss both fragments.
Running heads are noise the model has to learn around
A repeated document title or page number, extracted inline with body text every few hundred words, is not harmless filler. It is a token sequence the model sees thousands of times across a corpus, in a position (interrupting a paragraph) where it never appears in genuinely well-formed writing. Left uncleaned, it becomes a learned pattern rather than an obvious anomaly, one more small distortion in the model's sense of what ordinary text looks like.
A practical cleaning checklist for training corpora
None of this requires a bespoke NLP pipeline for a single document. The same four corrections matter whether you are preparing one file or scripting a corpus of ten thousand:
- Rejoin hyphenated words using a rule that only collapses the break when a lowercase letter follows, so genuine hyphenated compounds survive.
- Merge hard-wrapped lines back into full sentences, using the absence of terminal punctuation as the signal that a line continues.
- Strip running heads and folios by detecting text that repeats near the top or bottom of most pages.
- Preserve real structure, meaning paragraph breaks and list items, since a chunker needs those boundaries as much as it needs clean sentences.
For a single document or a quick sanity check before a larger cleaning run, the text cleaner applies exactly these four corrections and shows a receipt of what changed, entirely in the browser. The field guide explains the detection logic behind each rule in more detail, which is useful reference if you are implementing similar cleaning in a batch pipeline rather than doing it by hand.
Clean training data is not a cosmetic improvement. It is the difference between a model learning your domain's vocabulary and a model learning your domain's vocabulary plus a permanent, low-level layer of PDF formatting noise it was never supposed to see.