Arihant Global Ventures Inc. Machine-readable Markdown for this page: send Accept text/markdown, or open /guides/deduplication-at-scale-document-corpora/index.md. Site overview for language models: /llms.txt. Full guide text: /llms-full.txt.

Canadian enterprise & public-sector advisoryPrincipal-led deliveryToronto, ON, Canada

Arihant Global Ventures Inc.
Discuss an Assessment
← All guides

Guide · December 29, 2025 · 3 min read

Deduplication at Scale: Collapsing Duplicates in Million-Page Corpora

The same report appears in a case file four times. Exact hashing, near-duplicate detection on OCR text, and version chains — how to collapse duplicates for readers without ever deleting the record.

Document AIData EngineeringOCR

Open any long-running case file and you will find the same document more than once. A specialist’s report gets faxed, scanned, emailed, and scanned again. A form is submitted, then resubmitted with one field changed. Nobody did anything wrong — files accumulate over years, through many hands, and duplication is the natural state of a large corpus.

The cost is real. Duplicates inflate every downstream step: more pages to OCR, more chunks to index, more noise in retrieval. Worse, they corrupt judgment. A reviewer who reads the same opinion three times unconsciously weights it as three opinions. Deduplication is not housekeeping. In a corpus measured in millions of pages, it is a correctness feature.

Level 1: Exact duplicates

The easy case. Two scans of the same document produce files that may differ as bytes — different scanners, different compression — while being identical as documents. Hashing the file bytes catches only true file copies. The reliable approach is to hash the normalised OCR text: same text, same hash, exact duplicate. This runs after OCR, costs almost nothing, and typically removes a meaningful share of a mature corpus before any intelligence is applied.

Level 2: Near duplicates

Harder and more valuable. The same report scanned twice will OCR slightly differently — a smudge becomes a different character, a header is cropped. Text similarity techniques (shingling with MinHash-style fingerprinting is the classic, well-understood approach) find documents that are 95% the same document. These collapse into a group with one canonical representative.

The threshold matters. Set it too aggressively and you merge documents that differ in one material fact — a corrected date, an amended diagnosis. In case files, a one-line difference can be the whole point. Conservative thresholds, with borderline groups routed to human review, are the professional default.

Level 3: Versions are not duplicates

The same form submitted in March and again in June with updated information is not a duplicate. It is a version chain, and collapsing it would destroy the record’s history. The pipeline must distinguish “same document, copied” from “same template, new facts.” Dates, signatures, and field-level diffs are the signals. Versions stay — ordered, linked, and labelled — while true duplicates collapse.

There is a subtler case in medical files: the same report legitimately sent to two different parties, filed twice with different cover sheets. For reading purposes it is one report. For the record, both filings happened. The answer is to collapse for the reader and preserve for the record: one canonical copy in the reading view, every instance still present and accounted for in the underlying file.

In regulated and adjudicative contexts, deduplication must never mean deletion. The pipeline marks duplicates, links them to their canonical copy, and keeps a complete audit trail: what was grouped, by which method, at what confidence, and who confirmed borderline cases. The original file remains reproducible at any time. What changes is what the reviewer has to wade through — and what retrieval returns. A deduplicated index means search results show the report once, cited properly, instead of four near-identical chunks crowding out everything else.

Where dedup sits in the pipeline

Deduplication runs after OCR and before indexing, and it is idempotent: when a corrected or additional copy of a document arrives later, it joins its group rather than creating a new island. Done in this order, it compounds — cheaper processing, cleaner retrieval, and reviewers who can trust that repetition in the file means emphasis in the evidence, not accidents of filing.

Start with a conversation.

A free 30-minute introduction to discuss your AI governance challenge, constraints, and whether we can help. No project brief required.

Contact us for current availability. Scope, timing, and fees are agreed before paid work begins.