What is the difference between near-duplicate detection and exact deduplication in document review?

Both techniques reduce the volume of material a team must review, but they answer different questions. Exact deduplication asks whether two files are identical. Near-duplicate detection asks whether two documents are substantially similar even when they are not the same file. Understanding the distinction matters for cost, consistency, and defensibility in e-discovery.

Exact Deduplication

Exact deduplication identifies files that are bit-for-bit identical, typically by computing a cryptographic hash value (such as MD5 or SHA-1) for each item. If two files produce the same hash, they are treated as the same document, and duplicates are suppressed so a reviewer sees the item only once.

Key points:

• It is precise and reproducible: the same input always yields the same hash.
• It works within a custodian’s data or across an entire collection (often called global or horizontal deduplication).
• A single changed byte (different metadata, an added signature, a converted format) produces a different hash, so two clearly related documents may not be flagged as exact duplicates.

Near-Duplicate Detection

Near-duplicate detection compares the content of documents and groups those that are highly similar based on shared text, often expressed as a percentage of overlap. A final contract and its prior draft, or an email and a lightly edited reply, would not have matching hashes but would be recognized as near-duplicates.

This is useful for:

• Grouping related documents so reviewers can code them consistently.
• Speeding review by letting a reviewer assess a cluster together rather than item by item.
• Surfacing small but meaningful differences between versions.

How They Work Together

The two are complementary, not interchangeable. Exact deduplication removes true copies; near-duplicate detection organizes what remains. Used together, they cut redundant effort and promote consistent decisions across similar documents.

Whatever method you use, document your process. Producing parties should be prepared to explain their approach if challenged, and the governing standards differ by jurisdiction (U.S. federal courts, state courts, and other countries each have their own rules). Proportionality and reasonableness, central themes in the Federal Rules of Civil Procedure and in Sedona Conference guidance, support these volume-reduction tools when they are applied transparently and defensibly.