Processing and De-Duplication of ESI

Once electronically stored information (ESI) has been identified, preserved, and collected, it cannot move directly into attorney review. Raw collections are voluminous, inconsistent in format, and riddled with redundancy: the same email may exist in a dozen mailboxes, the same attachment may be duplicated across shared drives, and system files irrelevant to any dispute may outnumber substantive documents. Processing is the engineering stage of electronic discovery that transforms this heterogeneous mass into a normalized, searchable, and de-duplicated dataset suitable for filtering and review. It is where the bulk of cost reduction in a discovery project is realized, and where defensibility is most easily won or lost.

Processing sits squarely within the analysis-through-review phases of the widely used reference model for electronic discovery. Done well, it shrinks the review population by an order of magnitude while preserving the metadata, family relationships, and chain of custody that make the resulting production defensible. Done carelessly, it can strip critical metadata, break attachment families, or silently discard responsive material, exposing a producing party to challenges over completeness and authenticity.

What Processing Actually Does

Processing is a sequence of automated operations applied to collected ESI. The core steps generally include:

• Decompression and expansion, where container files such as ZIP archives, mailbox stores (PST, OST, mbox), and disk images are recursively opened so that every embedded item is exposed as a discrete object.
• File identification and normalization, where each item’s true type is determined by its binary signature rather than its extension, and items are catalogued consistently.
• Text and metadata extraction, where the searchable text of each document and its associated metadata fields (author, dates, recipients, file paths, and the like) are pulled into an index.
• Optical character recognition (OCR) for image-only documents that contain no extractable text.
• Exception handling, where corrupt, encrypted, or password-protected files that cannot be processed are logged for remediation rather than ignored.

A central discipline throughout is the preservation of document families: an email and its attachments must travel together as a unit, and the parent-child relationship must be maintained so that review and production decisions are made coherently. Equally important is preserving metadata. Original system and application metadata is often more probative than document content, and reckless processing that overwrites timestamps or discards fields can destroy evidence and invite spoliation arguments.

Culling and Filtering

Before or during processing, parties apply culling techniques to remove material that has no bearing on the matter. The most common is de-NISTing, the removal of known, benign system and application files identified against a published reference library of file hashes maintained by a federal standards body. These executables, fonts, and operating-system artifacts carry no evidentiary value and can be safely excluded.

Beyond de-NISTing, parties typically apply date-range filters, keyword and search-term filters, file-type filters, and custodian or source filters to narrow the population to potentially relevant items. The scope and methodology of these filters are frequently the subject of negotiation between opposing parties and should be documented, because a producing party may later need to demonstrate that its culling was reasonable and proportional.

De-Duplication: Hashing and Scope

De-duplication is the identification and suppression of identical copies so that each unique item is reviewed only once. The mechanism is cryptographic hashing: an algorithm computes a fixed-length value (a hash, using functions such as MD5 or SHA-1) from a file’s contents. Two items that produce the same hash are treated as identical; a single changed byte produces a completely different value. For email, hashing typically combines a defined set of fields (such as sender, recipients, subject, body, and sent time) so that messages are compared on their substantive content rather than on incidental storage differences between mailboxes.

The critical decision is the scope of de-duplication:

• Custodian-level (vertical) de-duplication suppresses duplicates only within a single custodian’s data, so each custodian retains their own copy of a shared document. This preserves a complete picture of what each individual possessed.
• Global (horizontal) de-duplication suppresses duplicates across the entire collection, keeping only one copy regardless of how many custodians held it. This maximizes volume reduction but requires careful tracking of all custodians associated with the suppressed copies.

Whichever scope is chosen, defensible practice requires maintaining a custodian or duplicate-custodian field that records every source where a suppressed copy was found, so that the producing party can still answer who possessed a given document. Related techniques include email threading, which groups messages in a conversation and identifies the most inclusive message containing all prior content, and near-duplicate detection, which clusters textually similar but non-identical documents to promote consistent review decisions.

Defensibility, Proportionality, and Documentation

Processing and de-duplication choices are not merely technical; they are legal decisions that must withstand scrutiny. The governing framework for civil litigation expects discovery to be proportional to the needs of the case, and it provides safe-harbor considerations for ESI lost through the routine, good-faith operation of systems. Influential consensus guidance from leading legal-practitioner bodies has long emphasized that parties should adopt reasonable, documented, and transparent methodologies rather than chase perfection.

The practical implications are consistent across authorities. Producing parties should preserve original metadata and document families; record the tools, settings, hash algorithms, and de-duplication scope used; log and remediate processing exceptions; and be prepared to explain and defend their methodology. Sound recordkeeping principles reinforce this: records should remain authentic, reliable, and usable throughout their handling, and processing must not compromise those qualities. Where parties disclose and, ideally, negotiate their processing protocols early, disputes over completeness and form of production are far less likely to derail the matter.

In short, processing and de-duplication are the stage where a discovery project becomes both affordable and defensible. The engineering goal is dramatic volume reduction; the legal goal is an unbroken, well-documented chain in which nothing responsive is lost and nothing produced can be credibly challenged as unreliable. Achieving both at once is the mark of a mature electronic-discovery program. For broader context, see the ediscovery topic hub.