How does processing reduce data volume before review in e-discovery (de-NISTing, deduplication, and date filtering)?

In e-discovery, processing is the stage between collection and review where raw electronically stored information (ESI) is normalized, indexed, and culled. Because attorney review is the most expensive stage and its cost scales with volume, reducing the document population before review is one of the highest-leverage steps in the entire workflow. Three culling techniques do much of that work: de-NISTing, deduplication, and date filtering.

De-NISTing

De-NISTing removes files that are not user-created content — operating-system files, application binaries, and other “known” system files that have no evidentiary value. These are identified against the National Software Reference Library maintained by NIST, which publishes hash values for common software. Matching files are filtered out automatically, so reviewers never see thousands of program files mixed in with documents and email.

Deduplication

Deduplication identifies copies of the same file or message — typically by comparing cryptographic hash values — so each unique item is reviewed once rather than many times. Deduplication can run within a single custodian’s data or globally across all custodians; the choice affects how the same document is tracked across the matter. Near-duplicate detection and email threading are related techniques that group similar items so a reviewer can assess them together.

Date filtering

Date filtering limits the population to the time period relevant to the dispute. When the parties agree on (or the court sets) a date range tied to the events at issue, items outside that window can be set aside. Date filtering should be applied carefully, since metadata dates vary by file type and time zone, and an overly narrow range can exclude relevant material.

Why it matters

These techniques support proportionality — the principle that discovery effort should be reasonable relative to the needs of the case. The Federal Rules of Civil Procedure embed proportionality in US federal civil litigation, though specifics differ across state courts and other countries. Done transparently and documented for the record, processing makes review faster and cheaper without sacrificing a defensible process. Parties commonly disclose their culling criteria so the methodology can be defended if challenged.