Map exact duplicate files without deleting them
Group matching file hashes and sizes while keeping every source record.
Tools for the documented steps
Practice relevance
Cross-practice idea · A general operational method. Review the stated tool and jurisdiction limits before use.
The problem
Group matching file hashes and sizes while keeping every source record.
Why it matters: Identical bytes can arrive from different custodians or batches. A duplicate map can make review easier, but preserving provenance and inconsistent metadata avoids treating a matching hash as permission to delete or withhold evidence.
Sources & evidence
Proposed / not tested. The public source describes the problem. Documentation supports only the stated method components; no result from this recipe has been verified.
What’s been verified
Source methods and reported problems have been reviewed. We have not run the adapted workflow. The sample below describes expected behavior on synthetic data, not a customer result.
What to measure: In a synthetic pilot, count correctly found exceptions, false alarms, and unresolved inputs. No time or cost saving is assumed.
- Pain evidence: Simmons & Simmons: Dealing with duplicates ↗Simmons & Simmons: Dealing with duplicates, 2021-09-29, older, is a first-person law-firm case study: the team found extensive duplicate Office files, including repeated PowerPoint attachments, increasing manual review/redaction work. This is a firm-authored account promoting its services, not independent performance validation; no claimed savings transfer to this recipe. Document production, practitioner comments 2026-07-22, independently describe the same document repeatedly attached to emails or held by multiple custodians. The original post also complains about repeated pages inside PDFs, which this narrower whole-file method does not solve.
- Pain evidence: Document production ↗Additional self-reported public account; no independent identity or outcome verification.
- Method: PowerShell Get-FileHash ↗PowerShell Get-FileHash documents content-based hashes, default SHA-256, and that identical content can have different names. A whole-file hash cannot identify duplicates of individual pages inside otherwise different PDFs.
Source checked: October 15, 2026. Pain accounts are self-reported. Documentation supports components; this adapted recipe and prompt have not been tested. We wrote these prompts. The adapted workflow hasn’t been tested.
Suggested approach
Start when: New production receipt or document-review preparation
- group administrator-generated exact whole-file hashes while preserving provenance
- Keep source references and report unknowns instead of filling gaps.
- The responsible reviewer checks the report before any separately approved action.
Limits and review
The assistant drafts or reports only. Check source records and review the output before a human performs any separately approved send or change. No automatic legal decision, money movement, account connection or publication is included.
Who reviews it: Litigation staff verifies samples and all provenance. Test two differently named identical files, two PDFs differing by one page and identical content in different families. This assists navigation, not discovery decisions. Local research batch discovery-inventory evidence can strengthen this entry rather than become a separate duplicate-detection entry.
What you need
Your firm-approved AI assistant · prepared files
- Existing PowerShell 7 on Windows and IT approval to run the bundled helper under the existing script policy. No install, elevation or execution-policy bypass is part of this route.
- An approved stable, fully local working snapshot and an existing output folder outside it on the same fixed local drive. Both paths must be absolute drive-letter paths with ordinary, non-reparse-point ancestors. No live/cloud/network/mapped-drive repositories, original forensic media or restricted paths.
- The records/IT owner must exclude credential/profile directories and all secret material before selecting the snapshot. The helper uses -Force and hashes every readable regular file, including hidden files; it does not detect or redact secrets.
- Records owner supplies scope and identity/custodian/family metadata; hashes establish byte equality only.
- Actual receipt/batch log, custodian aliases and document-family metadata supplied by the records owner.
- Firm-approved Excel; optional Copilot entitlement and tenant approval for the exact minimized data. Native Excel analysis can be done manually if Copilot is absent.
- Keep raw exports and identifying source pointers in a separate restricted location. The AI workbook contains only approved alias-based tables and visible policy/scope text, never raw client records on another tab.
- Use synthetic inputs first. Keep real data in firm-approved accounts and processing, with appropriate access, confidentiality and retention controls. Aliases alone do not anonymize records. Never paste credentials, API keys or unnecessary client details.
How to use
Advanced · Advanced proposed local-snapshot route; helper statically reviewed only and not runtime-tested
- SHA256 groups byte-identical files only. Different encodings, metadata or OCR layers can change hashes even when pages look the same.
- Equal file bytes do not justify deletion: custodians, attachments, families, production history and legal holds remain distinct.
- Advanced helper route: existing Windows PowerShell 7, approved stable snapshot, IT code review and a safe synthetic test under existing policy are prerequisites. The helper is not a security boundary against concurrent or malicious filesystem changes. No runtime test was performed here.
Check access and setup
- Actual account access, plan and source-data completeness have not been tested.
Collect real local file sizes and SHA256 hashes
PowerShell 7 on Windows · Existing PowerShell 7 session > reviewed Inventory-ApprovedLocalSnapshot.ps1
Have IT review the bundled helper and test it on a synthetic fixture under existing script policy before using approved work files. Set absolute snapshot and output paths on the same fixed local drive, with the output directory outside the snapshot. Before running, exclude credential/profile directories and secret material: -Force includes hidden files and there is no automatic secret detection. Run the command below with approved paths. The helper checks path forms, drive type and every directory ancestor, then walks ordinary directories one level at a time without following reparse points, records read/enumeration errors, hashes readable regular files with SHA256, and writes new inventory/issues/control files outside the input. It does not change source bytes. Reads may affect access metadata; this is not forensic acquisition.
- Expected output
- Native-cmdlet-derived local_row_id,relative_path,original_name,size_bytes,sha256,observed_last_write_utc,captured_at_utc plus issues/control logs. local_row_id is a run-local row reference, not a persistent document ID.
- Next step
- Inspect the issue/control logs before mapping the received-file manifest. Import CSV via Excel Data > From Text/CSV > Transform Data; set IDs, hashes and paths to Text. Do not double-click untrusted CSVs or enable formulas/macros/links.
- Check before continuing
- Any skipped/unreadable/changing file is a coverage gap. No claims about deleted files, embedded attachments, alternate streams, cloud versions or access permissions. No repair/delete action. Stop if any path is rejected or changes during capture; do not bypass checks, elevate permissions, install PowerShell or change execution policy as part of this recipe.
Map the observed records to the starter input
Excel and authorized staff · Separate analysis workbook > Home > Format as Table
Populate file-manifest.csv: assign file_id using a persistent audit crosswalk; map original_path,original_name,size_bytes,sha256 from observed files; obtain source_batch,custodian_alias,family_id,received_at from the actual receipt log. File creation/last-write timestamps are not receipt times. If the snapshot changed paths, preserve the documented original-path crosswalk rather than guessing. Put scope.txt and all errors/exclusions into Scope; keep raw identifying paths separate from the approved AI version. Create named tables for FileManifest. Put the policy and review scope in visible worksheets or paste them into the next prompt.
- Expected output
- FileManifest with true byte hashes and separately sourced provenance.
- Next step
- Check the mapping against each original source before using AI.
- Check before continuing
- Aliases and missing policy fields are staff-created; do not relabel a native export as the kit schema. Unknown stays blank/unknown, not false or zero.
Review the bounded evidence with Copilot
Excel Copilot, optional · Approved analysis workbook > Copilot icon at lower right > Chat only
Use Chat only and submit the prompt. Without Copilot: sort/group exact SHA256+size, retaining every source row and its provenance.
- Expected output
- Exact-byte duplicate map with provenance and uncertainty preserved.
- Next step
- Copy the draft exception list to the reviewer; retain row references.
- Check before continuing
- Check every flagged row and the scope. A fluent answer is not evidence of completeness; no automatic workbook or source-system edits.
Approve the review result
Final human reviewResponsible staff reviewer · Reviewed local worksheet
The records owner spot-checks byte-equal groups, confirms hashes/coverage and preserves every original and family association. Use the map for review organization only.
- Expected output
- Reviewed duplicate map; no files deleted or altered.
- Next step
- Any external message, source change, payment or deletion requires its own authorized process.
- Check before continuing
- Hash equality is technical evidence about bytes, not a legal retention, privilege or relevance decision.
What you’ll get
duplicate-map.csv: group_id, file_id, source_batch, sha256, size_bytes, original_path, custodian_alias, family_id, evidence_conflict, review_note. Preserve every file and report missing or inconsistent hashes.
Inputs and a sample test Synthetic example · not a measured result
Input files
- file-manifest.csv: file_id, source_batch, original_path, original_name, size_bytes, sha256, custodian_alias, family_id, received_at
- scope.txt: approved repository, export coverage, hash method and unresolved page-level duplicates
Synthetic example
Synthetic inputs are included in the starter kit. file-manifest.example.csv scope.example.txt
Expected output
F1/F2: exact same hash and 100-byte size, but preserve both file IDs, distinct batches/custodians/families and paths in a candidate duplicate group. F5: hash matches F1/F2 but size 101 conflicts with 100; flag inconsistent manifest evidence, exclude F5 from the exact-size group and require verification of the whole hash cluster. F3: different hash/size remains ungrouped. Do not infer near/page-level duplication. F4: missing hash remains unresolved, not unique-by-proof. No file is deleted, withheld, or selected for production based on this map.
Test a failure
Litigation staff verifies samples and all provenance. Test two differently named identical files, two PDFs differing by one page and identical content in different families. This assists navigation, not discovery decisions.
Try with the sample files
Follow the tool steps above and use the inputs named in each step. Where provided, native-* files are blank scaffolds and NATIVE-INPUT-MAPPING explains their fields. Original examples and the optional full review prompt use a separate prepared-input schema. Keep them separate unless the mapping explicitly connects them. Downloading a kit does not connect accounts or prove the workflow works.
Download inputs and checklist 9 files
Instructions, blank templates and separate synthetic examples; these files do not install an automation. Blank templates still need verified source data.
- Download starter kit All inputs, synthetic examples and review notes in one ZIP. No automation is installed.
- Start here: get the sources and prepare the review Documentation-checked acquisition, explicit manual mapping, exact prompts and human review limits
- file-manifest.csv Blank input template with exact recipe headers.
- file-manifest.example.csv Separate synthetic fixture input. Do not mix these rows into real exports.
- scope.txt Blank text input worksheet. Fill its fields with approved evidence; no default assumptions are supplied.
- scope.example.txt Separate synthetic text input fixture; use with the example CSV files only.
- Expected fixture findings Read-only expected output and failure conditions for the included synthetic input files.
- Setup and review checklist Recipe-specific prerequisites, file mapping, prompt, setup steps, checklist and limits.
- Advanced: approved local snapshot inventory helper Read an approved stable local snapshot and write inventory/error outputs outside its root; requires authorized IT review and synthetic validation first
Optional full review prompt for prepared inputs
Prepared-input review prompt with Your firm-approved AI assistant · prepared files
Proposed / not tested · editorial prompt. This prompt uses the original prepared-input schema shown in the sample test. Step prompts may use different tables. Follow NATIVE-INPUT-MAPPING where supplied; do not mix the original examples with native-step inputs without mapping and checking every field.