# Map exact duplicate files without deleting them

Canonical page: https://agents.paretoid.com/workflows/map-exact-duplicate-files-without-deleting-them/

Category: Operations & Reporting

Practice relevance: cross-practice. A general operational method. Review the stated tool and jurisdiction limits before use.

Jurisdiction/provider scope: 

Group matching file hashes and sizes while keeping every source record.

Status: Proposed / not tested. The public source describes the problem. Documentation supports only the stated method components; no result from this recipe has been verified.

Verification: Source methods and reported problems have been reviewed. We have not run the adapted workflow. The sample below describes expected behavior on synthetic data, not a customer result.

## The problem

Group matching file hashes and sizes while keeping every source record.

Why it matters: Identical bytes can arrive from different custodians or batches. A duplicate map can make review easier, but preserving provenance and inconsistent metadata avoids treating a matching hash as permission to delete or withhold evidence.

## Sources and evidence

- pain: [Pain evidence: Simmons & Simmons: Dealing with duplicates](https://www.simmons-simmons.de/en/publications/ckue45m3900ir0a91m16isv4s/dealing-with-duplicates-case-study)
  Simmons & Simmons: Dealing with duplicates, 2021-09-29, older, is a first-person law-firm case study: the team found extensive duplicate Office files, including repeated PowerPoint attachments, increasing manual review/redaction work. This is a firm-authored account promoting its services, not independent performance validation; no claimed savings transfer to this recipe. Document production, practitioner comments 2026-07-22, independently describe the same document repeatedly attached to emails or held by multiple custodians. The original post also complains about repeated pages inside PDFs, which this narrower whole-file method does not solve.

- pain: [Pain evidence: Document production](https://www.reddit.com/r/paralegal/comments/1v3i569/document_production/)
  Additional self-reported public account; no independent identity or outcome verification.

- method: [Method: PowerShell Get-FileHash](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/get-filehash?view=powershell-7.5)
  PowerShell Get-FileHash documents content-based hashes, default SHA-256, and that identical content can have different names. A whole-file hash cannot identify duplicates of individual pages inside otherwise different PDFs.

Source checked: October 15, 2026. Pain accounts are self-reported. Documentation supports components; this adapted recipe and prompt have not been tested.

## Suggested approach

Start when: New production receipt or document-review preparation

1. group administrator-generated exact whole-file hashes while preserving provenance

2. Keep source references and report unknowns instead of filling gaps.

3. The responsible reviewer checks the report before any separately approved action.

## What you need

- Existing PowerShell 7 on Windows and IT approval to run the bundled helper under the existing script policy. No install, elevation or execution-policy bypass is part of this route.

- An approved stable, fully local working snapshot and an existing output folder outside it on the same fixed local drive. Both paths must be absolute drive-letter paths with ordinary, non-reparse-point ancestors. No live/cloud/network/mapped-drive repositories, original forensic media or restricted paths.

- The records/IT owner must exclude credential/profile directories and all secret material before selecting the snapshot. The helper uses -Force and hashes every readable regular file, including hidden files; it does not detect or redact secrets.

- Records owner supplies scope and identity/custodian/family metadata; hashes establish byte equality only.

- Actual receipt/batch log, custodian aliases and document-family metadata supplied by the records owner.

- Firm-approved Excel; optional Copilot entitlement and tenant approval for the exact minimized data. Native Excel analysis can be done manually if Copilot is absent.

- Keep raw exports and identifying source pointers in a separate restricted location. The AI workbook contains only approved alias-based tables and visible policy/scope text, never raw client records on another tab.

- Use synthetic inputs first. Keep real data in firm-approved accounts and processing, with appropriate access, confidentiality and retention controls. Aliases alone do not anonymize records. Never paste credentials, API keys or unnecessary client details.

## How to use

Advanced proposed local-snapshot route; helper statically reviewed only and not runtime-tested

- SHA256 groups byte-identical files only. Different encodings, metadata or OCR layers can change hashes even when pages look the same.

- Equal file bytes do not justify deletion: custodians, attachments, families, production history and legal holds remain distinct.

- Advanced helper route: existing Windows PowerShell 7, approved stable snapshot, IT code review and a safe synthetic test under existing policy are prerequisites. The helper is not a security boundary against concurrent or malicious filesystem changes. No runtime test was performed here.

Access and setup:

- Actual account access, plan and source-data completeness have not been tested.

### 1. Collect real local file sizes and SHA256 hashes

Tool: PowerShell 7 on Windows

Location: Existing PowerShell 7 session > reviewed Inventory-ApprovedLocalSnapshot.ps1

Have IT review the bundled helper and test it on a synthetic fixture under existing script policy before using approved work files. Set absolute snapshot and output paths on the same fixed local drive, with the output directory outside the snapshot. Before running, exclude credential/profile directories and secret material: -Force includes hidden files and there is no automatic secret detection. Run the command below with approved paths. The helper checks path forms, drive type and every directory ancestor, then walks ordinary directories one level at a time without following reparse points, records read/enumeration errors, hashes readable regular files with SHA256, and writes new inventory/issues/control files outside the input. It does not change source bytes. Reads may affect access metadata; this is not forensic acquisition.

Prompt:

```text
& '.\Inventory-ApprovedLocalSnapshot.ps1' -Root 'C:\ApprovedSnapshot' -OutputPrefix 'C:\ApprovedAudit\capture-20261015-01'
```

Expected output: Native-cmdlet-derived local_row_id,relative_path,original_name,size_bytes,sha256,observed_last_write_utc,captured_at_utc plus issues/control logs. local_row_id is a run-local row reference, not a persistent document ID.

Next step: Inspect the issue/control logs before mapping the received-file manifest. Import CSV via Excel Data > From Text/CSV > Transform Data; set IDs, hashes and paths to Text. Do not double-click untrusted CSVs or enable formulas/macros/links.

Check before continuing: Any skipped/unreadable/changing file is a coverage gap. No claims about deleted files, embedded attachments, alternate streams, cloud versions or access permissions. No repair/delete action. Stop if any path is rejected or changes during capture; do not bypass checks, elevate permissions, install PowerShell or change execution policy as part of this recipe.

Execution status: Proposed helper, statically reviewed against official command documentation; no PowerShell runtime test or actual-file run performed

Step documentation: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-childitem?view=powershell-7.5 https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/get-filehash?view=powershell-7.5 https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/export-csv?view=powershell-7.5 https://support.microsoft.com/en-us/excel/get-started/import-or-export-text-txt-or-csv-files https://learn.microsoft.com/en-us/dotnet/api/system.io.driveinfo.drivetype?view=net-9.0 https://learn.microsoft.com/en-us/dotnet/api/system.io.path.getfullpath?view=net-9.0

### 2. Map the observed records to the starter input

Tool: Excel and authorized staff

Location: Separate analysis workbook > Home > Format as Table

Populate file-manifest.csv: assign file_id using a persistent audit crosswalk; map original_path,original_name,size_bytes,sha256 from observed files; obtain source_batch,custodian_alias,family_id,received_at from the actual receipt log. File creation/last-write timestamps are not receipt times. If the snapshot changed paths, preserve the documented original-path crosswalk rather than guessing. Put scope.txt and all errors/exclusions into Scope; keep raw identifying paths separate from the approved AI version. Create named tables for FileManifest. Put the policy and review scope in visible worksheets or paste them into the next prompt.

Expected output: FileManifest with true byte hashes and separately sourced provenance.

Next step: Check the mapping against each original source before using AI.

Check before continuing: Aliases and missing policy fields are staff-created; do not relabel a native export as the kit schema. Unknown stays blank/unknown, not false or zero.

Execution status: Official documentation checked 2026-10-15; account and workflow not personally tested

Step documentation: https://support.microsoft.com/en-us/excel/tutorial-import-data-into-excel-and-create-a-data-model

### 3. Review the bounded evidence with Copilot

Tool: Excel Copilot, optional

Location: Approved analysis workbook > Copilot icon at lower right > Chat only

Use Chat only and submit the prompt. Without Copilot: sort/group exact SHA256+size, retaining every source row and its provenance.

Prompt:

```text
Group FileManifest by exact valid SHA256 and size_bytes within Scope. Return each byte-identical group with every file_id, batch, custodian, family and receipt reference retained. Flag missing hashes/provenance or inconsistent size/hash combinations. Do not infer duplicates from names, merge family records, choose a canonical copy, classify relevance or recommend deletion. Different hashes are not proof of different substantive content. Cite rows.
```

Expected output: Exact-byte duplicate map with provenance and uncertainty preserved.

Next step: Copy the draft exception list to the reviewer; retain row references.

Check before continuing: Check every flagged row and the scope. A fluent answer is not evidence of completeness; no automatic workbook or source-system edits.

Execution status: Official documentation checked 2026-10-15; account and workflow not personally tested

Step documentation: https://support.microsoft.com/en-us/excel/copilot/get-started-with-copilot-in-excel https://support.microsoft.com/en-us/excel/copilot/frequently-asked-questions-about-copilot-in-excel

### 4. Approve the review result

Tool: Responsible staff reviewer

Location: Reviewed local worksheet

The records owner spot-checks byte-equal groups, confirms hashes/coverage and preserves every original and family association. Use the map for review organization only.

Expected output: Reviewed duplicate map; no files deleted or altered.

Next step: Any external message, source change, payment or deletion requires its own authorized process.

Check before continuing: Hash equality is technical evidence about bytes, not a legal retention, privilege or relevance decision.

Execution status: Official documentation checked 2026-10-15; account and workflow not personally tested

Step documentation: https://support.microsoft.com/en-us/excel/tutorial-import-data-into-excel-and-create-a-data-model

## What you will get

duplicate-map.csv: group_id, file_id, source_batch, sha256, size_bytes, original_path, custodian_alias, family_id, evidence_conflict, review_note. Preserve every file and report missing or inconsistent hashes.

## Human review and limits

Litigation staff verifies samples and all provenance. Test two differently named identical files, two PDFs differing by one page and identical content in different families. This assists navigation, not discovery decisions. Local research batch discovery-inventory evidence can strengthen this entry rather than become a separate duplicate-detection entry.

The assistant drafts or reports only. Check source records and review the output before a human performs any separately approved send or change. No automatic legal decision, money movement, account connection or publication is included.

What to measure: In a synthetic pilot, count correctly found exceptions, false alarms, and unresolved inputs. No time or cost saving is assumed.

## Original prepared-input schema

file-manifest.csv: file_id, source_batch, original_path, original_name, size_bytes, sha256, custodian_alias, family_id, received_at

scope.txt: approved repository, export coverage, hash method and unresolved page-level duplicates

Synthetic example input:

```text
Synthetic inputs are included in the starter kit.
file-manifest.example.csv
scope.example.txt
```

Expected synthetic output:

```text
F1/F2: exact same hash and 100-byte size, but preserve both file IDs, distinct batches/custodians/families and paths in a candidate duplicate group.
F5: hash matches F1/F2 but size 101 conflicts with 100; flag inconsistent manifest evidence, exclude F5 from the exact-size group and require verification of the whole hash cluster.
F3: different hash/size remains ungrouped. Do not infer near/page-level duplication.
F4: missing hash remains unresolved, not unique-by-proof.
No file is deleted, withheld, or selected for production based on this map.
```

Failure check: Litigation staff verifies samples and all provenance. Test two differently named identical files, two PDFs differing by one page and identical content in different families. This assists navigation, not discovery decisions.

## Starter files

Native blank scaffolds and original synthetic examples may use different schemas. Follow NATIVE-INPUT-MAPPING where supplied. These files do not connect accounts or install automation.

- [Download starter kit](https://agents.paretoid.com/assets/starter-kits/s14-starter-kit.zip) — All inputs, synthetic examples and review notes in one ZIP. No automation is installed.

- [Start here: get the sources and prepare the review](https://agents.paretoid.com/assets/starter-kits/s14/START-HERE.txt) — Documentation-checked acquisition, explicit manual mapping, exact prompts and human review limits

- [file-manifest.csv](https://agents.paretoid.com/assets/starter-kits/s14/file-manifest.csv) — Blank input template with exact recipe headers.

- [file-manifest.example.csv](https://agents.paretoid.com/assets/starter-kits/s14/file-manifest.example.csv) — Separate synthetic fixture input. Do not mix these rows into real exports.

- [scope.txt](https://agents.paretoid.com/assets/starter-kits/s14/scope.txt) — Blank text input worksheet. Fill its fields with approved evidence; no default assumptions are supplied.

- [scope.example.txt](https://agents.paretoid.com/assets/starter-kits/s14/scope.example.txt) — Separate synthetic text input fixture; use with the example CSV files only.

- [Expected fixture findings](https://agents.paretoid.com/assets/starter-kits/s14/expected-fixture-result.txt) — Read-only expected output and failure conditions for the included synthetic input files.

- [Setup and review checklist](https://agents.paretoid.com/assets/starter-kits/s14/README.txt) — Recipe-specific prerequisites, file mapping, prompt, setup steps, checklist and limits.

- [Advanced: approved local snapshot inventory helper](https://agents.paretoid.com/assets/starter-kits/s14/Inventory-ApprovedLocalSnapshot.ps1) — Read an approved stable local snapshot and write inventory/error outputs outside its root; requires authorized IT review and synthetic validation first

## Optional original prepared-input prompt

Destination: Your firm-approved AI assistant · prepared files

This original prompt uses its own prepared-input schema. Do not mix it with native-step tables without the supplied mapping.

```text
Group this sanitized manifest by exact SHA-256 value and file size. Retain every file ID, source batch, custodian and family relationship. Output exact whole-file duplicate groups and an inventory of files not grouped. Flag missing hashes and inconsistent sizes. Do not infer that repeated pages are duplicates, that identical copies are dispensable, or that any file should be withheld, deleted or excluded from production. Produce a review map only.

INPUTS
file-manifest.csv: file_id, source_batch, original_path, original_name, size_bytes, sha256, custodian_alias, family_id, received_at
scope.txt: approved repository, export coverage, hash method and unresolved page-level duplicates

OUTPUT
duplicate-map.csv: group_id, file_id, source_batch, sha256, size_bytes, original_path, custodian_alias, family_id, evidence_conflict, review_note. Preserve every file and report missing or inconsistent hashes.

Use only these synthetic or firm-approved inputs and preserve source row/page IDs. Keep missing or conflicting evidence unresolved. Return a draft for the named reviewer. Do not send messages, connect accounts, change records or permissions, move money, or make legal decisions. Keep source files unchanged. List missing prerequisites before continuing.
```

Proposed / not tested · editorial prompt

Published by Law Firm Agents. Proposed adaptation; not tested. Not legal advice.