Efficient candidate filtering
Files are grouped by exact byte size first, so unique-size files are never hashed.
Day 017 / Python / Local Automation
Duplicate Finder
/ Overview
A read-only Python CLI that confirms byte-identical files with SHA-256 and reports potential storage savings without deleting anything.
Repeated downloads and copied folders can waste storage, but matching filenames or file sizes are not reliable proof that two files are identical. Automatic deletion would also make a small utility unnecessarily risky.
/ Provided servicesClear thinking.
Clear thinking.
Practical delivery.
Content verification
Streaming SHA-256 confirms that every reported group contains byte-identical files.
Safe reporting
Human-readable and JSON reports use relative paths, skip symlinks and never delete, move or rename files.
/ Solution
Duplicate Finder recursively scans a chosen directory, skips symbolic links, groups candidates by byte size, then streams SHA-256 only for groups that may contain matches. It reports confirmed duplicate groups, relative paths and potentially reclaimable bytes in readable text or JSON while leaving every file untouched.
/ Impact
Byte-identical matches confirmed
Unnecessary hashing reduced
Potential storage savings calculated
No automatic file changes
Python standard library
Pathlib, os.walk, hashlib, argparse and dataclasses keep the utility portable and dependency-free.
SHA-256
Chunked hashing compares file contents without loading whole files into memory.
Unittest
Nine deterministic tests cover grouping, nested folders, storage totals, JSON, safety messaging and invalid input.