organizer

A FOSS CLI tool that sorts files already recovered from old storage (primarily Windows drives) into a clean, categorized directory tree for client delivery. Copy-only, idempotent, and resumable.

Not a data-recovery tool. It does not undelete, carve, or read raw disks. It takes a directory of recovered files and organizes them.

Install

Requires Python 3.14+ and libmagic (for MIME identification).

# Debian/Ubuntu
sudo apt install libmagic1
# macOS
brew install libmagic

# clone and install
git clone <repo-url> data-recovery
cd data-recovery
uv sync                # or: uv sync --extra rich  for nicer console output
uv run organizer --help

rich is an optional extra for colored log output and a progress bar during the copy phase. Without it, logging falls back to plain text with periodic progress lines every 100 files.

Quick start

# 1. Generate a default rules config you can edit
uv run organizer --init rules.toml

# 2. Dry run: scan + classify, write manifest, copy nothing
uv run organizer --source /mnt/recovered --output /mnt/sorted \
    --config rules.toml --dry-run

# 3. Real run: scan + copy
uv run organizer --source /mnt/recovered --output /mnt/sorted \
    --config rules.toml

# 4. (Optional) export the manifest for auditing
uv run organizer --output /mnt/sorted --export-manifest report.csv --resume

CLI reference

organizer [--source PATH ...] [--output PATH] [--config PATH]
          [--dry-run] [--resume] [--workers N] [--verify]
          [--export-manifest PATH] [--init PATH]
          [--verbose] [--debug] [--version]

Flag	Description
--source PATH	Source directory to scan. Repeatable for multiple sources.
--output PATH	Destination root. Manifest lives at <output>/_organizer/.
--config PATH	TOML rules file. Use --init to generate one.
--init PATH	Write a default TOML config to PATH and exit.
--dry-run	Scan + classify only. Writes manifest, copies nothing.
--resume	Continue an existing job. Required when a manifest already exists.
--workers N	Copy worker threads (default: min(8, cpu_count)).
--verify	Re-hash destination files after copy and compare to source hash.
--export-manifest PATH	Export the manifest to .json or .csv and exit. Bypasses --resume guard.
--verbose	Per-file decision logging.
--debug	SQL and internal detail logging.
--version	Print version and exit.

Exit codes

• 0 success
• 1 configuration error
• 2 usage error (missing --output, bad source, missing manifest, etc.)

How it works

scan → identify (libmagic + extension) → classify (TOML rules)
     → manifest (SQLite) → copy (.part + atomic rename) → reports

1. Scan walks each --source iteratively (not os.walk), skipping symlinks, excluded dirs, and permission-denied directories. Stat failures are recorded in the manifest with error_stage='scan' rather than aborting the run.
2. Identify uses libmagic as the primary signal with an ~120-extension fallback table. Each result carries a confidence score.
3. Classify applies the TOML rules in array order; first match wins.
4. Manifest (<output>/_organizer/manifest.sqlite, SQLite WAL) records every file's source, category, confidence, destination, SHA-256, and copy state. Per-file transactions. Survives source loss.
5. Copy streams source → <dest>.part (computing SHA-256 during the copy), fsyncs, then atomically renames to the final name. Never overwrites; collisions get a .N suffix. Optional --verify re-hashes the destination after rename.

Data safety

• Copy only. Source files are never modified, moved, or deleted.
• Never overwrite. Existing destinations are left alone; new files get a .N suffix (e.g. foo.txt → foo.1.txt).
• Atomic copies. Each file is streamed to a private _organizer/tmp/<uuid>.part staging area and atomically renamed into its category directory only after a size check (and optional hash verify). A crash leaves orphaned .part files in the staging area that are swept on the next --resume. The staging area never holds recovered files, so sweeping it can never delete real data.
• Idempotent. Re-running the same source+output is a no-op for files already copied. Adding a new source into the same output is safe.
• Source-changed detection. If a source file's size or mtime differs from the manifest, it is re-copied under a new collision name — the old destination is preserved.

Config / rules

--init writes a default TOML config with sensible rules covering all 18 assignable categories. TOML rules are the only classification logic. With no --config, every file lands in unknown. This is deliberate: zero config = zero rules.

Schema

[scan]
exclude_dirs = [".git", "node_modules", ...]   # optional, has defaults

[[rules]]
category = "photos"           # required, must be a valid category
mime = ["image/*"]            # any-of within field; AND across fields

[[rules]]
path_regex = ["(^|/)Windows(/|$)"]   # regex, case-insensitive, POSIX path
filename = ["Thumbs.db", "~$*"]      # glob, case-insensitive
category = "junk"

Matchers (per rule, AND across fields, any-of within a field)

Field	Match	Notes
path_regex	regex re.search, case-insensitive	Against source-relative POSIX path. Use `(^
filename	glob, case-insensitive	Against the file's base name.
mime	glob, case-sensitive	Against the MIME type (already lowercase).
extension	glob, case-sensitive	Against the extension without a dot.

Rules are evaluated in array order; first match wins. Each rule must have a category and at least one matcher. Confidence by matcher tier: path_regex → 1.0, filename → 0.9, mime → inherits identification confidence, extension → 0.5.

Categories

photos, videos, audio, documents, spreadsheets, presentations, archives, source, executables, disk-images, virtual-machines, 3d, cad, fonts, databases, email, system, junk, unknown

unknown is the fallback when no rule matches; it is not assignable as a rule category.

Output layout

<output>/
├── _organizer/
│   └── manifest.sqlite        # SQLite WAL manifest
├── photos/
│   ├── img.jpg
│   └── img.1.jpg              # collision suffix
├── documents/
│   ├── notes.txt
│   └── report.pdf
├── archives/
│   └── backup.zip
└── unknown/
    └── mystery.dat

Files are flattened into their category directory (the source's relative subdirectory structure is not preserved). This keeps client delivery simple.

Manifest

The SQLite manifest at <output>/_organizer/manifest.sqlite is the source of truth. Two tables:

• sources(id, path UNIQUE, label, added_at) — each --source registered once; re-runs reuse the same source_id.
• files(id, source_id, relative_source, source_size, source_mtime, mime, extension, category, confidence, destination, dest_size, source_sha256, copied, duplicate, error, error_stage, scanned_at, copied_at) — one row per file. UNIQUE(source_id, relative_source).

Export with --export-manifest to JSON or CSV for auditing.

Resuming

• The first run against an empty --output starts fresh.
• Any subsequent run against an output with an existing manifest requires --resume (a guard against accidentally merging into a finished job).
• --dry-run writes the manifest but copies nothing; re-run without --dry-run (and with --resume) to perform the copy.
• Orphaned .part files from a crashed run are swept automatically on resume.

Verification

By default, copy integrity is checked by size, and a SHA-256 of the source is computed during the streaming copy (no extra read pass) and stored in the manifest. Pass --verify to additionally re-hash the destination after the rename and compare it to the stored source hash. Mismatches remove the destination so the next run re-copies cleanly.

exFAT / external delivery (out of scope)

organizer assumes a Linux native filesystem (ext4/btrfs) for the staging --output because it relies on POSIX atomic rename, fsync, WAL SQLite, and Unix permissions. If the final client delivery target is an exFAT/FAT32 volume, copy the organized tree there manually after the run completes:

cp -R /mnt/sorted/ /mnt/exfat_delivery/

exFAT lacks atomic rename guarantees, does not support sparse files the same way, and ignores Unix permissions — so running organizer directly against an exFAT --output is unsupported.

Limitations

organizer is conservative by design — it never deletes source data and never overwrites existing destination files — but a few edge cases are worth knowing about before you trust it with a real recovery job:

• Concurrent runs against the same --output are not safe. The collision lock is in-process only. Two organizer processes pointed at the same output directory can race on destination resolution and one's atomic rename can clobber the other's. Run one at a time per output tree.
• mark_copied failure after a successful copy can produce a duplicate on the next run. If the SQLite write fails after the file is already on disk (e.g. disk full, manifest DB locked beyond the retry budget), the manifest still says copied=0. The next --resume re-copies the file under a .N collision name. The original copy is not lost — you just end up with two. The failure is logged loudly.
• A source file that changes between runs leaves the old destination orphaned. When the same source path is re-scanned with a different size or mtime, organizer resets copied=0, writes a new destination under a .N collision name, and never touches the old destination file. You'll need to clean those up manually if you want a tidy tree. This is the conservative choice: organizer never deletes anything.
• No --move, --dedupe, --hardlink, or --quiet in this version. These are in the spec but intentionally deferred. Copy is the only operation; delete the source yourself once you've verified the output.
• Scanner sorts each directory's entries. This is deliberate (deterministic output) but costs a sort per directory. For extremely large directories this is a minor overhead.

Development

uv sync --extra rich
uv run pytest              # 124 tests
uv run organizer --help

Module layout under organizer/:

Module	Responsibility
models.py	FileRecord dataclass
manifest.py	SQLite WAL manifest
scanner.py	Directory walk
identifier.py	libmagic + extension MIME identification
categories.py	Category constants
rules.py	Rule + classify()
config.py	TOML load + --init default
copier.py	Atomic copy + verification
report.py	JSON / CSV export
organizer.py	CLI + orchestration