drive-mind

One tool for the files you want to keep. dm turns a drive into a Merkle snapshot of CIDs and records where verified copies live, so the original bytes can eventually be deleted safely. Milestone M0 is built: the same CIDs Kubo produces, plus the sha256, md5, sha1, Dropbox and OneDrive hashes cloud providers publish, so a file in a cloud listing is recognized without downloading it. Nothing is copied, repaired or encrypted yet.

Why a catalog

What exists, where, and how it was verified lives in signed, append-only commit streams. SQLite is a disposable index that dm catalog rebuild regenerates and proves identical, so the catalog works with the drive unplugged. An unplugged drive is unavailable, never lost: absence changes nothing.

What it does today

• dm import --in-place streams each file once, in constant memory, into a dm-unixfs-1 CID: the IPIP-499 unixfs-v1-2025 layout, 1 MiB raw leaves, checked against Kubo up to 1.1 GiB files and through HAMT directory sharding.
• Identical files are one CID. Symlinks are stored, never followed. A file that changes mid-read is retried, then recorded as unstable with no CID rather than a wrong one.
• dm verify re-hashes replicas; a replica is verified only if its bytes still produce the recorded CID. dm cat checks every leaf before output and falls back to another copy when one is damaged.
• dm node imports drives on a remote machine over ssh, and dm node pull keeps a verified, read-only mirror of that node's catalog for local queries.
• Re-import skips files whose size, mtime and inode are unchanged: 13 minutes became 2.2 seconds on an 80 GiB ext4 drive.

What it learned

On two real USB drives, six parallel readers ran at half the speed of one on spinning disks, so the importer defaults to one worker there. Hashing is not the limit: a million-file catalog fold runs near 9,300 files per second, and a cold external drive is bound by the drive.