Where This Came From
At Bank of America I was application manager for a metadata portal used by the commercial side of the bank. It did not hold customer data. It held data about the data. It tracked what types of information each application stored, and where.
That sounds abstract until you watch what it does. Once the bank could see its own data described in one place, two things fell out that nobody could get at before. Where a given kind of data actually lived, across hundreds of systems. And how much of it was being captured more than once. The same customer attribute collected by four applications, none of them aware of the other three.
The education was not the portal itself. It was the idea underneath it. You do not have to touch the data to get control of it. You describe it, and the description becomes the thing you search and reason about. The systems stay where they are.
This project grew from those seeds, pointed at my own mess instead of the bank's.
What I Built
Fifteen years of documents and photos spread across a Mac, with the ordinary version of the same problems. I could not find things. I was afraid to delete anything. I had no idea how many copies of anything I had.
The same answer applies at this scale. Build the metadata layer first, in three steps, each one earning the next.
1. Find what I misplaced. A scanner walks a read-only replica of my files and populates a SQLite database. Enrichment passes describe what is inside them. Search runs keyword, semantic, or the two fused. This is the step that answers "I know I have it somewhere." It is also the step that surfaces duplicates, the same way the bank's portal did. Wire transfers and wedding photos instead of customer attributes.
2. Label it in categories I did not think of first. A local model reads each document and assigns labels from a taxonomy I define. Document type, lifecycle, sensitivity. Nothing leaves the house. The labels become filters in the same search engine, so a category I invent in month three applies to everything scanned in month one. The taxonomy is a plain text file. Changing it is an edit and a re-run, not a migration.
3. See those labels in the file system itself. The Linux box exports a manifest. A job on the Mac reads it every morning and rebuilds a folder tree of symlinks. by-document-type, by-lifecycle, by-sensitivity. The categories become folders I browse in Finder. Not one original file has moved.
Left, the search page across 3,045 indexed files. Right, the same database rendered as folders in Finder. Every folder on the right is built from the labels on the left, and every entry in it is a symlink.
How It Fits Together
The Decision I Would Defend
The database is the source of truth. Every folder tree is a generated, disposable view.
That one rule makes the hard questions easy. Should this file go in Taxes or Insurance? Both. It is two symlinks. What if I change my mind about the categories? Delete the tree and regenerate it. What if a label is wrong? Fix one row and regenerate.
Every one of those is a crisis in a system that organizes by moving files. In a system that organizes by describing them, it is a non-event. It also means the worst thing that can go wrong here is a bad view, not a lost file.
Fifteen years of my own files went through this one. Some of that reached test fixtures and commit messages, so the repo stays closed until I have scrubbed the history. It goes public after that. Happy to walk through the code before then.
The guards went in before any features, because the worst failure available here is silent and unattended. Roots are an allowlist. Every path is re-checked with symlinks resolved before descending, so one stray symlink cannot turn a scoped scan into a full-disk scan. A file count drop over 20 percent aborts before any write. Dry run is the default.
A resumable enrichment framework and the passes that fill it. SHA-256 hashing, content typing by magic bytes, EXIF, perceptual image hashing, text extraction into a full-text index, and text embeddings. Each pass only touches what it has not already done, so a week-long backfill survives being interrupted.
Keyword search, semantic search, and the two fused on rank rather than score, because bm25 and cosine are not on the same scale. A review page with filters for document type, lifecycle, sensitivity, root, file type and year. This is the point where the project stopped being an inventory.
The box exports a manifest. A job on the Mac rebuilds the symlink tree every morning and keeps the last three generations, so a bad build is one symlink swap away from being undone. A stale manifest or a sudden count drop refuses the build rather than replacing a good tree with a broken one.
Detect, embed, cluster, review. The policy matters more than the model here. A person only appears under by-person once I have confirmed them. Everything the clustering merely guessed goes to a separate faces-not-sure folder, so a wrong guess is never presented as a fact.
Classification runs on a small model that fits entirely in the GPU. A larger model is the escalation tier for files the small one abstains on, which is how most production AI systems actually work. Low-confidence labels land in a review queue instead of the tree.
Deletion is the last thing I will build and the one I trust least, so nothing is ever really deleted. The metadata row outlives the bytes and records where they went. One search box then covers files on disk, files in cold storage, and files already gone.
