gwt-rawstore¶

A research repo that commits every scanned page grows to gigabytes of packs. Clones get slow, and a partial clone fetches blobs one at a time and can hang the machine. gwt-rawstore keeps the bytes in an S3-compatible bucket (Garage, MinIO, AWS) and keeps only a small text manifest in git.

Standard library only, Python 3.9 or newer. Requests are signed with AWS SigV4 and use path-style addressing (<endpoint>/<bucket>/<key>), which Garage and MinIO need.

What it stores¶

• Key: <prefix><sha[:2]>/<sha256>/<original name>. The same file uploaded twice is the same object, and the key alone proves what the content should hash to.
• Manifest: rawstore.tsv in the file's own folder, columns sha256, size, key, path (path relative to that folder). It is sorted and rewritten atomically, so it diffs cleanly. It does not touch any other manifest you keep in the folder.
• Integrity: the upload carries the sha256 as x-amz-content-sha256, so the server refuses a body that arrives damaged. The object also carries it as x-amz-meta-sha256. A file is recorded in the manifest, and removed locally with --remove, only after a HEAD confirms the object and its size.

Configuration¶

Environment first, then the file in $GWT_RAWSTORE_CONFIG (default ~/.config/gwt-rawstore/config, KEY=value lines). Keep that file mode 600 and out of git.

For Garage, a dedicated bucket and key:

garage bucket create my-raw
garage key create my-raw            # prints the key id and secret once
garage bucket allow --read --write my-raw --key <key id>

Commands¶

gwt-rawstore put <files or dirs> [--remove] [--all] [-j 4]

gwt-rawstore get <path|sha> [...] [-o TARGET]

gwt-rawstore verify [dirs] [--deep]

gwt-rawstore migrate <dir> [--remove | --evict] [--dry-run] [-j 4]

It refuses to run when tracked files are missing from the working tree (a sparse checkout), since it cannot hash what it cannot read. Run it on a full clone, for example on a server, then pull the commit on the laptop.

--dry-run shows what a run with the same flags would do, and does none of it. It hashes each binary and sends one HEAD for its key; it never uploads, never writes a rawstore.tsv or .gitignore, never runs git rm or git add, and never deletes a file. Each file gets a line, would put or have (the store already holds the object with that size), and says whether its folder's rawstore.tsv already has a row with the same sha256 and size. Then come the totals: files and bytes to upload, files and bytes already stored, and how many folders would get a manifest written. With --remove or --evict it adds the files and bytes that would be untracked, or untracked and deleted, counted by extension, so compressed data such as .json.gz shows up before it leaves git. It refuses a sparse checkout with the same error as the real run, and exits 1, naming each file, when it cannot read a file, a HEAD fails, or the store holds the object with the right size but another sha256, which the real run would refuse.

gwt-rawstore migrate raw --evict --dry-run

Images that live in the bucket¶

To keep a folder of source images (say raw/) out of git for good:

1. Download new ones with gwt-fetch ... --out raw/<folder> --rawstore: each file goes to the bucket and is deleted locally once confirmed, and gwt-fetch reads rawstore.tsv so it never downloads it again (gwt-fetch).
2. Move the ones already committed with gwt-rawstore migrate raw --evict on a full clone, then commit.
3. Pull that commit on the laptop. Git deletes the untracked images from its working tree and keeps the rawstore.tsv and manifest.tsv files, which gwt-fetch --rawstore reads to skip what is in the bucket. gwt-rawstore get <path|sha256> brings back one image when you need it.

After --remove or --evict, git ignores every file under dir with a migrated extension, including scans saved there later without --rawstore: they never show in git status and are in no bucket. Store them with gwt-rawstore put --remove <dir>; it is safe to repeat, since files already in the bucket come back as have. Migrate prints the same reminder.

gwt-rawstore check-staged [--max-kb N]

pre-commit:
  commands:
    rawstore:
      run: gwt-rawstore check-staged

Tests¶

python3 -m unittest rawstore/test_gwt_rawstore.py

The tests run against an in-process S3 stub that, like Garage, refuses a body that does not match its x-amz-content-sha256, plus the signed example from the AWS SigV4 documentation, so the signer is checked against a known answer.

Repo health: doctor¶

gwt-rawstore doctor [--repo PATH] [--json] [--max-packs 500] [--max-tmp-gb 1] [--max-binary-gb 1]

A read-only check of how a research repo uses the disk, for repos that many agents read and commit to. It needs no store configuration. Each finding says what to do; where that is a command, the fix: line is the command alone, paths quoted, ready to paste. Exit status 0 when nothing is wrong, 1 when something is, 2 when it cannot run (no such folder, not a git work tree, no git).

It also prints as information gc.auto and maintenance.auto, and, in a quiet partial clone, whether GIT_NO_LAZY_FETCH is set in the shell running doctor; neither is a finding. A relative cache variable is looked up from the current folder and from the top of the repo. On Linux the Time Machine check is skipped with a note.

It never fetches. It runs only git version, rev-parse, count-objects -v, config --get, config --get-regexp, ls-files and tmutil isexcluded, never a command that reads an object, and starts every git with GIT_NO_LAZY_FETCH=1 as well. A binary outside a sparse checkout is skipped, not read from the object store. Why these checks: hard-won lessons 7, 8 and 55 to 57.

python3 -m unittest rawstore/test_doctor.py

The tests build small repos in a temp folder: many tiny packs, a tracked image, a big tmp/ file, a .cache folder, and a blob:none partial clone over file://. One test records every process doctor starts and checks that no pack appears; its control step then reads a blob outside the sparse checkout and shows that a lazy fetch in that clone does add one.