gwt-read sources¶

gwt-read reads several record sites through one shape: the same files, the same gate and cache, the same STOP rules. Each site is a source, declared once in a registry, and gwt-read builds the source's subcommands, its gate, its cache rules and its canary controls from that declaration.

gwt read --help lists every source the installed version knows.

What a source declares¶

A source is one module, read/gwt_<id>.py. It sets SOURCE, the slug the ledger and each candidate carry, and SPEC:

and the functions add_search_args(sp) and search(a, R), plus add_record_args(sp) and record(a, R) when the site has a record page. search asks for every page through R.get(url, headers, refresh=a.refresh, page=gwt_reader.page_check(SPEC)), writes the envelope and candidates with gwt_reader.write_search, prints the lines with gwt_reader.print_paths, and returns 0 with rows or 2 with none.

A source that reads a JSON API, not HTML pages, works the same way with one change: it passes expect=<a function that checks the answer's shape> to R.get instead of page=, as familysearch does with is_search. An answer of another shape is a STOP. It leaves result_markers and negative_markers empty, because with no marker every page classifies unknown, and tells "none" by the answer's own count field (zero rows in a well-shaped answer).

An open API calls gwt_reader.get_api(R, url, headers, expect=...) instead of R.get. It is the same call, except that a 5xx or no answer at all (a timeout, a refused connection) means the service is down: a wall (exit 3, the gate hears a hit), never an error a lane logs and moves past. A 4xx stays an error. openarch, wikitree, archiveorg and matchid read this way.

The page classifier¶

gwt_reader.classify(page, SPEC) gives one of three answers:

So a page skeleton with no rows, a block page, or wording the site changed is never logged as "no results". Pick a result marker that only appears on a row (a row's class, a record link), never the empty container the skeleton also has.

Canaries¶

A canary is a public search with a known minimum: a very common name with a count the site has given for years, with room to spare. Never use a family's own surname or a person from someone's tree; the canary is in public code. gwt read <id> control runs them without the cache, and search --controls before|after|both runs them around one search (canary controls). An offline source has none.

Add a source¶

1. Copy read/_template.py to read/gwt_<id>.py and fill in SPEC and the functions. The template reads an invented site, so it runs as written.
2. Add the module name to SOURCE_MODULES in read/gwt_reader.py, and its line to [tool.hatch.build.targets.wheel.force-include] in pyproject.toml. The wheel check imports every module the registry names.
3. Write read/test_gwt_<id>.py with invented fixture pages under read/fixtures/<id>/: one with rows, one with the site's "nothing found" words, one skeleton. The Template class in read/test_registry.py is the pattern.
4. Put the source's mutant rows in read/mutants.d/<id>.txt (same format as mutants.txt), so sources added at the same time never edit one file. At least: the negative marker removed, the result marker removed, the pace made faster.
5. Write docs/sources/<id>.md: what the site holds, its terms and robots.txt, the pace and why, the canary and its minimum, what a candidate is.

Before the first live run, read the site's robots.txt and terms, send the project User-Agent, and keep to a person's pace. A source that needs a login, solves a captcha or works around a block does not belong here.