# The autonomous research loop The [pipeline]() describes who decides what. This page wires the command-line tools into the loop an agent runs on its own: find the gaps, queue them, search without repeating or hammering a site, verify, and hand the findings to the judge. Every step below is a tool in this repo; nothing here writes to the tree. ``` flowchart LR D[gwt-check dump] --> LD[gwt-leads] LD --> Q[gwt-queue] G[gwt-gate wait] --> L[Lane: claim] Q --> L L --> LG{gwt-ledger check --exact} LG -- same search done --> QD LG -- new --> F[gwt-fscache / gwt-fetch] F --> E[Evidence file] E --> LA[gwt-ledger add] LA --> QD[gwt-queue done or fail] QD --> Q E --> V[Verifier, then judge] V --> A[gwt-apply, gwt-check] A --> D ``` ## One round 1. **Set the shared paths.** Every lane and the seeder use one queue file, one ledger directory and one cache directory. Each lane has its own name and works one site, with that site's own gate directory, so a block on one site does not pause the lanes on another: ``` # shared by everyone export GWT_QUEUE=work/cola.tsv GWT_LEDGER=work/buscado.tsv GWT_LEDGER_DIR=work/ledger \ GWT_FSCACHE_DIR=work/cache # per lane: its name, its site, and that site's gate (one per rate-limited site or account) export GWT_AGENT=lane-2 SRC=familysearch GWT_GATE_DIR=work/gate/familysearch mkdir -p work ev ``` Keep `work/` untracked and on one machine: the queue is rewritten whole under a local `flock`. 2. **Find the gaps and seed them.** `gwt-check dump` pages the whole tree, so do not run it while that tree's writer applies a change list. Write the seeds to a file and chain with `&&`, so a bad dump (exit 2) stops the round instead of seeding nothing: ``` gwt-check dump work/tree.json.gz && gwt-leads --root I0001 --source "$SRC" --format queue --per-gap --limit 200 work/tree.json.gz > work/seeds.jsonl && gwt-queue add --jsonl work/seeds.jsonl --dedupe ``` The queue now holds the lines that stop, the missing vitals and the uncited facts, direct ancestors first, one task per gap. People who may be living and private records are held back, their ids on stderr. `--dedupe` skips a task already queued, running or done. Seed with `--per-gap`: its dedupe key carries the gap code, so a gap that appears in a later round gets a task of its own. In the default grouped mode the key is only person, kind and source, and a new gap of a person whose task is already done folds into that done task and is never searched. See [gwt-leads]() and [gwt-queue](). 3. **Each lane loops** until nothing is due: ``` while :; do gwt-gate wait "$GWT_AGENT"; rc=$? if [ "$rc" -eq 3 ]; then sleep 300; continue; fi # the site is paused: wait without holding a task [ "$rc" -eq 0 ] || { echo "gwt-gate exit $rc" >&2; break; } row=$(gwt-queue claim --lane "$GWT_AGENT" --source "$SRC") || break # 1: nothing is due id=$(cut -f1 <<<"$row"); person=$(cut -f2 <<<"$row"); kind=$(cut -f4 <<<"$row") source=$(cut -f5 <<<"$row"); query=$(cut -f6 <<<"$row"); token=$(cut -f17 <<<"$row") gap=$(cut -f19 <<<"$row" | cut -d: -f4-) # the gap code of a --per-gap task lsrc=${source:-any}; [ "$kind" = search ] || lsrc="$lsrc-$kind" # a verify is not a search lquery="$query $gap" # one ledger row per gap searched # reuse a finished search only on an exact match: same source, identical query, newest row first rc=1 [ -n "$source" ] && { prior=$(gwt-ledger check --exact --person "$person" --source="$lsrc" --query "$lquery"); rc=$?; } if [ "$rc" -gt 1 ]; then gwt-queue release --id "$id" --token "$token"; echo "gwt-ledger exit $rc" >&2; break fi if [ "$rc" -eq 0 ]; then # score, person, label, source, query, date, RESULT, EVIDENCE, agent, NEXT_DATE, control # the first row that counts as done (hit, none, partial, out-of-coverage) and is not due again top=$(awk -F'\t' -v today="$(date +%F)" '$7 ~ /^(hit|none|partial|out-of-coverage)$/ && ($10 == "" || $10 > today) { print; exit }' <<<"$prior"); result=$(cut -f7 <<<"$top") if [ -n "$top" ]; then gwt-queue done --id "$id" --token "$token" --result "$result" \ --evidence "$(cut -f8 <<<"$top")" --note "reused the ledger row of $(cut -f6 <<<"$top")" continue fi fi # ... search through gwt-fscache or gwt-fetch, write ev/$id.txt, set $result to hit, none or partial. # Before every request that is not a cache hit: gwt-gate wait "$GWT_AGENT". Exit 3 there means the site # paused mid-task: `gwt-queue fail` if anything was fetched, `release` if nothing was, then `continue`. # On a long task, call `gwt-queue renew --id "$id" --token "$token"` before the lease runs out. # On a block, see below, then `continue`. # $ctl: this batch's saved `gwt read control` output (runs/-control.txt), the proof a none # needs; without it the none is logged as unproven in the ledger and the queue alike, and searched again. [ "$result" = none ] && [ -z "$ctl" ] && result=unproven gwt-ledger add --person "$person" --source="$lsrc" --query "$lquery" \ --result "$result" --evidence "ev/$id.txt" ${ctl:+--control-ok "$ctl"} || { gwt-queue fail --id "$id" --token "$token" --note "gwt-ledger add failed"; break; } gwt-queue done --id "$id" --token "$token" --result "$result" --evidence "ev/$id.txt" done ``` The gate is asked twice. Before the claim it is a probe: a lane that finds the site paused sleeps without holding a task, so a long pause costs no tries. Then the search step asks it again before every request that goes to the network. `gwt-fscache` and `gwt-fetch` never call the gate themselves, and one task can make many requests. A cache hit spends no request and needs no wait. Exit 3 is the only pause; any other non-zero exit from `gwt-gate wait` is an error and stops the lane, and so does an error from `gwt-ledger` (exit 2). Exit 1 from `check` is a plain miss. When `gwt-ledger add` fails after a search, the task is failed before the lane stops, so its result is searched again rather than closed with no ledger row. The ledger has no task-kind column, so the loop gives each kind its own ledger source: `familysearch` for a search, `familysearch-verify`, `familysearch-sweep` and `familysearch-followup` for the others, and `any` in place of an empty source. A task with no source logs its rows under `any` but never reuses one: the reuse step runs only when the task names its source. Without that, a `verify` task would share the row of a `search` task on the same person, source and query, and close with a result nobody checked. The suffixes are reserved in [gwt-ledger](). Pass the value as `--source="$lsrc"`, with the `=`, so it can never be read as a flag. The ledger query is the task's query plus its gap code, so each gap of a person has its own row: a `none` for "no parents recorded" does not close the "no death" task. A ledger row stands in for a search only when it is an exact match (`check --exact`: same source, identical normalized query), its result [counts as done]() (`hit`, `none`, `partial` or `out-of-coverage`; never `blocked`, `denied` or `unproven`), and its `next_date`, if any, is still in the future. The loop reuses the first such row, not simply the first row printed: a newer `blocked` or `unproven` row can sit above it. A `none` is logged with the control run of the batch ([gwt-read canary controls]()), or as `unproven`. `check` lists equal rows newest first, so a later `hit` wins over an older `none`. A fuzzy match is a hint for a person, never a result for a script: the search runs. A reused row's result and evidence go into the task unchanged. Nothing here turns a block into an absence. **On a block** (a 429, a captcha, a block page): `gwt-gate hit "$GWT_AGENT" ""` pauses every lane on that site; log the try with `gwt-ledger add ... --result blocked`, a row the loop above skips when it looks for one to reuse; then `gwt-queue fail`, so the task backs off and returns later. For an IP or account block that lasts hours, `gwt-gate block ""` stops every lane on that gate for that long. On a 403 that refuses one record, `gwt-gate denied "$GWT_AGENT" ""` logs it without pausing anyone. On a 401, `gwt-gate dead` pauses every lane without slowing the pace and exits 4: the session is gone, so stop the lane. A "no result" from a site that was blocking is a `fail`, not a `none`. `gwt-queue release` is only for an attempt where nothing ran. 4. **Verify and judge.** The evidence files go to the verifier and then the judge exactly as in the [pipeline](). A `hit` creates a `followup` task (`gwt-queue add --kind followup`) for whatever it opens next: the parents named in an act, the parish of a marriage. 5. **Apply and check.** The judge's rulings become a change list; `gwt-apply` writes it and `gwt-check` verifies the result. The next `gwt-check dump` is the next round's input: a gap that was filled no longer shows up in `gwt-leads`. A gap whose search came back `none` still shows up, and `--dedupe` keeps it from being queued again, while a gap that is new this round gets a task. ## What each tool prevents | Failure | Tool | | --- | --- | | Two lanes search the same thing | `gwt-queue` hands each task to one lane; `gwt-ledger` remembers finished searches | | A lane dies holding work | `gwt-queue` reopens a claim when its lease runs out; the dead lane's token no longer works | | A blocked site is hammered | `gwt-gate` gives every lane on a site one pace and pauses all of them on a hit | | A finished search is queued again | `gwt-queue add --dedupe` also matches done tasks; `gwt-ledger check --exact` stands in for a search only on an identical one | | A repeated request spends rate budget | `gwt-fscache` answers it from disk and never stores a block | | Effort goes to the wrong branch | `gwt-leads` ranks the root's direct ancestors first | | A transient failure loses a lead | `gwt-queue fail` backs the task off and parks it after `--max-tries` attempts | ## Limits - The loop finds and ranks leads. It does not decide that a record is the right person. That stays with the verifier and the judge. - `gwt-leads` only sees what the tree already holds. A branch with no people in it has no gap to report: seed those tasks by hand with `gwt-queue add`. - Keep the queue, ledger, gate and cache directories on one machine, or on a filesystem with working `flock`. Never commit the queue file. - `--dedupe` only knows the queue. A search logged in the ledger from outside the loop is caught by the `check --exact` step, not at seeding.