gwt-queue¶

gwt-ledger remembers what was already searched. Nothing remembered what comes next: each lane picked its own targets, the planner's priorities lived in a brief nobody else could read, and a lane that died mid-task took its claimed work to the grave. gwt-queue is one TSV file that holds the pending research tasks, hands them out to lanes one at a time, and reopens the ones a dead lane dropped.

Standard library only, Python 3.8 or newer. Run it as queue/gwt-queue (symlink it onto your PATH if you like) or python3 queue/gwt_queue.py.

The file¶

One row per task, tab-separated, with a header line:

The queue path comes from --queue, else $GWT_QUEUE, else ./cola.tsv in the current directory. Times are UTC; a value with no zone is read as UTC.

Keep the file untracked, and on one host. Every command rewrites it whole under a local flock, which does not hold across git clones or across machines.

The loop¶

A planner (a person, a judge, or gwt-leads) seeds the queue. Every lane then loops:

while row=$(gwt-queue claim --lane lane-2); do
    id=$(cut -f1 <<<"$row"); token=$(cut -f17 <<<"$row")
    # ... run the task, write the evidence file; on a long task, now and then:
    gwt-queue renew --id "$id" --token "$token"
    gwt-queue done --id "$id" --token "$token" --result hit --evidence hits/ana-peiro-fs.txt
done

• claim takes the highest-priority task that is open and due (at equal priority the older one, q2 before q10), marks it yours with a lease, and prints the row. Column 17 is the claim's token. Exit 1 means nothing is due: the lane is finished, or waits for the next wave.
• renew pushes the lease out again. A lane on a task longer than its lease must renew, or the task reopens under it.
• done works only on a task that is claimed under your token. A lane whose claim went stale and was handed to another lane holds a dead token: its done, fail and release are refused (exit 2) and change nothing. fail and release follow the same rule, so a finished task is never reopened. done closes the task with a result and the evidence file. Mirror the row into gwt-ledger so the negative (or the hit) is never searched twice: the queue tracks the work, the ledger tracks the search.
• fail is for an attempt that ran and could not finish: a site block mid-search, a missing login. The task reopens with a back-off of --backoff-minutes (default 60) per attempt so far. After --max-tries attempts (a top-level option, default 5) it becomes failed and waits for a person to read it.
• release is only for "nothing ran": the gate was paused, the lane is shutting down before it searched. It reopens the task and gives the try back. --retry-after MINUTES keeps it unclaimable for a while, so a lane that keeps finding the gate paused does not re-claim the same task at once. Anything that touched the source must fail instead.
• claim --kind KIND and claim --source SOURCE take only tasks of that kind or for that source, so a lane signed in to one site claims only that site's work.
• report is the open-leads list: what is due now, by priority (the first 20; --limit N for more), then how much waits and how much failed. Read it at the start of a round; the judge reads it too.
• list prints the rows, filtered with --status, --person, --lane and --result (the rows closed with that result, for example blocked or denied). A blocked or denied row stays in that list for good: add --dedupe re-seeds the task as a new row, so the old row is still listed after the new one found something. Check list --person KEY before redoing a search by hand.

Two lanes never receive the same task: every command runs under an exclusive lock on <queue>.lock, and the queue file is replaced atomically (temp file, fsync, rename), so a crash leaves the old queue or the new one, never a half-written file. Keep the .lock file next to the queue; do not delete it while lanes run. A lane that dies holding a claim loses it when the claim's own lease runs out: the next claim, report or stats reopens the task. The lease is stored on the row, so no caller can revoke a live claim early. The dead attempt keeps its try, so a task that kills every lane that takes it parks as failed after --max-tries, like one that keeps failing, and shows up in report and stats.

Seeding¶

add one task (--person, --label, --kind, --source, --query, --priority, --note, --next-at for a task not claimable before that instant, --dedupe-key), or many with --jsonl FILE (- for stdin): one JSON object per line with person and any of label, kind, source, query, priority, note, next_at and dedupe_key. A batch runs under one lock and one write, and a bad line stops it before anything is written.

--dedupe skips a task that is already queued, running or done: the same dedupe_key when either side has one, else the same person, kind, source and query. A done task counts only when its search ran (hit, none, partial or out-of-coverage, the results gwt-ledger counts as done); one closed as blocked, denied, restricted, paid, in-person or unproven does not stop a re-seed. A failed match is skipped too, unless --retry-failed is given. On an open match the new label, query, note and priority replace the old ones, so a re-seed carries the latest picture.

gwt-queue add --person "Ana Peiró Toledo" --label "Ana Peiró Toledo" \
    --kind sweep --priority 100 --note "direct line, round 31" --dedupe

--dedupe only knows this queue. A search logged in gwt-ledger from outside the queue is not seen, so check the ledger before a lane runs a task.

Keep the queue next to the ledger: buscado.tsv says what was searched, cola.tsv says what is still to do.