gwt-read¶

export GWT_FS_TOKEN_FILE=~/.fs-session        # your own session token, mode 600 (see below)
gwt read familysearch search --given Ana --surname Zorvalo --by 1851 --place Villazul --out runs/
gwt rank runs/<key>.candidates.json --given Ana --surname Zorvalo --facts "by=1851 bp=Villazul"
gwt read familysearch record 1:1:QZZ1-AAA --out runs/
gwt read pares search --given Ana --surname Zorvalo --out runs/     # public, no account
gwt read pares record 9000001 --out runs/
gwt read genae search --given Ana --surname Zorvalo --out runs/    # public, no account
gwt read boe search --name "Ana Zorvalo Quemada" --out runs/      # public, no account
gwt read insee search --files deces-1985.txt deces-1986.txt --surname Ortega --surname Sanz --out runs/   # offline

gwt-read runs a record search, or reads one record, on a record site, and writes the answer in the shape every record-site reader in the toolkit shares. It reads four sites: FamilySearch historical records, PARES, the catalogue of the Spanish state archives, Genae, a volunteer index of parish records that gwt-read only searches: its rows are leads, not records, and the BOE and Gaceta de Madrid text search, whose hits are mentions of a name in an official act. It also reads one source offline: INSEE's French deaths files, downloaded once and read with no request at all. It is slow on purpose and cached, because FamilySearch suspends accounts for volume: one of ours was suspended in 2026 after about 15 parallel lanes called the API through one night (hard-won lesson 1).

It never logs in and never asks for a password. PARES, Genae, the BOE and the INSEE files need no account. For FamilySearch you sign in in your own browser, and the tool uses that session's token. Standard library only.

The reader shape¶

gwt-mh (MyHeritage) and gwt-read (FamilySearch, PARES, Genae, BOE, INSEE) write the same files, from read/gwt_reader.py:

Each command prints the lines a lane needs for its ledger row, one per line, tab-separated:

source  familysearch
evidence    runs/fs-ana-zorvalo-1851-1a2b3c4d5e.json
candidates  runs/fs-ana-zorvalo-1851-1a2b3c4d5e.candidates.json
results 2 total, 2 read

A reader never writes the ledger: the lane runs gwt-ledger check before the search and gwt-ledger add after, with that source and evidence.

Every request goes the same way (the INSEE source makes none). A fresh answer in gwt-fscache is served from disk with no network call and no wait. Only on a miss does the request wait for its slot in gwt-gate, then leave. An answer that is not a result stops the run: nothing is written, nothing is cached, and the gate hears about it so every lane on the account pauses.

Canary controls¶

A search that finds nothing proves nothing unless the site could have answered. A site under a soft block can answer HTTP 200 with an empty page skeleton, a site can change its "no results" wording, and a filter can stop working. Each source therefore declares canary searches: public, neutral queries that any working copy of the site answers with at least a known number of rows.

gwt read pares control                                         # run the canaries now
gwt read boe search --name "Ana Zorvalo Quemada" --controls both --out runs/   # canaries before and after

control runs each canary without the cache, through the gate like any request, and prints one line each. It exits 0 when every canary reads at least its minimum, and 3 with a void line otherwise:

control control --given Cristóbal --surname Colón   25 read, want at least 10   ok

A PARES search that gets an empty page checks it once against the canary before it counts as a wall, at the gate's normal spacing. When the canary answers, the problem is that query, not the site: the search exits 7 (unreadable), the unreadable query is neither saved nor cached (the successful form and canary answers may be) and the gate only writes an UNREADABLE line to its log, so one odd query no longer pauses every lane or voids its batch. Record the row as unproven in gwt-ledger (open, to retry or narrow), never none. When the canary is empty too, reads short of its minimum, cannot be reached or meets a paused gate, the wall stands as before, and an empty page on the canary search itself is always a wall; a canary that answered short may be cached, and the probe always reads it live.

--controls before|after|both on search runs the canaries around that search, and reads the search itself without the cache, since the canaries only prove the site works now. A short canary before the search means no search. A short canary after it, or one stopped by a block or a captcha, makes the run void (exit 3): the envelope and candidates files the search wrote are removed (one removed line each), and so are the answers it cached, so nothing on disk reads as "none" and the next run asks the site again. To bracket a batch, run control before the first search and after the last one: a canary costs one search, which is two requests on PARES (the search form, then the search), so do not add it to every search.

A source added through the registry also names the marks of its results page: a row of results (hits), or the site's own words for "nothing found" (none). A page with neither is unknown, and unknown is a STOP, exit 3, never zero results. PARES, Genae and the BOE apply the same rule through their own page checks. gwt-read sources explains how a new source declares all this.

FamilySearch¶

The session token¶

FamilySearch has no API key for personal projects (hard-won lesson 4). Sign in at familysearch.org in a normal browser, then copy the value of the fssessionid cookie (in Chrome: Developer tools, Application, Cookies). Put it in a file only you can read and point GWT_FS_TOKEN_FILE at it, or put it in GWT_FS_TOKEN. It is sent as a Bearer token. It never reaches a file the tool writes, a cache entry or a log line, and the tests check that.

When the session expires the tool exits 4. Sign in again in the browser and copy the new token. Never script the login: FamilySearch's login blocks automation browsers (hard-won lesson 5).

Commands¶

The search uses the JSON endpoint the site's own search page calls (/service/search/hr/v2/personas), and the record uses the ark URL asking for GEDCOM X JSON. Both answers are turned into candidates by gwt-rank's own GEDCOM X reader, so gwt-rank ranks them like any other search. Each candidate keeps its role and principal flag, and place_orig lists the event place as the index typed it, when the record has one: the EVENT_PLACE..._ORIG fields (which often name the parish the standard place drops), EVENT_PARISH_ORIG and CP_MAR_PLACE_ORIG, and the typed place some collections keep only inside the JSON text of EXT_DATA, EXT_DATA_ORIG (EVENT_PLACE_ALT_ORIG) or FS_UNSUPPORTED (EVENT_PLACE_SUP). Searches stay in the cache for 7 days and records forever (FAMILYSEARCH_RULES in gwt-fscache).

The default pace, and why¶

One account starts at one request every 30 seconds (120 an hour). After an hour without a hit the gate shortens the gap by a fifth, never below one every 20 seconds, and a hit triples it, up to one every 120 seconds. The gate's own defaults (15 s, floor 8 s) are for a machine-wide pace shared by many tools; this one is for one account and stays well under the volume that got an IP blocked in 2026: 461 requests in 42 minutes, one every 4.8 to 6 seconds (hard-won lesson 2). A repeated search costs nothing, because the cache answers it before the gate is asked.

These numbers are fixed in gwt-read (FS_PACE); gwt gate --dir ~/.cache/gwt-gate-fs-default status shows where an account's pace stands now. After a 403, the gate stays blocked for 24 hours; gwt gate --dir ... clear <reason> ends the block early, which you should only do once you know why it happened.

A 403 for one record: restricted or blocked¶

A 403 is a block by default, and it parks the account's gate for a day. Some records are refused to every account of an access level, though, and treating that 403 as a block paused every lane for nothing. So a 403 on record (an indexed record, a 1:1: ark) is checked before the gate blocks:

1. When the answer says Access to artifact denied, or names the record's permissions (ThemisPrm...) and none of them is ThemisPrmAnyone, the record is restricted. No other request is made. This reads the 403 body, so it is best effort: the wording is FamilySearch's to change.
2. Otherwise, with --control-ark set, that control record is read again, without the cache, through the gate. If it answers, the account and the IP are fine, so the 403 was about this record: restricted. If the control is refused too, or its read fails in any other way (a 5xx, an answer of the wrong shape, a 429, a paused gate), the first 403 stays a block, and the gate is blocked as before. This is the dependable check: set it.
3. Without --control-ark, a 403 that says neither is a block, as before.

--control-ark must be a 1:1: record ark other than the one you read; anything else exits 64. A restricted record is logged in the gate log as a denial (gwt gate denied), which pauses nothing. gwt-read reads no images, so a 403 on a film image is outside this check.

A restricted record exits 5, writes an envelope with access: restricted, the permissions it named and an empty candidates file, and prints results restricted .... Log it as restricted, never as "none": someone with the right access can still read it. A 403 on a search is always a block. Pick a control record your account has read before, and never one that is restricted itself.

Account safety rules¶

• One pace per account. Every lane that uses an account points at the same --account (or --gate-dir), so the account sees one slow client, not several polite ones.
• Never run parallel lanes on one account. Lanes on the same account share its gate and wait their turn. Do not add accounts to go faster, and do not move to another account or IP to dodge a block.
• Stop on 429 and 403. The tool exits 3 and the gate pauses or blocks every lane on the account. Do other work. A search made after a block time that says "no results" is not a negative (hard-won lesson 3).
• Keep a daily budget per lane. We use about 80 requests a day.
• Cache first. Share one cache directory between lanes, so a record any lane has read is never asked for again. The cache does not record which account fetched an answer: a record read with an affiliate library's access is replayed to a free account that shares the cache. Give an account its own --cache-dir if that matters.

What each kind of access sees¶

FamilySearch decides who sees an index or an image collection by collection, following the record custodian's terms. Clicking the camera icon on a record says where its image can be seen. Rows marked unconfirmed are what we have seen or read in community answers, not on a FamilySearch help page.

Some images can only be seen at the custodian or on a partner site, not at a center or a library. gwt-read uses the access of the account whose token it holds, nothing more.

Not yet¶

gwt-read does not download images or full-text transcriptions. gwt-fetch downloads archive images; keeping FamilySearch originals in rawstore with a manifest is a follow-up.

PARES¶

PARES (Portal de Archivos Españoles, https://pares.cultura.gob.es) is the catalogue of the Spanish state archives: the Archivo Histórico Nacional, Simancas, the Archivo General de Indias, the Corona de Aragón and others. A search finds archival descriptions (a lawsuit, a file of proofs of nobility, a licence to sail to the Indies), not indexed people. It is public: no account, no token.

Commands¶

What a candidate is¶

A PARES candidate is one description. Its title is the name gwt-rank matches the person's names against; a long title the results page cuts is read whole from the link. Each also carries archive, dates (of the file, as written), reference (the signatura) and url, and a record adds fields: every field on the description page, labelled as the site words them. A description is never the record's principal and carries no birth or death, because the dates of a file are not a person's: gwt-rank orders PARES results by how well the names match and never calls one strong. Open the description and read it before any fact goes into the tree.

Requests, pace and cache¶

The site answers a search only inside a session. Without one it answered its own error page in our test, so a search that is not in the cache first fetches the search form for the session cookie, then the results: two requests, each waiting for its slot in the gate. The cookie lives in memory for that run only and never reaches the cache or a file. A record needs no session and costs one request. Requests ask for Spanish pages (Accept-Language: es).

The gate starts at one request every 15 seconds, never faster than one every 10, and a hit triples the gap up to one every 120 (PACE in read/gwt_pares.py). Searches stay in the cache for 7 days and descriptions for 90 (PARES_RULES in gwt-fscache). A page that is not the one asked for (no results block on a search page, no title on a description page) is a STOP like a block: nothing written, nothing cached, every lane pauses.

Terms we rely on¶

• robots.txt (read 2026-10-25) has one group, User-agent: *, which disallows only /*docroot/ and some language paths under /man/. The search (/ParesBusquedas20/catalogo/find) and the descriptions are not disallowed.
• The legal notice (https://pares.cultura.gob.es/comunes/aviso-legal.html) licenses reuse of the portal's content under Real Decreto 1495/2011, including copying and "extracción, reordenación y combinación de la información", on condition that the meaning is not distorted, the source is cited ("Origen de los datos: Ministerio de Cultura") and no endorsement is implied. It has no clause against automated access.
• Some images carry their own restrictions. gwt-read reads descriptions only, never images.

What each kind of access sees¶

PARES has no paid tier.

Not yet¶

No advanced search (dates, archive, fonds, reference number) and no second page of results: a search reads the first page the site gives (25 rows). No image download; gwt-fetch downloads archive images.

Genae¶

Genae (https://genae.org) is a free volunteer index of parish baptisms, marriages and burials of the Region of Murcia and nearby villages of Albacete, Alicante, Almería and Granada. It is public: no account, no token. genae search asks the site's own search for the four blocks it shows (baptisms, grooms, brides, burials) and merges them into one candidates file for gwt-rank.

gwt read genae search --given Ana --surname Zorvalo --province mu --out runs/

Each row becomes a candidate. A baptism gives the child with the parents' first names. A marriage gives two candidates, one per spouse, with the full name only (the site does not split it) and the other spouse as a relative. A burial gives a death event. A marriage both spouses match appears in the groom and the bride blocks; it is written once. Rows the site marks as phonetic suggestions carry phonetic: true. gwt-rank reads neither the marriage event nor the phonetic mark, so a groom or bride ranks on names and spouse only, and the lane reads phonetic itself. Only the first page of each block is read (15 rows); a block that fills it prints as 15+ (cut: narrow the search). The gate keeps one request every 20 seconds at best (PACE in read/gwt_genae.py), so one search costs about a minute. Searches stay in the cache for 7 days (GENAE_RULES); when all four blocks come from the cache, the results line ends in (cache) and the evidence has from_cache: true.

A page that is not the block asked for (an empty frame, the search form, a captcha) is a STOP: nothing written, nothing cached, every lane pauses. A block the site answers with its notice that no books are indexed for those years in that place is a real answer with no rows.

Genae publishes no terms for scripted use, and its robots.txt disallows /buscar/ for crawlers. The reader keeps to a person's pace: one search of four requests a minute at most, each answer cached for a week, and it never pages. For the same reason Genae has no canary: gwt read genae control and --controls do not exist, so a Genae zero carries no proof the site could have answered. Genae rows are an index made by volunteers: a hit is a lead, and the act or the image behind it is the evidence.

BOE and Gaceta de Madrid¶

The BOE's open-data API (https://boe.es/datosabiertos/api/api.php) answers by date and by law, never by a person's name, and its daily summaries start in 1961. The site's own search (https://www.boe.es/buscar/gazeta.php) takes a phrase in the text of the acts of the Gaceta de Madrid and the BOE. The collection starts in 1661, but the site says its text search works only from 1836 to 1959, and gives approximate results. It is public: no account, no token. boe search asks it for one phrase and reads the first page.

gwt read boe search --name "Ana Zorvalo Quemada" --born 1880 --out runs/
gwt read boe search --name "Ana Zorvalo Quemada" --born 1880 --parents "Juan y Antonia" --pdf-context 2 --out runs/

The phrase is always sent in quotes, so the site reads it as one name; it ignores accents. Each result becomes a candidate of role mention, never a principal, with the act's title in the note. A mention in an official act (a pardon, a pension, a call for places, an appointment) says a name was printed there. It does not say which person it is, and a common name hits many namesakes, so gwt-rank never ranks a mention above weak. A row whose title holds the whole name is marked [name in title]; the others matched in the act's text only.

Every act gets a verdict (also at the start of its note), and the search prints the three counts:

• excluded: published before the birth or over 100 years after it, or the act writes other parents at every place the name stands.
• consistent: the act writes the tree's parents after the name at least once.
• open: anything else.

No act is dropped and no verdict is proof. The name must stand as whole words ("Ana Zorvalo" is not inside "Juana Zorvalo"); a rank or title before it ("al Cabo Ana Zorvalo") changes nothing. When the text layer of the PDF does not hold the whole name (edicts and lists split names across columns), the note says [name not whole in the text layer: open the PDF]. The BOE prints in two or three columns, so the words around a name can mix lines of the next column; read them as a pointer to the page, never as the sentence.

The search does not index the annexes «Anuncios, avisos y Administración de Justicia» of many years, where edicts and requisitorias sit. Zero hits does not mean the person was never printed: gwt-fetch gazeta downloads whole issues to search their text. gwt-read boe ranks a person's mentions; gwt-fetch gazeta lists and downloads acts and issues.

The BOE's robots.txt disallows /buscar/gazeta.php? and /datos/pdfs/ for crawlers. The reader keeps to a person's pace: one search typed by a person at a time, one request every 6 s at best and never faster (PACE in read/gwt_boe.py), searches cached 7 days and PDFs 90 days (BOE_RULES), no parallel requests and no batch mode. The results link each act's PDF under /datos/pdfs/; the reader fetches the same file from /gazeta/dias/yyyy/mm/dd/pdfs/, which robots.txt allows, and uses the site's link only for a row without a date. To search several names, loop over search and rank the files together: gwt rank runs/*.candidates.json.

A page that is not a results page (the search form, a captcha) is a STOP: nothing written, nothing cached. So is an HTML page served at a PDF URL.

INSEE French deaths files¶

INSEE publishes the "fichier des personnes décédées", every death registered in France since 1970, as one fixed-width text file per year, and per quarter and per month for recent years, under the Licence Ouverte: https://www.data.gouv.fr/datasets/fichier-des-personnes-decedees/ . You download the years you need once. insee search reads those files on your machine and never touches the network, so it has no gate and no cache. Each line whose surname matches a --surname becomes a candidate for gwt-rank:

gwt read insee search --files deces-1985.txt deces-1986.txt --surname Ortega --surname Sanz --out runs/
gwt rank runs/<key>.candidates.json --given Tomás --surname Ortega --facts "by=1921 bp=Villazul"

To try it without a download, read/fixtures/insee-deces-1985.txt holds six invented lines in the layout: --files read/fixtures/insee-deces-1985.txt.

Which files to pass:

• Never mix a year's file with its quarter or month files. They overlap, so the same death comes back twice with the same id, and gwt-rank lists both rows. Use the yearly file for a closed year, and the quarter or month files only for the year still running. The same file given twice does the same.
• A death can sit in a later year's file. A file holds the deaths registered that year, and a late registration carries an earlier death date (a 1997 death in the 1999 file). Pass every year from the earliest possible death year onward.

Here --surname is a list of surnames to keep, not the first and second surname it is for the other sources. To find who to look for, gwt-leads --kinds missing-vital --format json lists everyone with no death event (gap death-missing). gwt-leads holds back anyone who may still be alive, and those are often the people a post-1970 deaths file can answer: matching them against a file on your own machine is the private use --include-living is for. Never feed that list into searches on outside sites.

The layout is the one the dataset page gives ("Dessin d'enregistrement"), positions counted from 1:

Every candidate is the principal of its line, with role Principal. Its id is the death date, place code and act number, which is how INSEE itself names a death. The full dates, the codes, the country, the act number, the file and its line stay in the candidate as fields gwt-rank does not score: gwt-rank scores years, and a year plus the birthplace is already enough for strong. A strong row is a lead for the verifier, never an identification. The reader writes no change list: the verifier reads the line and writes the gwt-apply change. INSEE warns that its files are published once and never corrected, and that 1970 to 1975 may have gaps.

INSEE gives no death commune name, only its code, so events.death.place is the code: give gwt-rank dp=<code>, or leave dp out. A dp=<town name> never scores.

INSEE also publishes, in the same dataset, the "fichier des oppositions": deaths whose publication someone has opposed, listed by death date, death place code and act number, which is exactly the candidate id. The reader does not read it. Before anything you publish, drop the candidates whose id is in that file.

Every line of every file is checked, matched or not. A line whose columns do not line up stops the run with exit 2, the file and line in the message, and nothing written: a date that is not eight digits, a sex that is not 1 or 2, a line too short to hold the death date (162 characters), or longer than two records (352 characters: two records glued on one line, or a file whose lines end in a carriage return alone, which would otherwise lose every record after the first). Real lines are 198 characters. A UTF-8 byte order mark at the start of a file is dropped. Blank lines are skipped. The death place code and the act number may be cut short, since the dataset page warns that some lines end in meaningless characters.

Some lines break INSEE's own rules with every column still in place. Dates are kept as written: INSEE's files hold days that never existed (29 February of a common year). They also hold a month 13 (1996), a day 60 (1991) and a full 80-character name with no * (2012), one line each in those years. These are what INSEE wrote, so the line is kept, never skipped and never fatal: the year is the date's first four digits, a name with no * is the surname whole with no given name, and the candidate lists the field under "malformed" (["birth_date"], ["name"]). The envelope gives each file's count of such lines under malformed, matched or not, and stderr names the file and the count.

The files are ASCII in practice: names and communes in upper case, without accents. The columns count characters, not bytes. A few lines, mostly foreign birthplaces, are not ASCII, and the encoding differs between years:

So each line is read as UTF-8 when it is valid UTF-8 (plain ASCII is). Otherwise it is read both as CP850 and as Latin-1, and the reading with more accented letters of the French, Spanish and Portuguese alphabets (and °) wins; a tie goes to CP850. Such a candidate carries "encoding": "cp850" or "encoding": "latin-1": its odd characters are our guess, so read them against the file.

The envelope lists each file with its sha256, its line count and how many lines matched, and the surnames asked for. The results line prints N of M lines in K files.

So exit 2 means "none found" only when the candidates file was written. A download cut short at a line boundary cannot be seen from the file alone: compare the sha256 and line count the envelope lists for each file with the download.

Tests¶

cd read && python3 -m unittest -v test_gwt_read test_gwt_pares test_gwt_genae test_gwt_boe test_gwt_insee test_registry test_controls

test_registry checks every source against the registry's shape, the three-state page classifier on fixture pages (a skeleton page is unknown, a STOP), and that a restricted FamilySearch record never reaches the gate while a refused control record still blocks it. test_controls checks that a canary skips the cache, that a short one voids the run, and that --controls brackets a search.

Invented people, descriptions and ids (the PARES pages in read/fixtures/ keep the site's markup with invented content), a fake transport and a fake gate: the tests make no request to any site. They check that a search writes an envelope and candidates gwt-rank ranks, that a repeated search is served from the cache with no network call, that the gate is asked before every request, that a 403, 429, captcha, sign-in page or empty frame stops the run with nothing written or cached, that an empty PARES page its canary answers is unreadable (exit 7, only a line in the gate log) and stays a wall when the canary cannot answer either or is the search itself, and that the token never appears in a written file or a log line. The INSEE tests use invented fixed-width lines (read/fixtures/insee-deces-1985.txt, and lines built in the test). They check that the source asks for no gate, cache or network, that each line out of the layout exits 2 with nothing written, and that a value INSEE wrote out of its own rules is kept and marked.