gwt-mh¶

gwt-mh searches MyHeritage's historical records, opens record pages, and keeps everything a record gives on our own storage: the record data, the citation, the collection's metadata and, where the plan allows it, the original image. It works on a free account and does more on a paid one. It finds out which plan the account has from what the pages show, never from a setting.

pip install playwright        # over CDP no browser download is needed
G="myheritage/gwt-mh --cdp http://<chrome-host>:9223 --storage-state ~/.config/mh/state.json --account ana"
$G search --given Ana --surname Zorvalo --by 1851 --exact --out runs/        # -> runs/<key>.json + <key>.candidates.json
rank/gwt-rank runs/<key>.candidates.json --given Ana --surname Zorvalo --facts "by=1851"
$G fetch --collection 12040 --item 800001- --search runs/<key>.json --out records/
$G record --collection 10182 --item 70001 --out records/
$G plan                       # what this account's plan has been seen to allow

1. What it is, and why a browser¶

MyHeritage sits behind Imperva. A logged-out browser context from our addresses got an Imperva bot page on every research, record and catalogue page. Plain HTTP with copied cookies would risk a challenge on the account itself. So every request goes through a real Chrome, reached over CDP, inside a context signed in as the account. The tool opens the same URLs a person opens and reads the answers from inside the page.

Searches are read from the site's own search JSON, not from the page text. Opening a /research?...action=query URL makes the page call POST /web-family-graphql/search_in_historical_records_split/, and the tool captures that answer with a response listener. It never builds GraphQL calls and never handles the bearer token. The answer is structured: id, name, link, collection (is_free, has_images, record count), thumbnail and the display fields. The server already hides what the plan does not cover. A hidden value comes back as a link to /paywall?...subscription_type=data with the word Día, Mes or Lugar. The letters the page shows in its place are decoration, so text scraping would read nonsense.

Logging in is per account and private (vault, a code mailed to the owner, which box), so it is not in the toolkit. A login script writes the account's Playwright state file (mode 600), and gwt-mh --storage-state FILE opens a fresh context of its own from it. It never attaches to a context already open in that Chrome, because that context may belong to another tool or agent. A session is never copied to another account or tool.

It reuses gwt-gate (one gate per account, start=17 floor=10 ceil=60, plus a random 0-8 s), gwt-fscache (cache key with the plan in it), gwt-rank (it writes the {"candidates": [...]} shape), gwt-ledger and gwt-rawstore, without changing any of them.

2. Plans¶

Prices read on 2026-10-24 from https://www.myheritage.es/pricing for Spain, in EUR per year, VAT excluded, first year then renewal.

The monthly tree backup ("Copia de seguridad") is a separate add-on at 5 EUR a month.

3. Free vs paid¶

Observed means we saw it on an account with that plan in October 2026: Basic on a free account, Omni on a paid one. Documented means MyHeritage's pricing page or help says so and we did not see it. Premium, PremiumPlus, Data and Complete were not observed directly. The Basic observations come from a single run on one free account on 2026-10-24; a second check of the original-image and OldNews rows is still due, and that account was locked for 24 h by the attempt to make it (see Traps).

FamilySearch-sourced collections on MyHeritage (the FamilySearch Family Tree, collection 40001, about 1.18 billion profiles) were masked on Basic like any other paid collection. We found no free tier for them. Free MyHeritage access exists for Latter-day Saint members through FamilySearch partner access, and as Library Edition on site at FamilySearch centers and many public libraries.

4. What a paid plan adds to gwt-mh¶

5. Plan detection¶

Every capture adds observations {at, signal, value, url} to plan.json in the cache dir. The capabilities come from them, and the newest observation of each signal wins, so a lapsed or renewed plan is picked up.

images is kept per collection (collection in the observation) and the capability is only a summary. It never decides whether fetch tries an image: on a plan with records true, every record of a collection with images gets the image step. One index-only collection therefore never switches images off for the others.

The label is basic (records false), data+ (records true), omni (records and OldNews pages), else unknown, which behaves as Basic. It goes into every manifest row and into the cache key, so a masked Basic answer is never served to a paid account. OldNews results are excluded from the positive signal because they show places even on Basic.

6. Safe use¶

• One session per account, one gate per account (--gate-dir, default ~/.cache/gwt-gate-mh-<account>). The session takes <account>.lock in ~/.cache/gwt-mh-locks ($GWT_MH_LOCK_DIR), keyed on the account and not on the gate dir, so two runs with different --gate-dir still cannot share an account; the second one stops with exit 7. The kernel drops the lock when the process dies.
• A random 10-25 s between page loads (the gate's 17 s start, down to 10 s after an hour without a hit, plus 0-8 s), and a 2-4 min rest every --rest-every loads (default 45). An Omni account searched 148 times at that pace with no wall.
• On a wall, stop. The Imperva texts (_Incapsula_Resource, Incapsula incident ID, Security check, the Spanish Control de Seguridad, an hCaptcha), "actividad inusual" / "unusual activity", and the account lockout ("El acceso se ha desactivado temporalmente ... actividad irregular" / "temporarily disabled ... irregular activity") stop the run with exit 3 and block the account for 24 h: in its gate (gwt-gate block 24) and in <account>.wall next to the account's lock, which every session checks before it opens a browser. So every later run on that account, whatever its --gate-dir, stops with exit 6 before it requests anything. An image download is read the same way: a 403 or 429, or a page with those texts instead of image bytes, is a wall, and a 401 or a login page is exit 4; the run stops there and fetches no other link and no thumbnail. They are matched without regard to case and before the /login test, because the lockout is served on /login. A logged-out page (/login, a verification-code prompt, Iniciar sesión with no account menu) becomes exit 4. The tool never clicks anything on a wall; a person looks.
• Never log in again after a wall. The lockout lasts 24 h, and each new login attempt from the same account or address during it can extend it. Three logins in one day from one address got a free account locked and the address an Imperva hCaptcha. After exit 3, make no request on that account or from that address for at least 24 h; the gate enforces the account half, the address half is the caller's. Exit 4 allows one new login, never a retry loop.
• Read only. The tool opens URLs and reads them. It has no code path that saves a record, confirms or rejects a match, uploads, sends a message, starts a trial or buys. matches lists cards and never opens a review.
• Never share, export or copy a session or its cookies between accounts or tools.
• Session tokens never reach disk. The page HTML carries several (fgToken, webClientFgToken, mhXsrfToken, csrf_token) besides the search's bearer_token; every envelope is scrubbed before it is cached or written.

Exit codes (also in gwt-mh --help):

7. Storage: the most we can keep¶

fetch writes DIR/<collection>/<item>/ and a manifest.tsv with file kind url sha256 size width height plan original fetched_at collection_id item_id. Kinds, in the order tried:

1. record-html, record-text, record-json (every field parsed from the page), normalized (the gwt-rank record), citation, collection (name, record count, description, provider). On Basic, where the record page is paywalled, record-json is the search record itself, the citation is built from it, and collection comes from the search's collection block.
2. image: the record page's image download link fetched with the session (at most two links, https only: on a MyHeritage site (myheritage. plus one of the country domains listed in MH_TLDS, and the one --host names) with image or scan in the path, or an image file on MyHeritage's image CDN (myheritageimages.com). Any other host is skipped, and so is a URL with a user name or a backslash before its host. On a MyHeritage site, so is any link whose unquoted path or query contains gedcom, export, subscri, checkout, purchase, upgrade, buy, paywall or install, in any case, or the words app, dna, trial or mobile. That list does not apply on the image CDN, which only serves files and whose newspaper titles and signed tokens can hold those letters (El_Exportador, sig=Abuyz09). A download is judged by its final URL after redirects: one that ends on /login is a logged-out session (exit 4) even with no page, and one that leaves those hosts is dropped), or else the largest image the page loaded, by pixel area. Tried on every record of a collection with images once records is true.
3. oldnews-page and oldnews-ocr: the unblurred page from OldNews.com (a link whose host is oldnews.com or a subdomain of it) and its text, only when oldnews_pages is true. The blurred page Basic gets is never saved; it adds nothing to the thumbnail.
4. thumbnail: the search thumbnail, original=no.
5. screenshot-fallback: only when step 2 or 3 fails on a plan that allows it, original=no.

Width and height come from the image header. Files are written as *.part and renamed. Then gwt-rawstore put --remove moves the HTML and image kinds to the bucket (--no-rawstore skips it); the .txt, .json and .tsv files stay in git.

The OldNews image path names the newspaper, date, issue and page (<title>/<YYYY>/<MM>/<DD>/<issue>/<page>.jpg). The normalized record keeps that as mh.image_hint, which is often enough to find a free copy of the same page in a national library.

Not measured yet: how a paid account serves image bytes, and at what size. Steps 2 and 3 fall back to a marked screenshot when they fail, so nothing is lost before that measurement.

8. Search output and the ledger¶

search writes the envelope (<key>.json: URL, final URL, page text, HTML, captured search JSON) and <key>.candidates.json, and prints the ledger source and evidence path: the reader shape gwt-read writes for FamilySearch too. Sources: myheritage, myheritage-oldnews (collection 12040 and other OldNews collections), and myheritage-espejo-fs-<cid> for a collection that mirrors a FamilySearch one. The ledger compares sources as exact slugs, so a mirror row never counts as the FamilySearch search done. gwt-mh never writes the ledger; the lane runs gwt-ledger check before and add after.

Each normalized candidate is the matched person (principal), with relatives under relatives (father, mother, parent, spouse, child, other). A masked value is null or "" and its field is listed in mh.masked, so a verifier can tell "hidden on this plan" from "not in the record". Extra fields stay under mh and the ranker ignores them.

9. The FamilySearch-mirror test¶

Some MyHeritage collections are copies of FamilySearch indexes. mirror-check decides whether one can stand in while FamilySearch is unavailable. Input: a JSON list of about 20 FamilySearch records we already have (name or given/surnames, event, year, place, father, mother). For each, it runs an exact search in the MyHeritage collection and compares name, year, place and parents, only where both sides show the field. The row in mirrors.tsv is cid fs_collection n found agree date verdict plan compared:

• equivalent: at least 90% found and every found one agrees on every compared field;
• partial: at least half found;
• not: otherwise.

Only equivalent lets MyHeritage go first for that collection, under myheritage-espejo-fs-<cid>; the FamilySearch row stays due. On Basic, place and parents are masked, so the verdict rests on name and year, and the compared column says so. A verdict reached on a paid plan is judged again before it is trusted on Basic.

10. FamilySearch, for a follow-up¶

FamilySearch gives original images for many collections at no cost; some are limited to signed-in users, affiliate libraries or FamilySearch centers. gwt-read searches FamilySearch and reads its records in the same shape as gwt-mh, and gwt-fetch downloads images, but neither yet keeps the image, the record JSON and the citation in rawstore with a manifest like this one. Aligning them is a follow-up.

Commands and options¶

Global options go before the command:

Traps¶

• MyHeritage's 24 h lockout ("El acceso se ha desactivado temporalmente ... actividad irregular") appears on /login after a password is sent. Read as "logged out" it invited another login, which keeps the lock going. It is a wall now. Repeated logins from one address in one day also bring an Imperva hCaptcha on every page; a re-check belongs on another day, from an address that has not just logged in.

• OldNews search results show places on a free account. Counting them as paid evidence labelled a free account data+.

• The record page puts the place on its own line under the date, starting with a space and a no-break space.
• A credentialed in-page fetch to the image CDN fails on CORS; image bytes fall back to the context's request client, which sends the same cookies. That retry is a second request and takes its own gate slot.
• A wall used to be a gate hit only, which pauses 20 min, while the lockout lasts 24 h, and an image download that met a wall was skipped as "not an image" and the next link was tried. Both now block the gate for 24 h and stop the run.
• The catalogue cards show no region or years; the site filters those through the page, and the tool does not guess that filter's parameters.

Tests¶

cd myheritage && python3 -m unittest -v test_gwt_mh

Synthetic fixtures only, no network, no browser; the session runs against a fake page, context and Playwright on a fake clock. They cover the 24 h block after a wall, in the gate and per account, walls and logged-out answers in image downloads, the download filter as fetch uses it, the account lock, the new context, the exit codes, the normalizer (checked against gwt_rank._bad_record), plan inference and its precedence, walls, the cache key and token scrubbing, image paths and headers, the manifest, fetch planning, ledger slugs, the mirror verdict, the record page from saved HTML and from page text, match cards and the catalogue.