gwt-fetch¶

A research agent that opens a browser to save a register page is slow and leaves no trace of where the file came from. gwt-fetch downloads the file straight from the portal and writes one manifest.tsv row per file, so every image in your tree can be traced back to its source URL and checksum.

Python 3.11 or newer with requests. Run it as fetch/gwt-fetch, symlinked onto your PATH if you like, or install the package (pipx install gramps-web-toolkit, see Getting started) for gwt fetch and gwt-fetch. The code is in fetch/gwt_fetch/. Tests: cd fetch && python3 -m pytest.

iiif works with any IIIF archive in any country. digibib, gazeta and geo cover Spanish sources only. europeana searches every provider in any country; --provider Hispana narrows it to Spain.

gwt-fetch iiif URL [--pages 1-5,9] [--labels REGEX] [--native] [--out DIR] [--list]
gwt-fetch iiif --image-base URL/info.json [--pages 1-20] [--out DIR] [--list]
gwt-fetch crop IMG [...] (--box x,y,w,h [--from-size WxH] | --grid 2x3 [--overlap 10%] | --half left|right) [--max 1600] [--out DIR]
gwt-fetch digibib URL [--pages ...] [--out DIR] [--list]
gwt-fetch gazeta search (--text WORDS | --title WORDS) [--from YYYY-MM-DD] [--to YYYY-MM-DD] [--limit N]
gwt-fetch gazeta fetch BOE-A-YYYY-N [...] [--out DIR]
gwt-fetch europeana QUERY [--provider any|NAME] [--data-provider NAME] [--country NAME] [--year-from Y] [--year-to Y] [--iiif-only] [--rows N]
gwt-fetch geo PLACE [...] [--types Municipio,poblacion] [--province NAME] [--limit N]
gwt-fetch --delay 2 ...   # seconds between requests to one host, default 1
gwt-fetch iiif URL --out DIR --rawstore   # also digibib and gazeta fetch: images go to the bucket

What every download does¶

• One row per file in DIR/manifest.tsv: file, label (the canvas label or document title), image_url (the exact URL that served the bytes), manifest_url (the manifest or document page), sha256, fetched_at (UTC).
• It never overwrites. Files are written as *.part and renamed when complete, so a finished file on disk is always whole.
• Re-running is resuming. The same command skips finished files. A finished file without a row (the run died between rename and append) gets its row on the next run. A file listed in the folder's rawstore.tsv counts as finished even when it is no longer on disk, so images moved to the bucket are not downloaded again.
• --rawstore keeps images out of the working tree. After the run, every binary file gwt-fetch wrote in --out (images, PDF) is uploaded with gwt-rawstore and deleted locally once a HEAD confirms the object, its size and its sha256. Text files stay. A file the bucket does not confirm stays on disk, has no rawstore.tsv row, and the run exits 1; the next --rawstore run tries it again. The bucket settings are gwt-rawstore's, and a missing one stops the run before anything is downloaded.
• Polite by default. One request per second per host, retries on 429 and 5xx with backoff, honours Retry-After. The user agent is Mozilla/5.0 (compatible; gwt-fetch/0.1; +https://github.com/GeiserX/gramps-web-toolkit); Antenati refuses agents that don't start with Mozilla/5.0.
• Exit status. 0 when everything selected was fetched, 1 when some item failed (each failure is printed with the URLs tried and the HTTP status), 2 on a network or parse error.

Keep downloads out of the repo. Point --out at your research folder or /tmp.

iiif: any IIIF manifest, or a viewer URL it knows¶

Give it a IIIF Presentation manifest URL (v2 or v3), or a viewer URL from a portal below. It downloads each selected canvas through the Image API at full/max (the largest size the server allows), falling back to full/full on v2 servers and then to the static image named in the manifest.

--pages takes 1-based canvas positions (1-5,9,12-); --labels takes a regular expression matched against the canvas label (--labels '^pag\. (1|2)$'). --list prints position, label and URL and downloads nothing.

full/max is not always the native size: Antenati caps it at 10 megapixels. --native reads each canvas's info.json and asks for full/<width>,/0/default.jpg at the width it gives. The run checks the width of the image that comes back, and when the server sent another width (rescaled up or down) it saves nothing for that canvas, prints both widths and exits 1. Read names and dates at native width: a preview is for finding the act, not for reading it. A canvas already saved from full/max in the same --out folder is kept, and the native file goes beside it as NNNN_<label>_native.jpg; a canvas already saved at native width is skipped without a request. With --list, --native prints each canvas's info.json URL.

The cap is the server's own maxArea in info.json, and --native asks above it, on servers that already take seconds per image. Use it for the pages you will read, not for whole registers.

--image-base: an Image API with no manifest¶

Some viewers serve a IIIF Image API (.../info.json) and no Presentation manifest; French departmental archives are the common case. Their full/max and full/full can redirect to the HTML viewer, and a width above the native one comes back upscaled without an error. --image-base takes the image's base URL or its info.json and fetches it at native width, with the same width check as --native. Put {page} where the page number goes and give --pages with an end (1-20, not 12-), since there is no manifest to count the pages:

gwt-fetch iiif --image-base 'https://archives.example.fr/iiif/ark:/00000/abc/f{page}/info.json' --pages 10-12 --out /tmp/reg

Where the archive pads its page numbers (f_0007), write {page:04}. --pages without {page} in the URL, a range that selects nothing, and --labels stop with a message.

Each file is NNNN_<last path part>.jpg (0010_f10.jpg), and its row has the info.json URL as manifest_url. A page whose info.json fails is reported and the rest still download. Passing an info.json as the plain URL stops with a message that points here.

These archives serve plain manifests that gwt-fetch iiif reads as they are; build the URL from the item's id. Each template answered with a manifest in November 2026:

Antenati traps from real downloads (two Italian birth registers of the 1850s and 1860s):

• The image server is slow: 10 to 60 seconds per image, about 20 on average. Five pages take close to two minutes, so run big selections in the background and let the resume logic carry a run that gets cut off.
• A plain curl gets 403 from both antenati.cultura.gov.it (search and unit pages) and the image server. The user agent has to start with Mozilla/5.0, and image requests need the portal as Referer.
• The web viewer shows about 1600 px and full/full returns a Cloudflare 403, which led one researcher to settle for 1600 px. full/max returns the largest size allowed: 3530×2716 for one page, 2122×1684 for another, depending on the scan. --native asks for full/<width>, and gets the scan's own width.
• A canvas label is only pag. N. To find an act, list the canvases (--list), fetch a few pages around the estimate into /tmp, read the «Num. d'ordine» on each, then fetch just the right page into your research folder.

crop: cut a saved page into pieces¶

A register page is often too big to read in one look, and a double page holds two. crop cuts a saved image and writes each piece as a JPEG beside it:

• --box x,y,w,h cuts one box, in the image's pixels. With --from-size WxH the box was measured on a preview of that size and is scaled to the real image. A box that reaches past the edge is refused.
• --grid 2x3 cuts 2 columns by 3 rows. --overlap 10% makes each tile reach 10% of its size into its neighbours, so a line on a tile edge is whole in one of them.
• --half left|right cuts the left or right half of a double page.
• --max 1600 shrinks each piece so its longest side is at most 1600 px. Leave it out to keep native pixels.

--from-size without --box, and --overlap without --grid, are refused rather than ignored.

$ gwt-fetch crop /tmp/reg/0012_f12.jpg --grid 2x3 --overlap 10%
/tmp/reg/0012_f12_grid2x3-r1c1.jpg
...

Pieces are named <original>_box-x-y-w-h.jpg, <original>_grid2x3-r1c2.jpg or <original>_left.jpg. Each gets a row in the same folder's manifest.tsv, with the same columns as a download: label is crop x,y,w,h of <original> sha256:<original's sha256>, image_url is the original's URL from its own row (or its file name) with #xywh=x,y,w,h in the original's pixels, and manifest_url is the original's. Cropping a piece again keeps one #xywh= on the original: the new box is moved and scaled (after --max) into the original's pixels. Like a download, a piece is never overwritten, and re-running skips finished ones. If that folder's manifest.tsv was written by another tool (other columns), crop refuses to add to it; --out DIR writes the pieces and their rows to another folder, and image_url then names the original by its path from there.

crop needs Pillow: pipx install 'gramps-web-toolkit[crop]', or pip install Pillow beside a clone. Without it, crop prints how to install it and exits 2.

digibib: DIGIBIB viewers without IIIF¶

DIGIBÍS's DIGIBIB runs BVPH, the Biblioteca Virtual de Defensa, Galiciana and many regional digital libraries. None of these installs serve IIIF. The viewer URL is .../catalogo_imagenes/grupo.do?path=N. Each page view (&presentacion=pagina&posicion=K) names one image as imagen_id.do?idImagen=M, and that URL with &formato=jpg returns the full JPEG. gwt-fetch digibib walks those page views. When the object is a single file, grupo.do serves the file itself (often a PDF); it is saved as 0001_document.pdf.

Tested: Biblioteca Virtual de Defensa (grupo.do?path=359360, 17 pages, 3764×2443 JPEGs), BVPH (grupo.do?path=1000178404, 72 pages), Galiciana Arquivo Dixital (grupo.do?path=264703, one PDF).

prensahistorica.mcu.es and the other *.mcu.es sites return HTTP 429 to some egress addresses (seen from a corporate VPN exit) on every request, while the same request from a home connection works. If every request gets 429, change network rather than raising --delay.

gazeta: the historical Gaceta de Madrid¶

The BOE open data API (/datosabiertos/api/) covers consolidated legislation and the modern BOE and BORME summaries only, with no endpoint for the «Gazeta» colección histórica (1661-1959). gwt-fetch therefore uses the public pages, whose URLs are stable:

• Search: GET https://www.boe.es/buscar/gazeta.php with the form's own fields as query parameters. Each field is a campo[i]/dato[i] pair, joined by operador[i] (and, or, not):

Plus page_hits (only 50, 200, 500, 1000 or 2000), sort_field[0]=FPU, sort_order[0]=asc and accion=Buscar. Results are <li class="resultado-busqueda"> blocks with the title, «núm. N, de DD/MM/AAAA, página(s) P», the department and the item PDF with its reference. The output is TSV: ref, date, gazette, number, page, department, title, pdf_url, issue, issue_pdf. The site returns at most 2000 hits per search; narrow the dates for more. - Whole issue: for the Gaceta de Madrid, issue is GMD-{year}-{number} and issue_pdf is https://www.boe.es/gazeta/dias/{YYYY}/{MM}/{DD}/pdfs/GMD-{year}-{number}.pdf, with the number unpadded (GMD-1870-40, not -040). - Document by reference: https://www.boe.es/buscar/doc.php?id=BOE-A-1870-1111 (title, «Publicado en», a link to the PDF). - PDF: https://www.boe.es/datos/pdfs/BOE//{year}/{issue:03}/A{first:05}-{last:05}.pdf, the scanned page or pages of the original issue.

Traps we hit with real searches:

• --text Surname1 Surname2 matches the two words anywhere in the item, so a two-surname search returns several times more hits than it should, mostly noise. --text '"Surname1 Surname2"' is the phrase, and returns the same hits the BOE search form returns in a browser.
• References are not all BOE-A-: BOE-I-1892-51060 and BOE-B-1957-12194 came back from the same search. gazeta fetch takes any BOE-<letter>-<year>-<n>.
• An item's title can hide what matters. BOE-A-1864-10018 is titled «Gobierno de la provincia de Zaragoza. Registro de la Propiedad de Yeste.», yet its two pages are a long list of farms in Moratalla and Nerpio with every owner's name. The full-text search finds such names; the title never would. Read the PDF before you call a hit a homonym.
• The search does not index the annexes «Anuncios, avisos y Administración de Justicia» of many years. For those, download the whole issue (issue_pdf) and search its text.

To rank the acts that name one person as gwt-rank mentions, with a verdict from the birth year and the parents, use gwt-read boe search.

gazeta fetch REF saves REF.pdf and REF_text.txt. The text is the HTML text when the BOE has transcribed it. Otherwise it is the PDF's own OCR layer through pdftotext (poppler), when that is installed. That OCR is rough on 19th-century type but good enough to search.

$ gwt-fetch gazeta search --text Betanzos --from 1890-01-01 --to 1890-03-31 --limit 3
ref date    gazette number  page    department  title   pdf_url
BOE-A-1890-676  1890-02-02  Gaceta de Madrid    33  325 Ministerio de la Gobernación    Estación Central de Telégrafos.- Telegramas recibidos ...   https://www.boe.es/datos/pdfs/BOE//1890/033/A00325-00325.pdf
$ gwt-fetch gazeta fetch BOE-A-1890-676 --out research/raw/fuentes/gaceta

europeana: Spanish records with a manifest¶

The Europeana Search API needs a free key: get one at https://pro.europeana.eu/pages/get-api and export EUROPEANA_KEY=.... Without a key, the command prints a message and exits 0, so pipelines carry on.

By default it searches every provider. --provider NAME filters to one aggregator, for example --provider Hispana for Spain (4.5 million records: BVPH, Galiciana, BNE, the regional digital libraries and more). Use --data-provider to narrow to one institution as Europeana names it, for example "Galiciana: Digital Library of Galicia" or "Virtual Library of Historical Press". --country spain keeps records whose Europeana country is that one, whoever provided them. Output is TSV with a manifest column for each record, which gwt-fetch iiif can fetch.

--iiif-only adds sv_dcterms_conformsTo:*iiif*, keeping only records whose provider serves a real Image API. In October 2026 that returns zero Hispana records: no Spanish provider exposes IIIF through Europeana, while France, Austria and the UK expose millions. For Spanish records, use the Europeana manifest without --iiif-only; iiif swaps each thumbnail for the provider's original file. Where that provider blocks scripted clients (Galiciana's library, BNE), you still need the browser.

geo: Spanish places to coordinates and INE code¶

CartoCiudad's geocoder (IGN/CNIG, https://www.cartociudad.es/geocoder/api/geocoder/candidates then /find) turns a place name into coordinates, and every result carries the INE municipality code (muniCode, five digits: province plus municipality). No key needed.

$ gwt-fetch geo Betanzos Ortigueira --province "A Coruña" --limit 1
query   type    name    municipality    ine_code    province    lat lng cartociudad_id
Betanzos    poblacion   Betanzos    Betanzos    15009   A Coruña    43.278535   -8.213380   1200023058
Ortigueira  poblacion   Ortigueira  Ortigueira  15061   A Coruña    43.684475   -7.850976   1200024424

--types picks CartoCiudad result types: Municipio (the municipality, centroid), poblacion (a settlement), toponimo (named places such as churches and parish cemeteries, useful for parishes) and callejero (streets). Historical names that no longer exist won't match; geocode the modern municipality and keep the historical name in your own record.

Spanish open-data projects we looked at¶

From the awesome-spain list (https://github.com/GeiserX/awesome-spain):

• CartoCiudad / pycartociudad / caRtociudad: used. geo calls the same public API directly, so it needs no extra dependency.
• INE municipality codes (municipios-provincias-comunidades-INE-spain, ccaa-provincias-municipios): not needed, because CartoCiudad already returns the INE code with each match.
• spanish-names-surnames (INE name and surname frequencies): not a fetch source. It may help a name-normalising step later.
• boe-mcp, MCP-BOE, rOpenSpain/BOE, Datania BOE: they wrap the modern BOE open data API, which does not cover the Gazeta collection. gazeta is the part that does.
• API-CNIG, mapSpain, es-atlas: map display and boundaries, outside what fetch does.