Skip to content

Development

Each folder is one tool with its own tests. A change touches one folder, its page under docs/, and its line in the README when the tool's one-line summary changes.

Run the tests

python3 -m pip install requests pytest
python3 .github/scripts/run_tests.py              # every package
python3 .github/scripts/run_tests.py --only apply # one package (comma-separate several)
(cd spanish-poller && node --test)

run_tests.py finds every top-level folder with Python tests (a test_*.py file or a tests/ folder), runs python -m pytest -q inside it, then the folder's ci-extra.sh when there is one (check/ci-extra.sh runs the gwt-check self-test), and prints one ok or FAIL line per package. A new tool needs no CI edit: give its folder tests and CI runs them. A folder that ships a tool (a gwt-* wrapper or a pyproject.toml) with no tests fails the run, unless UNTESTED in the script names it with the reason.

CI runs the same script on every pull request (ci.yml), and node --test for spanish-poller in its own job.

CI also builds the package in a wheel job: .github/scripts/check_wheel.sh installs the wheel into a clean virtualenv outside the clone and runs gwt --version, gwt <tool> --help and gwt-<tool> --help for every tool, then gwt check selftest. A new Python tool goes in three places: its module in [tool.hatch.build.targets.wheel.force-include] and its gwt-<tool> line in [project.scripts] of the root pyproject.toml, and a row in TOOLS in core/gwt_core/cli.py. Miss the first and the wheel job fails.

No test needs a Gramps Web server or a FamilySearch or MyHeritage account. The apply tests run against an in-process fake of the Gramps Web API, and the gwt-mh tests read invented fixture pages.

Mutants: prove a test can fail

A test that passes whether or not its guard is there protects nothing. <pkg>/mutants.txt lists guards with the mutation that removes each one:

gwt_apply/ops.py | if name == "merge_person" and set(op.get("use_loser") or []) - set(USE_LOSER): | if False: | validate refuses an unknown use_loser vital

The four fields are the file (relative to the package), the exact text to replace (it must occur exactly once), the replacement, and what the guard protects. python3 .github/scripts/run_tests.py --mutants [--only PKG] copies the package to a scratch folder once per row, applies the replacement, and runs the package's tests there. The run fails when a mutant survives (the tests still pass), or when a row's text is not found.

Rows can also go in <pkg>/mutants.d/*.txt, same format, read after mutants.txt in name order; a new gwt-read source puts its rows in read/mutants.d/<id>.txt, so changes made at the same time never edit one file. When you add a guard, add its row and a test that fails with the row applied. CI runs the mutants of every package a pull request touches, and all of them on a push to main or when a pull request changes .github/.

Releases and the changelog

The version comes from git tags (hatch-vcs): a v0.2.0 tag builds 0.2.0, and a commit after it builds 0.2.1.devN+g<sha>. gwt --version in a clone prints the same shape, read from git. Pushing a v* tag runs release.yml: it builds the sdist and wheel, publishes them to PyPI through trusted publishing, and creates the GitHub release from the tag's section of CHANGELOG.md. It stops before publishing when that section is missing or has no ### Change lists heading. A manual run of the workflow is a dry run that publishes a dev build to TestPyPI only. To release, rename ## [Unreleased] to ## [X.Y.Z] - YYYY-MM-DD, merge that, then tag the merge commit and push the tag.

Every pull request that changes how an existing change list behaves (an op newly refused, a newly required field, a different skip or duplicate rule, a different default, in gwt-apply or in what gwt-import-excel writes) adds a line under ### Change lists in the Unreleased section of CHANGELOG.md. Other user-facing changes go under Added, Changed or Fixed.

Build this site

python3 -m pip install -r docs/requirements-docs.txt
python3 -m zensical serve                          # http://127.0.0.1:8000, rebuilds on save
python3 -m zensical build --clean --strict         # the build CI runs
python3 -I .github/scripts/site-check.py site      # the check CI runs after it

The site is built with Zensical from docs/ and zensical.toml. The strict build fails on a broken link, a link to a missing heading, a link that leaves docs/, or a link to an excluded file (a capitalised .md); link a file outside docs/ by its GitHub URL. site-check.py fails on what the build lets through: a page left out of the nav, a nav entry for a missing file, a missing image, a link rooted at /. A push to main that changes the docs publishes the site.

Rules for a change

  • Every change reaches main through a pull request with green CI. Commits follow Conventional Commits.
  • No private data. Examples and test data are invented: no real people's names from anyone's tree, no real case numbers, tree ids, hostnames, IP addresses, accounts or email addresses. Public records, such as a Gaceta page or a published parish entry, are fine with their source.
  • Never commit a secret, a certificate or a configuration file with credentials.
  • A new trap goes into Hard-won lessons: what happened, why, and the rule.

Security problems go through the security policy, never a public issue. The toolkit is licensed GPL-3.0-or-later.