Skip to content

Testing

This page is for contributors. It says what CI runs, how the end-to-end test works, and how to change its data.

The stub runs in nextflow.yml check that the Nextflow wiring holds together, and container-test.yml runs each changed image's own tool on the fixture. Neither runs a pipeline step as the scripts and modules call it, so a step can be broken in real use while both stay green. The E2E workflow (.github/workflows/e2e.yml) closes that gap: it runs the real tools, in the pinned containers, on a small slice of a public genome, and checks what they write.

Every nextflow run step in nextflow.yml has a time limit of its own, well under the job's 45 minutes. It is 10 minutes for the runs without containers, which take under a minute, and 20 for the stub run that pulls images, which took about four minutes in the run we timed. When a step of that job fails or is cut by its limit, the job prints which tasks were running and which were waiting to start, and uploads .nextflow.log as the artifact nextflow-logs-<line> (pinned or 26.10). Without that log the first two hangs of the retry step could not be explained (Lessons learned).

The fixture

The test data is a slice of HG002, the Genome in a Bottle (GIAB) son of the Ashkenazi trio. HG002 is a public, consented reference sample, so nothing in the fixture is personal data.

scripts/ci/build-fixture.sh builds it. It reads only the regions below from GIAB's 60x GRCh38 BAM on GIAB's S3 mirror (https://giab.s3.amazonaws.com/) over HTTPS (samtools fetches byte ranges through the .bai, so the 126 GB file is never downloaded) and samples them down to about 30x.

Region (GRCh38) Why it is there
chr1:109,600,000-109,800,000 GSTM1 (pypgx reads depth over it)
chr2:233,600,000-233,800,000 UGT1A1 and UGT1A4
chr4:68,500,000-68,700,000 UGT2B15 and UGT2B17
chr16:28,580,000-28,630,000 SULT1A1
chr19:40,800,000-41,050,000 CYP2A6, CYP2A7 and CYP2B6
chr20:10,000,000-10,500,000 small variants; the planted ClinVar record (in SNAP25); the GIAB truth slice
chr22:42,000,000-42,300,000 CYP2D6 and CYP2D7 with flanks
chr12:47,800,000-47,950,000 VDR, the control gene pypgx normalises depth with
chr10:94,700,000-95,000,000 CYP2C19 and CYP2C9
chr6:29,900,000-33,100,000 HLA
chr5:69,900,000-71,100,000 SMN1 and SMN2
chrX:73,700,000-74,000,000 non-PAR chrX
chrX:1,000,000-1,200,000 PAR1
chrY:2,700,000-3,000,000 non-PAR chrY (HG002 is male)
chrY:6,800,000-7,000,000, chrY:12,600,000-12,800,000, chrY:14,600,000-14,800,000, chrY:19,600,000-19,800,000, chrY:20,500,000-20,700,000 five more single-copy stretches of chrY (TBL1Y, USP9Y, NLGN4Y, KDM5D, EIF1AY), each in a different piece of the single-copy part step 33 reads, so its mosaic loss-of-Y estimate rests on about 120 blocks of 10 kb over six pieces (since fixture-v7)
chrM the whole mitochondrial genome, sampled to about 500x (the source has thousands of x there)

The chr5, chrX and chrY slices are not used by every step yet. They are there so the paralog, ploidy, sample-QC and Y-haplogroup work needs no new fixture.

What the release holds:

File Content
HG002_R1.fastq.gz, HG002_R2.fastq.gz the sliced reads as name-sorted pairs, input for step 02
HG002_slice.bam (+ .bai) the same reads as GIAB aligned them
HG002_cyrius.bam (+ .bai) GIAB's alignment of the regions Cyrius (step 21) reads, sampled the same way: CYP2D6, CYP2D7 and its 3,000 depth-normalisation bins on chr1 to chr22, plus the two 50 kb flanks (chr22:42.05-42.10 and 42.20-42.25 Mb) step 21's depth check compares CYP2D6 with. Cyrius stops on the first bin whose contig the BAM lacks, so it cannot run on the BAM step 02 writes against the fixture reference
fixture_ref.fa.gz (+ .fai, .gzi, .dict) whole chr1, chr2, chr4, chr5, chr6, chr10, chr12, chr16, chr19, chr20, chr22, chrX, chrY and chrM from the NCBI GRCh38 no-ALT analysis set, the pipeline's default reference, so every coordinate and contig name is real
clinvar.vcf.gz, clinvar_chr.vcf.gz, clinvar_pathogenic_chr.vcf.gz (+ .tbi) ClinVar records inside the regions, built the way setup.sh builds the full files
planted.tsv one synthetic ClinVar record (CLNSIG Pathogenic, gene SNAP25, ID 900000001) at a SNV HG002 is homozygous for, so step 06 always has a hit with a known gene
HG002_vep.vcf up to 200 GIAB truth variants (HLA-A, -B, -C, CYP2C19, CYP2C9, SNAP25) annotated by VEP --database --everything; it stands in for step 13, whose offline cache does not fit a runner. --database gives no gnomAD frequencies, so steps 23 and 31 only run their no-gnomAD branch
revel_synthetic.tsv.gz (+ .tbi) a score file in REVEL's layout for the SNVs of the VEP subset. The scores are made up; they only show that a score track is applied
HG002_sv_manta_style.vcf.gz (+ .tbi) ten Manta-style SV records, input for AnnotSV and the SV readers
HG002_truth_chr20.vcf.gz (+ .tbi), HG002_truth_chr20.bed GIAB v4.2.1 truth for the chr20 slice
HG002_GRCh38_1_22_v4.2.1_benchmark.vcf.gz (+ .tbi), HG002_GRCh38_1_22_v4.2.1_benchmark_noinconsistent.bed, HG002_GRCh38_v5.0q_smvar.vcf.gz (+ .tbi), HG002_GRCh38_v5.0q_smvar.benchmark.bed GIAB's two HG002 truth sets, whole and byte for byte as GIAB publishes them, for benchmark-variants.sh --giab (its md5 check accepts nothing else). The GIAB e2e case reads them from here, so it needs no download. About 214 MB of the release. The build checks each file against the md5 the step accepts; MANIFEST.txt lists the md5s and sources
regions.bed, MANIFEST.txt, SHA256SUMS the slices, how this build was made (sources, depth, ClinVar date, build-script checksum), checksums

Why chr2, chr4 and chr16: pypgx (step 32) reads depth over the region of every gene it can call copy number for before it calls any of them, and samtools refuses a region on a contig the BAM does not have. One missing contig and step 32 calls no BAM-based gene at all, CYP2D6 included. Its GRCh38 region for GSTT1 is on an ALT contig, chr22_KI270879v1_alt, which the default reference does not have; step 32 leaves GSTT1 out on the fixture as it does on a real sample. Up to fixture-v4 the fixture carried that contig; fixture-v5 dropped it, so the e2e job runs on the contigs a default setup has.

The whole release stays under 1.5 GB. The build checks that both BAMs pass samtools quickcheck, that every .gz file passes gzip -t, that samtools idxstats shows reads on every primary contig of the reference, that the reference has no ALT or HLA contig, that the Cyrius BAM has reads on chr1 to chr22 and reads with MAPQ >= 1 in both CYP2D6 flanks, and that each GIAB truth file has the md5 benchmark-variants.sh accepts. After download the e2e job repeats the BAM, gzip -t, contig and size checks (tests/e2e/01-fixture.sh); the GIAB case checks the truth files by md5 and pgx-1-depth-check.sh checks the flank depth.

Where it lives and how to change it

The fixture is published as assets of a GitHub release, a prerelease that is never marked latest. Its tag is the one line in tests/fixtures/VERSION (now fixture-v7). The e2e job reads the same file, so the workflow never changes when the data does.

To change the data:

  1. Edit scripts/ci/build-fixture.sh.
  2. Bump tests/fixtures/VERSION (for example from fixture-v7 to fixture-v8).
  3. Push the branch and open the pull request. The build-fixture job runs on every pull request and on every push run on main, and does nothing while the release exists and was built by the same script. When the new tag has no release yet, it builds the data on a GitHub runner (30 to 70 minutes, most of it VEP querying Ensembl's public database) and publishes the new release, MANIFEST.txt last. A newer push to the pull request cancels a build in progress; a release left without MANIFEST.txt is built again by the next run. The e2e job of your pull request waits up to 75 minutes for it, and that wait counts against the job's time limit (timeout-minutes of the e2e job in e2e.yml). A pull request from a fork gets a read-only token and cannot publish the release: build-fixture fails at once, and a maintainer builds the release from a branch of this repository with job: build-fixture.

A pull request that changes the build script but keeps the old version fails build-fixture on purpose: the existing release was built by different code, and replacing its files would change the data under every open pull request. To rebuild a release in place anyway (for example after a failed upload), run the E2E workflow by hand with job: build-fixture and rebuild: true. It deletes the old MANIFEST.txt before it replaces the other files, so a cut rebuild is built again by the next run.

You can build it yourself on Linux with Docker, bgzip, tabix and python3 and about 10 GB of free disk: scripts/ci/build-fixture.sh /path/to/out.

The e2e job

scripts/ci/e2e-run.sh downloads the fixture, checks SHA256SUMS, lays out a GENOME_DIR the way setup.sh and step 13 would leave it, and runs the case files in tests/e2e/. It runs all of its cases even when one fails, so one run lists every broken step.

In CI the cases run in parallel legs, each case exactly once. On 2026-11-02 the 130 cases took 230 minutes one after another on one runner (run 41390434697), against a 300-minute job limit. Now:

  • e2e-plan splits the cases with scripts/ci/e2e-shards.sh. The leg e2e base runs the cases of stage base in scripts/ci/e2e-case-times.tsv (every numbered case: the fixture check, setup validation, alignment, DeepVariant, GATK, the single steps on HG002, the reports and the Nextflow run; and bash-step-08-hla), because most other cases read what those wrote. Every other case goes to one of E2E_SHARDS shard legs (e2e shard 1 to e2e shard 6).
  • The table holds each case's median time on six passing runs of 2026-11-01 and 2026-11-02, and its needs: the non-base cases whose outputs it reads. The needs were written from what each case says it reads (its header and its checks) and from the cases that failed in a split run without them (run 41455829120). A case and every case it needs, followed through, form a group that stays in one shard, in name order. Groups go to the least loaded shard, heaviest first. A new case with no row counts as 300 seconds and needs nothing. A case that runs well over 300 seconds also needs a row with its measured seconds, or the split balances it as a 300-second case.
  • The base leg runs first. It packs every file its cases wrote or changed (at its absolute path, with owners, modes and times) and uploads it as the artifact e2e-base. The shard legs start only after it passed (needs:), so a shard never holds a runner while it waits: when many runs share the account's runners, shards that started early and waited took the runners the base legs needed (run 41506673662 had three shards running and its base leg still queued). Each shard fetches the artifact, tried up to five times, takes the base leg's work area, unpacks the archive there, deletes it (about 4 GB the cases need) and runs its cases. When the base leg fails, the shards are skipped and e2e is red.
  • The shard that holds nextflow-from-fastq-2-nextflow and the one that holds nextflow-hardening-2-nf run the parity checks; the plan fails when either case and the case before it in its pair are in different shards.
  • The required check is the job e2e. It merges every leg's results into one table of case, result, time and job, followed by the failed checks of each failed case, and it is red unless the plan and every leg succeeded, every case of the split reported exactly one pass and each parity check passed once. A leg that was skipped, cancelled or cut by its limit leaves cases with no result, and that is red too. The full logs are in the e2e-logs-<job> artifacts.

tests/test_e2e_shards.sh (unit tests) checks the split against planted faults: a case listed twice or missing, a stale or malformed table row, a need on a case that runs later, and results missing, failed or reported twice.

To measure the table again, run the E2E workflow by hand with job: e2e and trace: true. That run puts every non-base case in one shard, so it runs as long as one serial run. After each case e2e-run.sh records its time and, with the file systems remounted strictatime, every file under the work area and the checkout whose atime moved during the case, attributed to the case whose run last changed that file's ctime. The job e2e then prints a new e2e-case-times.tsv in its summary and uploads it as the artifact e2e-case-times; it keeps each row's stage and needs and adds the traced needs. It refuses a trace that saw no case read another case's file. The traced needs are an upper bound, not the table to commit: a file counts for the case that changed it last, so a case that rewrites or touches a shared file (a report, an index) shows up as a need of every later case that reads that file, and a case that scans a folder reads everything in it. On the trace of run 41454755531 they joined almost every case into one group, so check each traced need against the case before you add it. A read the trace cannot see, such as a test for whether a file exists, is not in the table: such a case fails in its shard, and its row then needs a hand-written needs.

Every pull request starts the workflow. Its e2e-scope job checks whether the pull request touches scripts/, modules/, workflows/, bin/, conf/, assets/, tests/e2e/, tests/fixtures/VERSION, main.nf, nextflow.config, versions.env or the workflow itself. When it touches none of them the plan and the legs are skipped and the job e2e passes without running anything, so main can require the e2e check without blocking a docs-only change. The e2e job also runs once a month and by hand (job: e2e). It also runs on main after every merge that touches those paths, because main does not require a branch to be up to date before it merges: two pull requests that land close together each passed alone, and only the run on main tests them together. A docs-only merge starts no run. A run on main is never cancelled half way. A merge that lands while one runs waits, and GitHub keeps one waiting run, so a newer merge replaces the waiting one and the next run tests every merge since the last run. A red run on main names its commit, the newest merge it tested; the merges between the last green run on main and that commit are the suspects. The push paths in e2e.yml and the E2E_SCOPE_DIRS and E2E_SCOPE_FILES lists that e2e-scope reads are the same scope written twice, and the first step of e2e-plan fails when they differ. A push to any other branch starts no run: its pull request runs e2e, and an e2e check from a push run would sit on the same commit as the pull request's required e2e check. The DeepVariant case passes the fixture slices as INTERVALS, so DeepVariant calls only those slices instead of walking all 1.8 Gb of the reference, which alone took 27 to 48 minutes. It uses a standard GitHub-hosted runner (4 CPUs, 16 GB of RAM) after deleting preinstalled toolchains it does not use (Android, .NET, Haskell, CodeQL, Boost) to make disk room.

The images the cases use are cached in four GitHub Actions cache entries, one per group. scripts/ci/e2e-run.sh --image-groups names each image's group, taken from its repository name without the tag, so a version bump changes the key of one group only; a group whose key missed restores the newest older entry of that group and pulls only what changed. A group keeps its smallest images up to 1500 MB of zstd output, and the larger ones are pulled on every run. Four groups of 1500 MB and the two data entries below stay under the repository's 10 GB cache limit even while main replaces one group: 5.9 GB of images, 1.5 GB more for the group being replaced, and 2.0 GB of data. Images pinned by digest are not cached, because docker load gives them back without the digest, and they are left out of the group's key, so a digest bump misses no entry. Only the base leg saves, and only on main (or in a run by hand with save_images, on any branch), after it pulled every image the cases use and got through its cases, and only the groups whose key missed: a cache saved by a pull request can be read by that pull request alone and would only push main's entries out. The job summary lists every image with its size and what the cache did with it: restored on an exact key hit, restored from an older entry, saved, or pulled. Every merge in the e2e scope starts a run on main, so the cache follows main's images. GitHub drops an entry no run has read for 7 days, so after a quiet week the next run on main fills the cache again. Until this layout the cache was one entry, and by October 2026 it had grown to about 11 GB, over the limit, so every run missed it.

Two downloads are cached the same way, because they come from EBI's 1000 Genomes server, which is slow and at times stalls: MoChA's chr20 panel with its manifest (0.64 GB, case mocha-1), keyed on MOCHA_PANEL_MANIFEST_SHA256 and the chromosome, and the 1000 Genomes phase 3 sites VCF (1.36 GB, case archaic-ancestry-1), keyed on ARCHAIC_1KG_SITES_MD5. scripts/ci/e2e-run.sh hands each case its files just before it runs, at the paths setup.sh reads them from, so setup.sh downloads nothing; on a miss it downloads them first, checked against their pins. Only the leg whose cases include the case restores and saves its entry, and it saves only on main or in a run by hand with save_images, as with the images. The job summary says whether each key hit, and the case's log says where its files came from. Every download, there and in setup.sh, gives up on a transfer that stalls (FETCH_STALL, see fetch in scripts/lib/common.sh), and the next try resumes it.

How long it took in one job, before the split: a passing e2e job took about 105 to 205 minutes with 84 cases, and 230 minutes of cases with 130 (run 41390434697, 2026-11-02). The runner decides much of it, not the cache: the same CPU model took 106 minutes in one run and 166 in another. In October 2026, with 84 cases, the same code took 106 and 166 minutes on an AMD EPYC 9V45 (runs 39197502832 and 39075176392, the second of which also saved the cache), 112 minutes on an Intel Xeon 6973P (run 39088086725), 156 and 191 minutes on an AMD EPYC 7763 (runs 39075764278 and 39075201465) and 205 minutes on an AMD EPYC 9V74 (run 39197512594, which saved three groups). The image cache saves about 7 minutes of that: pulling every image took 9 minutes, restoring and loading the four groups took 2.5 (run 39075201465 restored 39 images in 118 seconds), and saving them all took 12. Each leg has a limit of 240 minutes (355 for a traced shard leg); the base leg waits up to 75 minutes for a fixture a pull request bumps. Each leg's first steps print the runner's CPU model and CPU count to the log and the job summary, so a later timing can name the hardware.

Before the first case of a full run, and of every leg, e2e-run.sh pulls every image the cases use that is not on the machine yet: the NAME_IMAGE lines of versions.env and of the case files, less a list of images no case runs. A pull that fails with a registry 5xx is tried again, up to four tries in all, 20, 40 and 80 seconds apart. Any other error, such as a missing tag or a rate limit, fails the job at once, and the tools' own docker run is never retried. A Docker Hub 500 on one image's manifest is what failed main's run 37980248545 the first time. scripts/ci/e2e-run.sh --self-test-pull checks the retry against a fake docker, and e2e-plan runs it. A run given a pattern skips the pre-pull, because most images reach a case through the steps it runs and cannot be picked per case; its cases pull their own images, with one try each.

What it covers:

  • Bash steps, in order: validate-setup.sh, 02 (alignment from FASTQ), 03 (DeepVariant), 03a (GATK HaplotypeCaller on the chr20 slice), 06, 07 (PharmCAT), 11, 12, 16, 16b, 20 (Mutect2 on chrM), 21 (Cyrius, on HG002_cyrius.bam), 32 (pypgx with its bundle), 27, then 30, 23 and 31 on the VEP subset with the synthetic score file, then 24 and generate-report.sh.
  • Nextflow: nextflow run main.nf -profile docker with real containers on the VCF and BAM the bash steps produced, through today's VCF+BAM samplesheet, with --tools clinvar,mosdepth,delly,vcfanno,roh,pharmcat,cpic, --max_cpus 4 --max_memory 14.GB.
  • Parity and the launcher: nextflow-from-fastq-*.sh run the bash steps 01b, 02, 03, 06, 07, 11 and 25 and the pipeline on the same FASTQ pair, and scripts/ci/parity-diff.sh compares what they wrote (Bash vs Nextflow parity). run-all-launcher.sh runs scripts/run-all.sh from the same reads in a GENOME_DIR of its own: it must publish the same files as the direct nextflow run, pass the same parity check against the bash steps, write both reports, start a second run from the BAM, VCF and gVCF the first one published with no FASTP, minimap2 or DeepVariant task, and on a third run take every task from the cache in under ten minutes. Each comparison has a negative control in the same run.
  • Manta and the Nextflow hardening (nextflow-hardening-*.sh): step 04 on a planted Manta-style VCF with two inversion breakend pairs (they must come out as two SVTYPE=INV records) and on the fixture BAM inside the fixture's regions (MANTA_CALL_REGIONS); then a Nextflow run with --tools clinvar,pharmcat,mosdepth,delly,manta,vcfanno,pypgx,telomere_hunter,mito_haplogroup,survivor_merge,y_haplogroup, which checks that every container ran with --network none and that no task built a .fai of the reference. The samplesheet declares HG002 male with --sex_check warn (indexcov reads the slices as female), so Yleaf runs; the SURVIVOR consensus of Manta and Delly must have records with SUPP 2, and Yleaf's table one row with markers read. The bash steps 22 and 37 then run on the same calls and BAM in a GENOME_DIR of hard links with a male indexcov .ped, and the workflow compares sv_merged/ and y_haplogroup/ with PARITY_ITEMS="sv_merged y_haplogroup" scripts/ci/parity-diff.sh, after its self-test on those two items.
  • Fine-grained ancestry (ancestry-fine-grained-1.sh): setup.sh --ancestry-fine-panel reads a reduced panel from gnomAD's bucket (the fixture's regions plus chr22:20-35 Mb, two people held out), then step 26b runs on HG002's gVCF, on HG002's GIAB genotypes at the panel's SNVs and on a planted half-Yoruba, half-Han-Chinese person, and the Nextflow module runs on the same genotypes. The painting checks have their negative control inside the case (the two sides of the planted junction swapped), and a red control refits the panel with every reference label moved to the next group and requires the fraction checks to fail on it.
  • Raw-data export and its round trip (consumer-export-1.sh): step 40 writes the 23andMe and AncestryDNA files from HG002's gVCF with the CI slice of the site table (tests/fixtures/consumer_export/sites.tsv.gz), and both pass the strict reader. Every site's genotype must match a second, independent reading of the gVCF, and a planted hole in the gVCF must turn that site into -- and give no site another genotype. scripts/chip-to-vcf.sh then reads both files back and lifts them to GRCh38 with UCSC's hg19ToHg38 chain: the records must land on the table's GRCh38 positions with the alleles of case 21's calls. Red controls shift the table by one position on GRCh38 (the export stops) and on GRCh37 (the round trip fails). The Nextflow module must write the same files as step 40.
  • Index times (run-all-sample-qc-and-stale-index-1-index-times.sh): every index the direct nextflow run and the launcher published is not older than its data file, and samtools and bcftools print no stale-index warning on the BAM and the VCF. Then a short VCF-only run with three planted pairs checks main.nf's completion handler: it sets the index times of the run's own files after the last copy, and leaves an index older than the run as it was. A copied pair with its index set an hour older is the negative control.

  • The Nextflow report's tables (paralogs-report-section-1-nf-report-tables.sh): HTML_REPORT runs for real on the CYP2D6 depth check, the cDNA read check and the phased compound-het pairs that earlier cases wrote, and its reports must show the depth-check line, the cDNA column with its flagged count and the pairs by phase; the same run without the three tables must show none of them. A stub run of every tool then checks that main.nf passes the cDNA table to each HTML_REPORT task that got ClinVar hits, against a copy of the pipeline with that join removed.

  • Report assets: a last case lists every external http(s) address in a src= or href= attribute of the generated HTML reports, so the remote files a report loads when opened are known. The list goes to the job summary.

Each check is a count or a column: a VCF with records, the sample name in the VCF header, a gene in the ClinVar hit, at least one called gene in PharmCAT's report.json, a CYP2D6 row from pypgx, the planted ClinVar row in the HTML report with its gene, its significance and no empty cell. An exit code alone never passes a case.

Many steps ask Docker for --cpus 8, which Docker refuses on a 4-CPU machine. The job puts a small shim first on PATH (tests/e2e/bin/docker) that lowers --cpus to the CPUs the machine has and passes everything else through.

Adding a case

A case is one executable file in tests/e2e/. It sources lib.sh, runs one step, checks its output and ends with finish:

#!/usr/bin/env bash
# What this case shows, in one or two lines.
. "$(dirname "$0")/lib.sh"

run_step 11-roh-analysis.sh "$SAMPLE"
check_step_exit 11-roh-analysis.sh
check_ge "per-site (ST) lines in the ROH output" \
  "$(grep -c '^ST' "${GENOME_DIR}/${SAMPLE}/vcf/${SAMPLE}_roh.txt" || true)" 100

finish

Cases run in name order in the C locale: the numbered base cases first, then files named <package-key>-<what>.sh, which can read everything the base cases wrote. Add your own file rather than editing the runner, the workflow or another package's case. A case that reads what another case outside the base stage wrote must name it in the needs column of its row in scripts/ci/e2e-case-times.tsv, or the split may put the two in different shards. lib.sh has the helpers: check, check_eq, check_ge, has, lacks, bcf and sam (bcftools and samtools from the pinned images, paths relative to GENOME_DIR), vcf_count, planted, sample_side_hits and pharmcat_called.

A new check has to be seen failing once before it is trusted: run it against the code before your fix, or against a deliberately broken input, and link that red run in the pull request.

What no e2e case runs

This is every step script that no e2e case runs, plus the full-size index build of step 02a. The first eight rows need data, hardware or a binary an e2e job does not have. The three long-read rows need long reads, and the e2e fixture has short reads only. The others could run, but no case runs them yet. The last column says what checks each one instead. An image's smoke test is a row of tests/smoke/commands.tsv that container-test.yml runs on the fixture when the image changes and once a month; it runs the tool, not the step's script.

Step Why no e2e case runs it What checks it instead
26b (fine-grained ancestry) on the full panel the install reads about 19 GB from gnomAD's bucket case ancestry-fine-grained-1 runs it on a reduced panel of the fixture's regions plus chr22:20-35 Mb
43 (archaic ancestry) on the whole genome setup.sh --archaic reads 150 GB of archaic genomes case archaic-ancestry-1 runs it on a slice; the archaic-hg002 jobs of public-evidence.yml run setup and step 43 on HG002's 22 autosomes, by hand
44 (ancient populations) on the full AADR panel the install reads the 7.1 GB AADR release and runs smartpca on it for an hour case ancient-similarity-1 runs it on a planted panel; the ancient-footprint job of public-evidence.yml runs setup and step 44 on HG002, by hand
13 (VEP, offline) the cache is 26 GB; the fixture's VEP subset was annotated once with --database instead the image's smoke test, which annotates 50 records with --database
17 (CPSR) the PCGR bundle (about 8 GB) plus a second VEP cache. CI never runs real CPSR the image's smoke test, and the fake-docker cases for the command line
05 (AnnotSV) the annotation data is 5.3 GB the image's smoke test; the row that annotates the fixture's SV records with the full data runs monthly and on dispatch only
04b (GRIDSS) needs a 31 GB Java heap the image's smoke test
02a on GRCh38 (BWA-MEM2 index build) about 90 GB of RAM case alignment-markdup-gvcf-sex-4-bwamem2 runs step 02a on an 8.4 Mb reference cut from the fixture's regions
01 (ORA to FASTQ) needs Illumina's orad binary, which runs outside Docker nothing
02b (long-read alignment) long reads; the e2e fixture has none the minimap2 image's smoke test, which maps synthetic HiFi reads with -x map-hifi; nothing runs the script
03e (Clair3) long reads; the e2e fixture has none the image's smoke test, with a recall check on synthetic HiFi reads and a check that the two model directories the script names exist; the fake-docker suite runs the script
04c (Sniffles2) long reads; the e2e fixture has none the image's smoke test on the synthetic HiFi alignment; tests/fake-docker/bash-step-partial-outputs.sh runs the script
15 (duphold) no case yet the image's smoke test on a planted deletion; tests/fake-docker/bash-step-sv.sh runs the script; the Nextflow module is stub-run
18 (CNVpytor) no case yet; it needs its own resource files the image's smoke test; the SV merge case reads a planted CNVpytor VCF, not one CNVpytor wrote
29 (somatic Mutect2) no case yet tests/fake-docker/bash-step-somatic-mito.sh checks the Mutect2 command line
03c (Strelka2) no case yet the image's smoke test, with a recall check against the truth slice
03d (Octopus) no case yet the image's smoke test, with a recall check against the truth slice; case bash-step-bash42 runs the script under bash 4.2 against the fake docker
04a (TIDDIT) no case yet the image's smoke test; settle-doubts question 4 runs the script with only a BWA-MEM2 index; case bash-step-bash42 and tests/fake-docker/bash-step-sv.sh run it against the fake docker
chip-to-vcf (array data to VCF) on a vendor download case consumer-export-1 runs it only on the 23andMe and AncestryDNA files step 40 wrote tests/test_chip_to_vcf.sh in the unit-tests job runs the script with the real bcftools and Picard images on two ten-row vendor files

The Nextflow ExpansionHunter module is only stub-run (-profile test_all -stub in nextflow.yml); the bash step 09 runs in case bash-step-09-str.

The settle-doubts job

scripts/ci/settle-doubts.sh answers questions a review could not settle by reading the code: whether GATK accepts a BAM without a read group, which Clair3 models the image ships, what TIDDIT does with only a BWA-MEM2 index, whether fastp trimming changes DeepVariant's accuracy against the GIAB truth, and so on. It prints one table row per question to the job summary, with the command and what was observed. It is not a gate. Run it by hand with job: settle-doubts; it also runs on a pull request, and on main after a merge, that changes the script.

The public-evidence workflow

.github/workflows/public-evidence.yml runs two opt-in steps whole, on public data only, and writes what they measured to the job summary. It is not a gate; it runs by hand only (workflow_dispatch).

  • ancient-footprint: setup.sh --ancient-panel on the full AADR release, then step 44 on a gVCF built from GIAB's HG002 v4.2.1 genotypes, with the peak memory of each step's containers (one maximum per step, not per container), the peak disk growth and the time. It fails if a command fails, if the sampler saw no container memory, or if step 44 used no site. What it measured is in step 44, Runtime.
  • archaic-hg002: setup.sh --archaic and step 43 on HG002, one job per autosome; archaic-share joins them into one report per population. A job fails if a command fails or step 43 analysed no site, and the job empty is its red control: an empty region list must make setup fail. What it measured is in step 43, Runtime.

The prs-memory job

It measures the memory of the two PRS tasks that read only the scoring files: PRS_PREPARE and pgsc_calc's FORMAT_SCOREFILES on the default score list (tests/test_prs_memory.sh --measure). One leg runs them under the memory conf/base.config gives them and must pass; the other runs them under what the labels gave before and must be killed. It runs only by hand (job: prs-memory) and is not a gate. What it measured is in PRS: Memory.

Running the e2e job yourself

The e2e run needs Linux, Docker, Nextflow at NEXTFLOW_VERSION in versions.env with Java 17, the gh CLI and about 25 GB of disk, and all 130 cases took 230 minutes one after another on a 4-CPU GitHub runner (2026-11-02). Run it on a machine you do not need for anything else:

E2E_WORK=/path/with/space scripts/ci/e2e-run.sh          # every case
E2E_WORK=/path/with/space scripts/ci/e2e-run.sh '3*'     # only cases whose file name matches
scripts/ci/e2e-shards.sh plan 6                            # the split CI would run

A partial run is for debugging: later cases read what earlier cases wrote.

Retaking the report pictures

The pictures of the HTML report (step 24), the CPIC report (step 27) and the MultiQC report (step 28) in these docs show DEMO-001, an invented sample. tests/demo/retake-screenshots.sh makes them again after a change to a report:

DEMO_TERMS_FILE=~/private-terms.txt tests/demo/retake-screenshots.sh /path/to/work

It writes DEMO-001 from a fixed seed (tests/demo/make_demo_sample.py; no real sample is read), runs steps 36, 27, 24 and 28 on it with the pinned images, and stops if a section of the report has no value, so no card says "Not run". It then draws the three pictures into /path/to/work/pictures with a pinned Chromium and Pillow, and checks them: only image data in each PNG, and on macOS the OCR finds DEMO-001 and none of the terms in DEMO_TERMS_FILE (your names and host names, one regular expression per line; keep that file outside the repository). The check runs a self-test first, on planted bad pictures. Docker must see the work folder and the checkout: with Colima, keep both under the home folder or /private/tmp.

The Demo pictures workflow (.github/workflows/demo-pictures.yml) retakes them on ubuntu-latest, by hand or on a pull request that changes tests/demo/ or the workflow. It sets PLAYWRIGHT_WITH_DEPS=1, so Playwright installs Chromium's system libraries itself, and returns the pictures as the demo-pictures artifact. On Linux the check reads the PNG chunks only, because the OCR half needs macOS: run tests/demo/check_pictures.py --ocr on a Mac before you commit pictures from the artifact. The workflow writes nothing to the repository.

Look at the pictures, copy them into docs/images/, and update the alt text where the cards changed. A new report card that the generator does not feed yet stops the script with its name; add its input file to make_demo_sample.py.

Telling a flaky test from a broken one

The Repeat a test workflow (.github/workflows/repeat-test.yml) runs one tests/test_*.sh or tests/test_*.py many times, each run in a fresh job on its own ubuntu-latest runner, six at a time. Start it from the Actions tab with the test's path and the number of runs (1 to 60, default 30); it also runs, with its defaults (tests/test_annotsv_benign_mane.sh, 30 runs), on a pull request that changes the workflow. A flaky test fails a few runs out of many; a broken one fails them all. Each run records its exit code, and the summary job lists every run and fails when any failed. One more run, control, runs a test that always fails, and the summary fails when the control passed, so "0 of 30 failed" is never a count that could not see a failure.