Step 24: HTML Report

What This Does

Generates a self-contained HTML dashboard summarizing all pipeline results. Open it in any browser — no internet connection needed.

Why

The pipeline produces output across many directories in different formats (VCF, TSV, HTML, TXT). This step consolidates everything into a single visual report with color-coded status indicators, variant counts, and key findings.

Tool

bin/collect_summary.py reads every step's output once into ${SAMPLE}/summary.json, and bin/render_report.py renders the report from it. scripts/generate-report.sh renders the text report from the same summary, so the two reports cannot disagree about a count. Both are Python standard library only.

Docker Image

• PYTHON_IMAGE

Pinned in versions.env; Image versions lists the current tag.

Input

All output directories from previous pipeline steps. The script detects which steps have been run. It reads the bash layout (${GENOME_DIR}/${SAMPLE}) and the Nextflow one (--outdir/${SAMPLE}, for example roh/, coverage/, hla/, pharmcat/), so GENOME_DIR=<outdir> ./scripts/24-html-report.sh <sample> reports on a Nextflow run.

Two more files, when present:

• ${SAMPLE}/run_manifest.tsv (bin/write_manifest.sh): the pipeline commit, versions.env, the digest of every image, the ClinVar file date, the VEP cache, the PCGR and pypgx bundles, whether pgsc_calc's ancestry panel is installed, the IPD-IMGT/HLA release of the hla.dat the run used (run-all.sh passes it as HLA_DAT; without it the one setup.sh installs, then the release in the T1K index folder name, and unknown when HLA calls exist but no database is found) and the header of each PGS scoring file. run-all.sh writes it when a run starts and again just before this step, so an image pulled during the run gets its digest; this step writes one when the sample has none.

• ${SAMPLE}/logs/run_status.tsv: when the latest run-all.sh run started, then one line per step. A step it did not start is skipped (<reason>), for example skipped (no BAM) or skipped (not in TOOLS). Each step it starts gets a planned row just before the pipeline starts, which stands until the step's result row replaces it; a meta finished_utc row closes the file when run-all.sh ends, and a step still planned then becomes not finished (Where a run stands). Once the pipeline has ended, each step it started is one of:

  • ok: the pipeline finished. After a pipeline that another step stopped, a step is still ok when each of its processes has a task that completed in this run or came from Nextflow's cache (-resume), and none was left unfinished.
  • failed: one of the step's tasks failed. A report-only tool's failure is ignored by the pipeline, which goes on without it. Ancestry (step 26) is computed inside the PRS run, so it is failed when PRS is.
  • not finished: the pipeline stopped on another step's failure before this step's tasks ran or ended.
  • skipped (expansion_hunter failed): Stranger (step 9b), which reads ExpansionHunter's calls.

  After a stopped pipeline run-all.sh reads these results from ${SAMPLE}/nextflow/process_status.tsv, which the pipeline writes as it ends: one row per process with its tasks that succeeded, came from the cache, failed, were ignored and were left unfinished. A process belongs to the step whose --tools name its own name starts with (PRS_SUMMARY belongs to prs); Yleaf's processes are named Y_*, the SV merge's SURVIVOR_*, haplocheck and the chrM extraction (HAPLOCHECK, MITO_EXTRACT_CHRM) count for step 12, T1K's index build and the KIR pass (T1K_BUILD, KIR_BUILD, KIR_TYPING) count for step 08, and INDEXCOV is step 16. A process with no task had not started, so its step is not finished; the exception is haplocheck, which reads step 20's Mutect2 calls and gets no task when step 20 did not run or failed. Without that file (Nextflow died before writing it) no step is ok.

  Each opt-in step script it ran after the pipeline is ok or failed. The two reports get no line. After a failed pipeline run-all.sh skips the opt-in scripts and still writes this report and the text report.

Command

./scripts/24-html-report.sh your_name

How the report is laid out

The report is one HTML file in seven chapters, in this order, each opening with the question it answers. The text report (scripts/generate-report.sh) has the same chapters in the same order. CHAPTERS in bin/render_report.py is the one table both read.

Chapter	Question	Sections
Quality	Can I trust this data?	Quality control (steps 16b, 16, 33, 02), variant calling (03)
Clinical	Is there anything to take to a doctor?	ClinVar screen and hits (06), clinical filter and secondary findings (23), cancer predisposition (17), variant prioritization (31), symptom-driven ranking (48), structural variants (04, 19, 18, 22, 05), mobile element insertions (50), repeat expansions (09), MUC1 repeat (52), mitochondrial variants (20), Lp(a) (46), clonal haematopoiesis (49), mosaic loss of chromosome Y or X (33), mosaic chromosomal alterations (51)
Carrier	What could I pass on to a child?	Carrier list (06), couple carrier screen (39), paralog genes and SMN1 (35b, 35)
Medicines	How might I respond to medicines?	PharmCAT (07), CPIC recommendations (27), pypgx, Cyrius and CYP2D6 (32, 21), HLA typing (08)
Ancestry	Where do my ancestors come from?	Fine-grained ancestry (26b), mitochondrial haplogroup (12), Y chromosome (37, 45), archaic ancestry (43), ancient populations (44), runs of homozygosity (11)
Traits and scores	What do studies say about my traits and risk scores?	Traits (42), polygenic risk scores (25), blood groups (41), telomere content (10)
Other and exports	What else did the run find or write?	Viral DNA (47), raw-data export (40)

After them, "What this report leaves out" lists the steps not run and what no step assesses. The report closes with "Genes and rsIDs in these results": one line per gene symbol or rsID the results name, with a link to each section that names it and how many variants there (below).

• Contents. A list on the left (on top on a narrow screen) names every chapter and section shown, each section with a dot for its state: blue complete, yellow stale or part not run (a section of several steps where one did not run), red failed, gray not run. A section with no card (a step such as step 39 that shows nothing when it did not run) is left out of both.
• Each section has an anchor (#clinvar, #cpic, ...). Under its first card title a line gives the step, the state, the newest file date and a link to the section itself. The state badge is never green: a step that ran is not a finding. The caveat each card already printed (a *_NOTE sentence of bin/render_report.py) now sits right under that card's title, before the numbers. The PRS caption keeps its ancestry caveat inside its own sentence.
• Five charts show how sure each finding is, each in its section and right below that section's caveat (the PRS chart below the caption whose sentence holds the caveat). They are inline SVG drawn by bin/report_charts.py from the same summary, with no script: hover a mark for its value, click it to jump to the table or section it comes from.
• ClinVar (#clinvar): one bar per review star, 4 to 0, and a grey bar for the low-confidence calls (GQ or DP below the call floor), which the star counts leave out. Yellow and grey only, never green. With no hit, a grey frame says that no hit is not an all-clear.
• Chromosomes (#roh): chromosomes 1 to 22, X and Y of GRCh38 with the runs of homozygosity of 5 MB or more and a tick per ClinVar hit, grey for a low-confidence call. A hit on chrM or on another contig is not drawn; the line under the chart counts it. A tick for one of the first 50 hits links to the hits table, a later one to the ClinVar section.
• chrM (#mito): each heteroplasmic site by position and allele fraction, the D-loop poly-C length variants as grey squares apart from them.
• Fine-grained ancestry (#ancestry-fine): each group the tables list, its share as a dot and its bootstrap range as a line.
• PRS (#prs): the percentiles on a 0 to 100 strip, only when pgsc_calc had an ancestry panel. A score with no percentile, or one of the other sex, says "no percentile" and gets no dot. Without a panel there is no strip: a raw score has no scale, and the caption already says so.
• Links to the step's own files, relative to the report, when the summary says they exist: the PharmCAT HTML report, the CPIC recommendations per gene, the CPSR HTML report (or the cpsr/ folder that holds it) and the Exomiser ranking table. The report sits in the sample folder beside each step's folder, in the bash layout and in a Nextflow --outdir. MultiQC and the couple report are not linked: the summary does not record them, and the couple report sits in the family folder.
• Nothing is fetched. A Content-Security-Policy (default-src 'none'; img-src data:; style-src 'unsafe-inline') lets the browser load nothing but the file itself and its inline pictures, and there is no JavaScript. The only addresses off this computer are the haplogroup links (FamilyTreeDNA, YFull, the docs) and the project link in the footer, each marked "opens a website"; the FamilyTreeDNA and YFull ones add that the address carries the haplogroup. The genome viewer links go to 127.0.0.1, this computer.
• Genes and rsIDs in these results. The closing list is built by bin/results_index.py. It holds every gene symbol and rsID that a section in state complete or stale names in summary.json, by the same rule scripts/owngenome-lookup.sh uses for "in the reports" (names_of in bin/genome_lookup.py). It adds the genes of the clinical filter (step 23) and variant prioritization (step 31), whose cards only count them: from their summary tables on the bash path, from the VCFs on the Nextflow path. The two give the same genes when a variant's most severe consequence is on the transcript the report ranks first (MANE, else canonical); the step 31 table keeps the gene of the most severe consequence, so where an overlapping gene's transcript carries the worse one, the two forms name different genes. A ClinVar hit's rsIDs come from step 13's VEP output (Existing_variation), because the hits file's ID column holds ClinVar's own id; without step 13 a hit has none. The Nextflow HTML_REPORT process does not get step 13's VEP output, so its index has no ClinVar rsIDs; step 24, which run-all.sh runs over the output folder afterwards, adds them. A failed, missing or unreadable section adds nothing. The browser's find (Ctrl+F) searches the list; there is still no script. A position is not in it: ask scripts/owngenome-lookup.sh for a genotype (genotype lookup).
• Dark on screen, light on paper.
• Print one section. Open a section from the contents (or its "link to this section"), then print: only that section prints, with the header (sample, generation date, pipeline commit, ClinVar file date, ClinVar in CPSR's bundle, HLA database), the list of what is not assessed and the disclaimer. A chapter prints the same way from its title in the contents. Without an anchor in the address the whole report prints, as it does in a browser without CSS :has().

Output

${GENOME_DIR}/${SAMPLE}/${SAMPLE}_report.html, ${GENOME_DIR}/${SAMPLE}/summary.json, ${GENOME_DIR}/${SAMPLE}/${SAMPLE}_charts.json and ${GENOME_DIR}/${SAMPLE}/${SAMPLE}_search_index.json.

${SAMPLE}_search_index.json is the list the report closes with, as JSON: schema_version 1, sample, generated_utc and terms, where each term (a gene in upper case, an rsID in lower case) maps to one item per section that names it: the section key, its report anchor, title, step, state, source file, a count of mentions (a variant counts once per section) and the first 20 mentions, each with its text and locus. The Nextflow HTML_REPORT process writes the same file beside its report, without the ClinVar hits' rsIDs (see above). bin/results_index.py query --index <file> BRCA2 answers one query at a shell. The file names genes of the results, so keep it on this machine like the report. On the live site its plugin (bin/owngenome_live_search.py, assets/live/plugins/search.js) serves the file only behind the site's password. The page fetches it once, and again only when the report is written anew, then filters it in the browser as you type. No request carries the query, so it is in no log. A prefix lists the terms that start with it (BRCA gives BRCA1 and BRCA2), and a position gets a pointer to scripts/owngenome-lookup.sh.

${SAMPLE}_charts.json holds the five charts as data and as SVG (schema_version, sample, generated_utc, charts), each with its section, its caveat and one entry per mark (value, range, locus, detail). The Nextflow HTML_REPORT process writes the same file beside its report. One input differs: Nextflow also passes the samplesheet's sex, so when no run status or manifest records a sex, its PRS chart marks the sex-specific scores against that sex and step 24's does not. It holds findings, so treat it like the report: keep it on this machine. On the live site its plugin (bin/owngenome_live_charts.py, assets/live/plugins/charts.js) serves this file only behind the site's password and shows the same charts; a click on a mark opens a panel with its detail, its caveat, a viewer/#<locus> link that opens the reads once the live site serves the genome viewer, and a link to its section of the report.

The cards, by chapter (cards not listed here are named in the chapter table above):

Quality

• Quality Control — mean depth (mosdepth), sex inferred from X/Y coverage (indexcov) against the declared sex, and from step 33: the sex somalier infers from the reads, FREEMIX (VerifyBamID2's estimate of reads from another person) against its warning threshold, and any other sample that is the same person
• Variant Calling — total variants, PASS count, SNPs, indels (one pass over the VCF)

Clinical

• ClinVar Screening — hit count by review stars, the ClinVar file date, and the hits table best-reviewed first with a Stars column. The count's badge is yellow when there is a hit and gray when there is none, never green. Each hit shows its gnomAD allele frequency from step 13's VEP output, and a line says where the frequencies came from and how many hits are above 1%, or why every frequency is unknown: a common allele with a P/LP record is often low-penetrance, so read every hit with the frequency the card shows, and look up a hit whose frequency is unknown. A low-confidence line counts the hits whose call is under step 6's GQ or DP floor: they are listed but not counted in the total. When step 6's cDNA read check ran, the card also counts the hits it flagged
• Clinical Filter and slivar — variant counts from steps 23 and 31
• Secondary-Findings Genes (ACMG SF v3.3) — the step 06 ClinVar hits, the clinical filter's rare HIGH-impact records and step 05's class 4 and 5 SVs in the 84 genes of the ACMG list, each gene's reporting rule applied (assets/acmg_sf.tsv, for example HFE homozygous C282Y only), and the variants a rule does not report listed apart with the reason
• Cancer Predisposition — CPSR status and the classification breakdown (read by column name from ${SAMPLE}.cpsr.grch38.classification.tsv.gz)
• Structural Variants — Manta, Delly, CNVpytor and the SURVIVOR consensus (2+ callers) counts; then step 05's class 4 and 5 SVs (AnnotSV) with their caller support, exon-edge and frequency checks. A call step 05's rules moved to class 3 (a coding tandem repeat, an immunoglobulin or T-cell receptor locus, the MHC, a pseudogene named by a disease gene's alias, or read depth that does not show the change) is listed apart with AnnotSV's class, the rule and the reason, and the class 4-5 count leaves it out
• Repeat Expansions — a table of every locus, the five key loci (HTT, FMR1, C9orf72, ATXN1, DMPK) first, with its repeat count, Stranger's status and notes; and counts of the loci outside the normal range (step 9b, Stranger), of those short reads could not size (not counted as normal), and the loci in an intermediate range (bin/str_intermediate_ranges.tsv). When Stranger flags RFC1, or another locus where the motif decides (expansion_hunter/${SAMPLE}_str_motifs.tsv), the card adds the verdict of the read-level motif count (step 9b). A locus the repeat catalog gives no thresholds (its NoThresholds field in expansion_hunter/${SAMPLE}_eh_catalog.json: MUC1, CEL, TYMS and PLIN4 with the STRchive catalog, among others) is marked not graded, with the catalog's reason, never normal. Any other locus outside Stranger's catalog has no status and is counted apart. Without step 9b the card says Stranger did not run; a Stranger file older than the ExpansionHunter result it came from is not read, and the card says to rerun step 9b. Repeat-expansion calls from short reads can be wrong at some loci
• Mitochondrial — chrM PASS variants and heteroplasmic calls (allele fraction 0.05 to 0.95; below 5% NUMT reads and noise dominate)
• Mosaic Loss of Chromosome Y or X — step 33's call on chrY (men) or chrX (women), the reason for it, and a note that this is a population-level association, not a diagnosis

Carrier

• Carrier List (this sample) — the step 06 ClinVar hits in the recessive and X-linked genes of assets/carrier_genes.tsv, for this sample alone; hypomorph and pseudodeficiency alleles set aside
• Couple Carrier Screen — when step 39 ran for this sample: the partner, the genes where both carry a change, the X-linked findings and the genes where only one partner does, each shared or X-linked gene with the chance for a child
• Paralog Genes (Paraphase) — when step 35b ran: copy number and call per gene, SMN1 from Paraphase beside Parascopy, with whether they agree, when step 35 ran too, and the count of variants on phased haplotypes
• SMN1/SMN2 Copy Number (Parascopy) — when step 35 ran: SMN1 and SMN2 copies at exon 7 with quality and filter, the two together (agCN), the SMA reading, and, when step 35 wrote its HBA table, the count of alpha-globin genes from depth or the reason it was not assessed
• Paralog Copy Number — not checked, gray, when neither step 35 nor step 35b left a result, naming a step that failed

Medicines

• Pharmacogenomics — PharmCAT version and PharmCAT/pypgx conflicts
• CPIC Drug Recommendations — genes with a non-normal phenotype, with more than one possible result, not called, called without a function phenotype (unclassified, no drug guidance), and a table of the non-normal genes
• CYP2D6 Across Callers — pypgx and Cyrius side by side, judged by step 36's rule: a no-call such as Indeterminate is not a call, and a Cyrius genotype counts only with Filter PASS (otherwise it is shown as not usable, with its Filter). The card says whether the two agree and, with step 36's table, its verdict and reason. The PharmCAT row is the call step 36 passed to PharmCAT, not a third caller: PharmCAT calls no CYP2D6 from a VCF
• HLA Typing — T1K alleles per locus, each with its own quality, and the IPD-IMGT/HLA release. A quality of 0 or below is marked low confidence; on HLA-A and HLA-B the card adds that the gene is not passed to PharmCAT. HLA typing from short-read WGS is approximate

Ancestry

• Fine-Grained Ancestry — when step 26b ran: the continental and regional fractions with their bootstrap ranges, and the chromosome painting
• Mitochondrial Haplogroup — haplogrep3's call, and haplocheck's contamination status ("not checked" when step 12 read the step 03 VCF instead of step 20's calls)
• Y-Chromosome Haplogroup — Yleaf's call, its marker count and QC-score, or "insufficient markers" (step 37, when it ran)
• Runs of Homozygosity — total, largest segment, segments, autosomal runs over 5 MB

Traits and scores

• Traits — step 42's panel: each row read with its genotype, reading and PubMed ids, the rows with no genotype, the genotype rules that read several sites together (traits/${SAMPLE}_genotype_rules.tsv, today the TAS2R38 taster haplotypes) with their result and how they were phased, and the APOE lines folded away behind a warning; "Not run" without step 42
• Polygenic Risk Scores — each score with its percentile among the most similar reference group, or "raw score only" without the ancestry panel. The caption says that most PRS scores were developed in European-ancestry populations and predict less well for other ancestries, even with an ancestry-adjusted percentile
• Blood Groups (predicted) — when step 41 ran: ABO, RhD, and per blood group system the predicted phenotype, genotype and confidence
• Telomere content (relative) — TelomereHunter's tel_content: intratelomeric reads per million reads with 48-52% GC. It is a relative measure, not a telomere length

Other and exports

• Raw-Data Export (23andMe v5 layout) — when step 40 ran: the sites with a genotype out of all the sites, the homozygous-reference sites taken from the gVCF's reference blocks, why the others are --, and the names of the two zips under consumer_export/

What this report leaves out

• Steps Not Run and Not Assessed by This Pipeline

The footer holds the run manifest.

Results from an earlier run

A step that is skipped or fails leaves the previous run's output on disk. When logs/run_status.tsv exists, a section whose file is older than the latest run-all.sh run, and whose step was not ok in that run, is marked STALE with the file's date and the step's result, in both reports. One failed tool does not make the other results stale. After a run that a tool stopped, a step whose tasks completed, or came from Nextflow's cache because its inputs and code had not changed, is ok: its section is shown as current even when its file is older than the run. Only the steps the stop cut short are not finished (the failed one failed), and an older file of theirs is marked stale until the same command, with -resume, completes. A step you re-ran by hand after that run writes a newer file and is shown as current. With KIR typing on, a KIR task that failed or was left unfinished marks step 08 failed or not finished, so the HLA section of that run is marked stale too, even when the HLA call came from the cache: HLA and KIR are one step. Variant calling (step 03) has no line in logs/run_status.tsv, so its section is never marked stale, even when its VCF is older than the run.

A section with no file whose step is recorded failed in logs/run_status.tsv says Failed, with the step, in both reports (the CYP2D6 rows too), and is listed under Steps Not Run with "(step N failed)". A section whose step never ran still says Not run.

The report for DEMO-001, an invented sample. Every number is made up. The picture stops above the ClinVar hits table, the first table that names genes, so the rest of the chapters and the disclaimer are not shown.

What changed since the previous run

scripts/generate-report.sh, which run-all.sh runs after this step, keeps each run's summary.json and run_manifest.tsv under ${SAMPLE}/history/ and writes ${SAMPLE}_delta.txt, .html and .json: the inputs and results that changed since the previous run of the same sample, such as a reclassified ClinVar hit, a new image digest or a moved PRS percentile. What changed since the previous run says what it compares.

Runtime

Not measured; run-all.sh runs it after the pipeline, outside Nextflow's trace. An estimate is about a minute, most of it reading the whole-genome VCF.

How to Open

# macOS
open ${GENOME_DIR}/${SAMPLE}/${SAMPLE}_report.html

# Linux
xdg-open ${GENOME_DIR}/${SAMPLE}/${SAMPLE}_report.html

# Windows (WSL)
start ${GENOME_DIR}/${SAMPLE}/${SAMPLE}_report.html

Notes

• The report is completely self-contained — all CSS is inline, no external dependencies
• Works offline in any modern browser
• Responsive layout (works on mobile/tablet)
• Steps that were not run show "Not run" and are listed under Steps Not Run — this is expected. A step recorded as failed shows "Failed" instead
• The report contains health findings: the ClinVar table lists your pathogenic and likely pathogenic variants, and other sections summarise pharmacogenomics, CPSR and the other steps. It holds no raw reads or full VCF, but share it only as you would share a medical record
• Re-run this script anytime to update the report after running additional steps
• The Nextflow HTML_REPORT module runs the same bin/render_report.py on the outputs of its run's QC, ClinVar, PharmCAT, CPIC, CPSR, clinical filter, slivar, ROH and mitochondrial haplogroup steps, so both reports print the same numbers for the same file. Its clinical filter and slivar cards show counts only. For every section on a Nextflow run, run this script with GENOME_DIR set to the --outdir