Troubleshooting Guide¶

Definitive reference for diagnosing and fixing problems with OwnGenome. Organized by symptom so you can Ctrl+F your error message and find the fix.

Before troubleshooting: Run docker info to confirm Docker is running, and check docker stats for resource usage. Most failures are either Docker resource limits or wrong file paths inside containers.

Docker Issues¶

Container exits immediately with no output¶

Symptom: docker run returns instantly. No error message. Exit code 137.

Cause: The container was OOM-killed (Out Of Memory). Docker silently kills containers that exceed their --memory limit. Exit code 137 = SIGKILL from the kernel OOM killer.

How to confirm:

# Check the last container's exit status
docker ps -a --latest --format "{{.Status}}"
# Output like "Exited (137)" confirms OOM

# Check system logs for OOM events
dmesg | grep -i "oom\|killed" | tail -10

# On Docker Desktop (Mac/Windows)
# Check Docker Desktop > Troubleshoot > Logs

Fix: 1. Increase the --memory flag in the script that failed (for step 3, DV_MEM). With run-all.sh you do not edit a script: the pipeline gives each task a memory request and doubles it on the one retry after an out-of-memory exit, up to --max_memory 2. Bound what runs at once. With run-all.sh, --max_memory (e.g. ./scripts/run-all.sh <sample> <sex> --max_memory 24.GB; without it, the machine's RAM) and THREADS=N (--max_cpus, by default the machine's CPU count) cap each task, not the total: Nextflow starts tasks until their requests fill the machine's CPUs and RAM, and on a Mac with Docker Desktop that is the Mac's, not the VM's. To bound the total, pass a config with an executor block (a shared host); on a Mac, also pass THREADS and --max_memory no larger than the VM's (e.g. THREADS=6 ... --max_memory 14.GB); a rerun reuses the steps that finished 3. Increase Docker Desktop memory allocation (see Docker Desktop not enough memory) 4. For DeepVariant run as a script, lower THREADS (its shard count) and DV_MEM together, e.g. THREADS=4 DV_MEM=16g ./scripts/03-deepvariant.sh <sample> <sex>

Container exits with code 1 but no error message¶

Symptom: Script fails, exit code 1, but the container printed nothing useful.

Cause: Most bioinformatics tools print errors to stderr, which may not be captured depending on how you ran the script.

Fix:

source versions.env   # from the repository root
# Re-run with full debug output
bash -x scripts/03-deepvariant.sh your_name 2>&1 | tee debug.log

# Or run the docker command manually with -it for interactive output
docker run --rm -it \
  --cpus 4 --memory 8g \
  -v "${GENOME_DIR}:/genome" \
  "${DEEPVARIANT_IMAGE}" \
  /bin/bash
# Then run the command inside the container to see the error

"No such image" / "manifest for ... not found"¶

Symptom: Error response from daemon: manifest for quay.io/biocontainers/toolname:tag not found

Cause: Biocontainer image tags change frequently. The hash suffix (e.g., --hd03093a_1) encodes the conda build environment and breaks between releases.

Fix: 1. Check the current tag at the container registry:

# For quay.io images
# Visit: https://quay.io/repository/biocontainers/TOOLNAME?tab=tags

# For Docker Hub images
docker search toolname

2. Image names that have caught people out, and what the pipeline uses instead (the current tags are on Image versions):

Tool	Wrong image	Use instead
Delly	an old `delly` 1.2.9 build tag	`DELLY_IMAGE`
ExpansionHunter	`weisburd/expansionhunter` (v2.5.5, a different CLI)	`EXPANSIONHUNTER_IMAGE`
AnnotSV	`bioinfochrustrasbourg/annotsv`	`ANNOTSV_IMAGE`
CPSR	`sigven/cpsr` (does not exist)	`PCGR_IMAGE` (bundles both)
MToolBox	`robertopreste/mtoolbox` (does not exist)	`GATK_IMAGE` (step 20 uses Mutect2)

If an image disappears entirely, check if the tool has an official Docker image on GitHub Container Registry (ghcr.io), Docker Hub, or the tool's documentation.

Images that were pulled are missing after a Docker cleanup¶

Symptom: setup.sh --pull-only pulled every image, but later validate-setup.sh reports Docker images not yet pulled, or a task fails because its image is not there. The missing ones are pinned by digest in versions.env (name:tag@sha256:...).

Cause: on Docker's classic image store an image pulled by digest has no local tag, so Docker lists it as dangling (docker images --filter dangling=true). A dangling-image cleanup (docker image prune, or docker rmi $(docker images -qf dangling=true)) deletes it unless a container is using it. We saw this once, on a host whose own cleanup job removes dangling images. On the containerd image store (the default for fresh installs since Docker Engine 29) the digest reference is the image's name, so Docker does not list it as dangling and that cleanup keeps it. docker info names the store: the containerd one shows io.containerd.snapshotter.v1 under the storage driver.

Fix: run ./scripts/setup.sh --pull-only. It pulls what is missing and gives each digest-pinned image a local tag, localhost/pgp-pinned/<name>:sha256-<digest>. A tagged image is not dangling, so those cleanups keep it. The tag is only a second name for the same image: every task still runs the digest reference from versions.env. It does not protect against docker system prune -a or a docker rmi of the image. setup.sh ends with a line that says how many of these tags it added (0 pgp-pinned tag(s) added when they were all there). validate-setup.sh warns when a digest-pinned image lacks that tag and Docker lists it as dangling, the case a cleanup deletes. When Docker does not list it as dangling (another local tag names it, or the containerd store keeps the digest reference as its name), validate-setup.sh only notes that the tag is missing: Docker does not list the image as dangling, so a dangling-image cleanup keeps it. Another tag keeps it only while that tag stays. If Docker does not answer which images are dangling, it warns that a cleanup may delete the image. The same command fixes all three.

"Permission denied" writing to mounted volumes¶

Symptom: OSError: [Errno 13] Permission denied: '/genome/sample/output' or similar.

Cause: Most bioinformatics Docker images run as a non-root user (e.g., UID 1000). When writing to bind-mounted host directories, the container user may not have write access.

Fix: Add --user root to the docker run command. All pipeline scripts already include this flag, but if you are running commands manually:

docker run --rm --user root \
  -v "${GENOME_DIR}:/genome" \
  your-image:tag \
  your-command

Note: Files created with --user root will be owned by root on the host. To fix ownership afterward:

sudo chown -R $(whoami) ${GENOME_DIR}/${SAMPLE}/

Docker Desktop not enough memory (Mac/Windows)¶

Symptom: Containers fail with exit code 137 even though your machine has enough RAM.

Cause: Docker Desktop runs inside a virtual machine with a fixed memory allocation. Default is often 2-4 GB, which is far too little for genomics.

Fix for macOS: 1. Open Docker Desktop > Settings > Resources 2. Set Memory to at least 16 GB (32 GB recommended) 3. Set Swap to at least 4 GB 4. Click Apply & Restart

Fix for Windows (WSL2): Create or edit %UserProfile%\.wslconfig:

[wsl2]
memory=24GB
swap=8GB
processors=8

Then restart WSL: wsl --shutdown and reopen your terminal.

Verification:

docker info | grep "Total Memory"
# Should show close to what you configured

Containers stuck / running forever¶

Symptom: A container has been running for much longer than expected. No output. Not using CPU.

How to diagnose:

# Check what's running
docker ps

# Check resource usage — is the container actually doing work?
docker stats --no-stream

# Check container logs
docker logs <container_id>

# If CPU is at 0% for a long time, the container may be stuck

Common causes: 1. Waiting for stdin: Some tools expect interactive input. Fix: add -i flag or ensure all parameters are passed on the command line. 2. I/O blocked on slow storage: NFS/SMB/USB mounts can stall Docker I/O. Check docker stats — if CPU is 0% and memory is stable, the container is likely waiting for I/O. 3. Java heap space: Tools like PharmCAT or GATK may be stuck in garbage collection. Fix: increase --memory.

Fix:

# Kill the stuck container
docker kill <container_id>

# Or if you started with run-all.sh, stop Nextflow with Ctrl-C: it stops its tasks'
# containers. The same command later reruns only the tasks that did not finish.

The run is silent and no container is running¶

Symptom: run-all.sh (or nextflow run) prints nothing new, docker ps lists no nxf-... container, the machine is idle, and the run does not end.

How to diagnose: Nextflow writes its two task queues to .nextflow.log every five minutes. run-all.sh keeps that log in $GENOME_DIR/<sample>/nextflow/; a direct nextflow run leaves it in the directory it was started from.

# After run-all.sh. After a direct `nextflow run`, give the path of .nextflow.log in its launch directory instead.
grep -E '(%%|!!) executor|^~> ' "$GENOME_DIR/<sample>/nextflow/.nextflow.log" | tail -n 20

!! executor local > No more task to compute next to %% executor local > tasks in the submission queue: 2 (the number is how many wait), with the same tasks listed each time, means tasks are waiting to start while nothing is running. A line tasks to be completed means a task is still running. That is the entry above, not this one.

Cause: Nextflow before 26.10.0 can lose track of the CPUs and memory it has free (Lessons learned). After that it believes less is free than there is, and a task that asks for more than that never starts. The tasks most exposed ask for all, or nearly all, of what the machine has: run-all.sh caps a task at the machine's CPU count and RAM, so that is every 8-CPU task on a machine with 8 CPUs or fewer, and DeepVariant and the alignment (32 GB each) on a 32 GB machine. We have seen this in the CI stub runs only, never in a real run. Nextflow 26.10.0, the release versions.env pins, fixes the race (nextflow-io/nextflow#7816), so this applies when you run an older release. On 26.10.0 or later, a run in this state has another cause: keep the log and open an issue.

Fix:

Stop Nextflow with Ctrl-C and run the same command again, on Nextflow 26.10.0 if you can (run-all.sh sets NXF_VER from versions.env). A new Nextflow process counts from the start, and -resume (which run-all.sh passes) reuses the tasks that finished. This follows from the cause; we have not had to do it on a real run.
To make it less likely, keep each task below the whole machine: THREADS below the CPU count and --max_memory below the RAM (e.g. THREADS=6 ./scripts/run-all.sh <sample> <sex> --max_memory 24.GB on 8 CPUs and 32 GB). This leaves a margin for a lost count; it does not remove the cause.

The work disk is filling¶

Symptom: A long run is still going and free space on the disk of the work folder (<sample>/nextflow/work with run-all.sh) or of GENOME_DIR keeps dropping.

Check: bin/run_headroom.py gives the used share of both disks, and warns at 90% (--disk-warn-percent or OWNGENOME_DISK_WARN_PERCENT to change it):

python3 bin/run_headroom.py --work-dir "${GENOME_DIR:?Set GENOME_DIR to your data directory}/<sample>/nextflow/work" --outdir "$GENOME_DIR"

A line disk work and results means both folders sit on one filesystem. The inode share counts too: a disk can run out of inodes before it runs out of bytes. Where a run stands says what each figure means.

Fix: free space before the disk fills, with the steps of the next entry. Deleting the work folder while the run goes breaks the run; delete only files of finished steps outside it. To move the work folder to a larger disk for the next run, pass Nextflow's -work-dir (or -w).

"No space left on device" mid-analysis¶

Symptom: Container crashes with "No space left on device", "ENOSPC", or "write failed: No space left".

Cause: Docker needs free space in TWO places: 1. Your data directory (GENOME_DIR) for output files 2. Docker's storage driver (usually /var/lib/docker) for container layers and temp files

Diagnosis:

# Check host filesystem
df -h ${GENOME_DIR}
df -h /var/lib/docker   # Linux
# Docker Desktop stores data differently — check Docker Desktop settings

# Check Docker disk usage
docker system df

Fix:

# Clean up Docker (removes unused containers, images, build cache)
docker system prune -a
# WARNING: This removes ALL unused images. Re-pull the default images with
# ./scripts/setup.sh --pull-only; the opt-in images come back with
# --parascopy-data or --yleaf-data.

# Safer: only remove stopped containers and dangling images. Run
# ./scripts/setup.sh --pull-only once first: it tags the digest-pinned
# images, which are dangling otherwise on Docker's classic image store and
# would be deleted too.
docker container prune
docker image prune

# Delete intermediate files from completed steps
rm -f ${GENOME_DIR}/${SAMPLE}/cnvpytor/*.pytor      # a few GB each
rm -f ${GENOME_DIR}/${SAMPLE}/delly/*.bcf            # After VCF conversion
rm -rf ${GENOME_DIR}/${SAMPLE}/nextflow/work         # After a good run-all.sh run; the next run starts from scratch

Space requirements per sample:

Phase	Space Needed	Cumulative
FASTQ input	60-90 GB	90 GB
BAM (step 2)	80-120 GB	210 GB
VCF + analyses	10-30 GB	240 GB
VEP output	2-5 GB	245 GB
CNVpytor .pytor file	2-5 GB	250 GB
Recommended free, from a BAM		500 GB
`run-all.sh` from FASTQ, in `<sample>/nextflow/work`: trimmed reads	60-90 GB	340 GB
... a second copy of the BAM	80-120 GB	460 GB (estimate)
Recommended free, from FASTQ		700 GB

The work/ rows stay until you delete <sample>/nextflow/work; Hardware and storage requirements explains them. The alignment writes no SAM file. The cumulative figures are sums of the rows, not measurements, and leave out the temporary files samtools sort writes during the alignment (not measured), which the 700 GB leaves room for.

Input Data Issues¶

Wrong genome build (hg19 vs hg38)¶

Symptom: Variant calling produces far fewer variants than expected (e.g., 100K instead of 4-5 million). Or ClinVar screen finds 0 hits.

How to detect:

source versions.env   # from the repository root
# Check BAM header for chromosome naming and lengths
docker run --rm -v "${GENOME_DIR}:/genome" "${SAMTOOLS_IMAGE}" \
  samtools view -H /genome/${SAMPLE}/aligned/${SAMPLE}_sorted.bam | grep "^@SQ" | head -3

# GRCh38 (hg38): SN:chr1  LN:248956422
# GRCh37 (hg19): SN:1     LN:249250621   (or SN:chr1 LN:249250621)

# For VCF files
docker run --rm -v "${GENOME_DIR}:/genome" "${BCFTOOLS_IMAGE}" \
  bcftools view -h /genome/${SAMPLE}/vcf/${SAMPLE}.vcf.gz | grep "^##contig" | head -3

Key differences:

Feature	GRCh37 / hg19	GRCh38 / hg38
chr1 length	249,250,621	248,956,422
Chromosome prefix	Often no `chr`	`chr` in UCSC-style files (what the pipeline needs); none in Ensembl-style files
Mitochondria name	`MT` (or `chrM`)	`chrM` (UCSC style) or `MT` (Ensembl style)
ALT contigs	No	Yes in the full assembly; the pipeline's reference (the no-ALT analysis set) leaves them out

The prefix alone does not tell the build: some providers deliver GRCh38 with Ensembl names (1, MT). Check chr1's length. A GRCh38 VCF with Ensembl names needs only a rename, which the Nextflow pipeline prints when it stops on it: see Starting from a Vendor VCF.

The Nextflow run stops with is not on GRCh38. Its header gives no contig lengths: the VCF's header has no ##contig length for chr1-22, chrX or chrY, so VCF_PRECHECK compared the REF bases of its first 1000 records with --reference, and more than 5% of them differ. The message gives the count and the rate, for example they differ in 45 of the first 60 records (75.0%). The VCF is on another build, or --reference is not the GRCh38 FASTA. Renaming the contigs does not help: get a GRCh38 VCF from your provider, or start from the FASTQ. A header that does give lengths stops with Its header gives ... contig length(s) that GRCh38 does not have instead.

Fix (GRCh37 data): Extract FASTQ from BAM and re-align to GRCh38:

source versions.env   # from the repository root
docker run --rm -v ${GENOME_DIR}:/genome "${SAMTOOLS_IMAGE}" \
  bash -c "samtools sort -n /genome/${SAMPLE}/aligned/old_hg19.bam | \
           samtools fastq -1 /genome/${SAMPLE}/fastq/${SAMPLE}_R1.fastq.gz \
                          -2 /genome/${SAMPLE}/fastq/${SAMPLE}_R2.fastq.gz -"

# Then run alignment (step 2) which uses GRCh38
./scripts/02-alignment.sh $SAMPLE

Do not use LiftOver on BAM files. Re-alignment from FASTQ is cleaner and avoids coordinate translation artifacts.

CRAM files instead of BAM¶

Symptom: Pipeline scripts expect .bam files but you have .cram files.

Fix: Convert CRAM to BAM (requires the reference genome used for encoding, which is usually GRCh38):

source versions.env   # from the repository root
REF_FASTA=reference/GRCh38_no_alt_analysis_set.fasta   # see 00-reference-setup.md#the-reference-path-on-every-page
docker run --rm --user root \
  -v ${GENOME_DIR}:/genome \
  "${SAMTOOLS_IMAGE}" \
  samtools view -b \
    -T "/genome/${REF_FASTA}" \
    -o /genome/${SAMPLE}/aligned/${SAMPLE}_sorted.bam \
    /genome/${SAMPLE}/aligned/${SAMPLE}.cram

docker run --rm --user root \
  -v ${GENOME_DIR}:/genome \
  "${SAMTOOLS_IMAGE}" \
  samtools index /genome/${SAMPLE}/aligned/${SAMPLE}_sorted.bam

If you get an error about mismatched references: The CRAM was encoded against a different reference build. You need the exact FASTA used during encoding, or extract FASTQ and re-align:

source versions.env   # from the repository root
docker run --rm -v ${GENOME_DIR}:/genome "${SAMTOOLS_IMAGE}" \
  bash -c "samtools sort -n /genome/${SAMPLE}/aligned/${SAMPLE}.cram | \
           samtools fastq -1 /genome/${SAMPLE}/fastq/${SAMPLE}_R1.fastq.gz \
                          -2 /genome/${SAMPLE}/fastq/${SAMPLE}_R2.fastq.gz -"

Paired FASTQ naming issues¶

Symptom: Step 2 (alignment) fails with "ERROR: File not found" for R1 or R2.

Cause: The pipeline expects FASTQ files named exactly:

${GENOME_DIR}/${SAMPLE}/fastq/${SAMPLE}_R1.fastq.gz
${GENOME_DIR}/${SAMPLE}/fastq/${SAMPLE}_R2.fastq.gz

Vendors use different naming conventions:

# Illumina standard
Sample_S1_L001_R1_001.fastq.gz / Sample_S1_L001_R2_001.fastq.gz

# BGI/DNBSEQ
V350012345_L01_1.fq.gz / V350012345_L01_2.fq.gz

# Other
sample.1.fastq.gz / sample.2.fastq.gz

Fix: Create symlinks or rename to match the expected pattern:

cd ${GENOME_DIR}/${SAMPLE}/fastq/

# If you have multiple lanes, concatenate first:
cat *_R1_*.fastq.gz > ${SAMPLE}_R1.fastq.gz
cat *_R2_*.fastq.gz > ${SAMPLE}_R2.fastq.gz

# Or create symlinks for single files:
ln -s original_name_R1.fastq.gz ${SAMPLE}_R1.fastq.gz
ln -s original_name_R2.fastq.gz ${SAMPLE}_R2.fastq.gz

Warning: If concatenating multi-lane FASTQ, ensure R1 and R2 files are concatenated in the same lane order. Mismatched pairs will produce corrupt alignments.

Corrupt or truncated downloads¶

Symptom: Tools crash with "unexpected end of file", "truncated file", or "not in gzip format".

How to detect:

source versions.env   # from the repository root
# Test gzip integrity (for FASTQ.gz, VCF.gz)
gzip -t ${GENOME_DIR}/${SAMPLE}/fastq/${SAMPLE}_R1.fastq.gz
# If corrupt: "unexpected end of file" or "invalid compressed data"

# Test BAM integrity
docker run --rm -v ${GENOME_DIR}:/genome "${SAMTOOLS_IMAGE}" \
  samtools quickcheck /genome/${SAMPLE}/aligned/${SAMPLE}_sorted.bam
# No output = OK. Error message = corrupt.

# Check VCF can be read
docker run --rm -v ${GENOME_DIR}:/genome "${BCFTOOLS_IMAGE}" \
  bcftools view -h /genome/${SAMPLE}/vcf/${SAMPLE}.vcf.gz > /dev/null

Fix: Re-download the file. For large files, use wget -c (supports resume):

wget -c https://url/to/your/data.fastq.gz

For reference data downloads that fail repeatedly, see the per-step sections for VEP cache and PCGR data bundle.

The BAM lost reads¶

Symptom: Step 02 or 02a, or the pipeline's ALIGN_MINIMAP2 task, stops with a line like:

ERROR: the BAM holds 24000 primary records, but 32000 reads went in (fastq/S_R1.fastq.gz 20000 records + fastq/S_R2.fastq.gz 12000 records): 8000 reads (25.0%) are missing.

Or validate-setup.sh prints [FAIL] BAM read count: ... primary records, but fastp wrote ... reads for an existing BAM with fastp's report (fastq_trimmed/<sample>_fastp.json) beside it.

Why it is checked: the aligner writes every read it is given once, mapped or not, and fixmate, sort and markdup drop none, so the BAM's primary records must equal the FASTQ records of R1 plus R2 (Read count). A BAM with fewer passes samtools quickcheck and every later step runs on it, but depth, copy number (CYP2D6), CNV, telomere content and every call that depends on coverage are off.

Common cause: R1 and R2 hold different numbers of records, for example one of them was copied or downloaded only in part, or the lanes were concatenated differently. minimap2 and bwa-mem2 stop at the end of the shorter file and only warn (query files have different number of records; extra records skipped from minimap2, the 2nd file has fewer sequences from bwa-mem2). The error line shows the records of each file.

How to check:

for r in R1 R2; do
  f=${GENOME_DIR}/${SAMPLE}/fastq/${SAMPLE}_${r}.fastq.gz
  gzip -t "$f" && echo "${r}: $(( $(gzip -cd "$f" | wc -l) / 4 )) records"
done

Fix: get both FASTQ files again, complete, and run step 02 again. For a BAM that validate-setup.sh fails, realign with step 02. If that BAM was aligned from the untrimmed reads (FASTQ_SUBDIR=fastq), fastp's count does not apply to it, but run-all.sh still stops on the [FAIL]. Realigning with step 02 writes qc/<sample>_read_count.tsv with input_source fastq/..., and validate-setup.sh then skips the fastp comparison with an [INFO] line. To go on without realigning, use READ_COUNT_CHECK=warn (below).

Going on anyway: for an input you know is odd, READ_COUNT_CHECK=warn makes steps 02 and 02a keep the BAM with a warning, and validate-setup.sh print a [WARN] line instead of the [FAIL]. For the Nextflow pipeline (run-all.sh or nextflow run), put this in a file and pass it with -c warn.config (run-all.sh passes options after the sex on to nextflow run):

process {
    withName: 'ALIGN_MINIMAP2' { ext.read_count_check = 'warn' }
}

The table qc/<sample>_read_count.tsv then says mismatch, and the text and HTML reports of the bash steps show the missing reads in red.

BAM not sorted or not indexed¶

Symptom: Tools fail with "file is not sorted", "index file not found", or "[E::hts_idx_load3] Could not load index".

How to detect:

source versions.env   # from the repository root
# Check if BAM is sorted
docker run --rm -v ${GENOME_DIR}:/genome "${SAMTOOLS_IMAGE}" \
  samtools view -H /genome/${SAMPLE}/aligned/${SAMPLE}_sorted.bam | grep "^@HD"
# Should show: SO:coordinate

# Check if index exists
ls -la ${GENOME_DIR}/${SAMPLE}/aligned/${SAMPLE}_sorted.bam.bai

Fix:

source versions.env   # from the repository root
# Sort the BAM (if not already sorted)
docker run --rm --user root \
  --cpus 4 --memory 8g \
  -v ${GENOME_DIR}:/genome \
  "${SAMTOOLS_IMAGE}" \
  samtools sort -@ 4 \
    -o /genome/${SAMPLE}/aligned/${SAMPLE}_sorted.bam \
    /genome/${SAMPLE}/aligned/${SAMPLE}_unsorted.bam

# Create index (always needed)
docker run --rm --user root \
  -v ${GENOME_DIR}:/genome \
  "${SAMTOOLS_IMAGE}" \
  samtools index /genome/${SAMPLE}/aligned/${SAMPLE}_sorted.bam

Note: The BAM index file must have the exact same base name as the BAM. If your BAM is sample_sorted.bam, the index must be sample_sorted.bam.bai (not sample_sorted.bai).

Per-Step Troubleshooting¶

Step 2: Alignment (minimap2 + samtools)¶

Problem: "Cannot find FASTQ" but files exist The script looks for ${SAMPLE}_R1.fastq.gz and ${SAMPLE}_R2.fastq.gz. See Paired FASTQ naming issues.

Problem: Alignment produces 0-byte BAM Usually means minimap2 failed silently. Run with bash -x to see the actual error. Common causes: - Wrong reference genome path - FASTQ files are corrupt (run gzip -t on both) - Docker --memory too low (minimap2 with the sr index needs about 20 GB on GRCh38, see step 2)

Problem: minimap2 index build takes forever The .mmi index build is a one-time step (1-3 minutes in the measured runs, runtime). If it seems stuck, check that the reference FASTA is not corrupt and the output path is writable.

Step 3: DeepVariant and a GPU¶

Symptom: You want to use a GPU to speed DeepVariant up.

What the pipeline does. Neither entry point has a GPU option. run-all.sh and scripts/03-deepvariant.sh both run the CPU image, DEEPVARIANT_IMAGE.

What a GPU could save: 1. Only call_variants can use a GPU; make_examples and postprocess_variants always run on the CPU. In the slowest measured run, 13.5 hours on 8 CPUs, call_variants took about 3 hours, so that is the most a GPU could save there. 2. Upstream measured a 2.5x speed-up of call_variants with one P100 GPU and 8 CPUs, against its 96-core case study (DeepVariant details). On those 96 cores, call_variants is 16 of 69 minutes (DeepVariant 1.10 metrics). 3. We have not measured a GPU run.

Running it yourself. The GPU image is the same tag as DEEPVARIANT_IMAGE with -gpu appended. It needs the NVIDIA container runtime and a driver that supports the CUDA version that image was built against; check the release notes of the DeepVariant version you run.

Step 3: DeepVariant crashes on Mac (amd64 emulation)¶

Symptom: DeepVariant container exits with code 137 or crashes with memory errors on Apple Silicon Mac.

Cause: DeepVariant is an amd64-only image. On Apple Silicon Macs, Docker runs it through Rosetta 2 emulation, which adds ~2x memory overhead and 3-5x CPU overhead. The default --memory 32g and --cpus 8 settings are too aggressive for emulation.

Fix: run step 3 by its script with fewer shards and less memory. THREADS sets the container's CPUs and DeepVariant's --num_shards, and DV_MEM its memory; the script still writes the gVCF and, for a male sample, calls chrX and chrY haploid outside the PARs:

THREADS=2 DV_MEM=12g ./scripts/03-deepvariant.sh your_sample male   # or female

Expected runtime on Mac: not measured. With 8 shards on Linux DeepVariant took 8 h 50 min to 13 h 30 min in three measured 30-40x runs; 2 emulated CPUs take much longer.

Alternative: If DeepVariant repeatedly crashes on your Mac, run ONLY step 3 on a cloud Linux instance (e.g., Hetzner CCX33 for ~$0.18/hr) and download the VCF. All other steps can proceed on Mac.

Step 4: Manta produces no variants¶

Symptom: diploidSV.vcf.gz is empty or has only header lines.

Common causes: 1. BAM has too few reads (low-coverage WGS below ~10X) 2. BAM was not indexed (Manta requires .bai) 3. Reference FASTA mismatch — chromosome names must match between BAM and FASTA

Diagnosis:

source versions.env   # from the repository root
# Check BAM read count
docker run --rm -v ${GENOME_DIR}:/genome "${SAMTOOLS_IMAGE}" \
  samtools flagstat /genome/${SAMPLE}/aligned/${SAMPLE}_sorted.bam
# "mapped" count should be 500M+ for 30X WGS

# Check chromosome naming consistency
docker run --rm -v ${GENOME_DIR}:/genome "${SAMTOOLS_IMAGE}" \
  samtools view -H /genome/${SAMPLE}/aligned/${SAMPLE}_sorted.bam | grep "^@SQ" | head -1
# Must use "chr" prefix (SN:chr1) to match GRCh38 reference

Step 5: AnnotSV produces no output¶

Symptom: AnnotSV runs but the output TSV is empty or missing.

Common causes: 1. Input VCF path wrong: AnnotSV reads step 15's duphold-filtered calls (duphold/${SAMPLE}_sv_filtered.vcf.gz) first. When that file is missing, has no .tbi or is older than Manta's calls, it falls back to the Manta diploidSV.vcf.gz and prints a NOTE saying so. For that fallback the script checks both manta/ and manta2/ directories. SV_VCF overrides both. 2. No PASS variants: If Manta flagged all SVs as filtered (no PASS), AnnotSV may produce empty output. 3. Image version mismatch: Some AnnotSV versions require specific internal database versions.

Fix:

source versions.env   # from the repository root
# Verify the Manta VCF has PASS variants
docker run --rm -v ${GENOME_DIR}:/genome "${BCFTOOLS_IMAGE}" \
  bcftools view -f PASS /genome/${SAMPLE}/manta/results/variants/diploidSV.vcf.gz | grep -c -v "^#"
# Should be > 0. Typical: 7,000-9,000 for 30X WGS.

If the Manta VCF is valid but AnnotSV still produces nothing, try running with verbose output:

source versions.env   # from the repository root
docker run --rm --user root \
  -v "${GENOME_DIR}:/genome" \
  "${ANNOTSV_IMAGE}" \
  AnnotSV \
    -SVinputFile "/genome/${SAMPLE}/manta/results/variants/diploidSV.vcf.gz" \
    -outputFile "/genome/${SAMPLE}/annotsv/${SAMPLE}_test.tsv" \
    -genomeBuild GRCh38 \
    -annotationMode both 2>&1 | tee annotsv_debug.log

Step 7: PharmCAT VCF format requirements¶

Symptom: PharmCAT exits with "Input VCF does not meet requirements" or produces empty report with no gene calls.

Cause: PharmCAT is strict about VCF format: - Must be block-gzipped (.vcf.gz, not plain .vcf) - Must have a tabix index (.vcf.gz.tbi) - Must be aligned to GRCh38 - Must contain a GT (genotype) FORMAT field - Multi-allelic sites must be decomposed

Fix:

source versions.env   # from the repository root
REF_FASTA=reference/GRCh38_no_alt_analysis_set.fasta   # see 00-reference-setup.md#the-reference-path-on-every-page
# Verify your VCF has the GT field
docker run --rm -v ${GENOME_DIR}:/genome "${BCFTOOLS_IMAGE}" \
  bcftools query -f '[%GT]\n' /genome/${SAMPLE}/vcf/${SAMPLE}.vcf.gz | head -1
# Should show something like "0/1" or "1/1"

# If your VCF is plain text, compress and index it:
docker run --rm --user root -v ${GENOME_DIR}:/genome "${BCFTOOLS_IMAGE}" \
  bash -c "bcftools view /genome/${SAMPLE}/vcf/${SAMPLE}.vcf -Oz \
    -o /genome/${SAMPLE}/vcf/${SAMPLE}.vcf.gz && \
    bcftools index -t /genome/${SAMPLE}/vcf/${SAMPLE}.vcf.gz"

# If multi-allelic sites are an issue, normalize:
docker run --rm --user root -v ${GENOME_DIR}:/genome "${BCFTOOLS_IMAGE}" \
  bcftools norm -m -both \
    -f "/genome/${REF_FASTA}" \
    /genome/${SAMPLE}/vcf/${SAMPLE}.vcf.gz \
    -Oz -o /genome/${SAMPLE}/vcf/${SAMPLE}_norm.vcf.gz

Note: PharmCAT 3.2.0 includes a VCF preprocessor. If direct input fails, try the preprocessor first:

source versions.env   # from the repository root
REF_FASTA=reference/GRCh38_no_alt_analysis_set.fasta   # see 00-reference-setup.md#the-reference-path-on-every-page
docker run --rm \
  --cpus 2 --memory 4g \
  -v "${GENOME_DIR}/${SAMPLE}/vcf:/data" \
  -v "${GENOME_DIR}:/genome" \
  "${PHARMCAT_IMAGE}" \
  python3 /pharmcat/pharmcat_vcf_preprocessor \
    -vcf "/data/${SAMPLE}.vcf.gz" \
    -refFna "/genome/${REF_FASTA}" \
    -o /data/ \
    -bf "$SAMPLE"

Step 9: ExpansionHunter fails immediately¶

Symptom: ExpansionHunter crashes with model or catalog errors.

Cause: The pipeline uses ExpansionHunter v5.0.0 (EXPANSIONHUNTER_IMAGE). The v5 CLI is different from the old v2.5.5 in the weisburd/expansionhunter image.

Key v5 CLI changes from v2.5.5: - --bam → --reads - --ref-fasta → --reference - --repeat-specs <directory> → --variant-catalog <file.json> - --vcf/--json/--log → --output-prefix (auto-generates .vcf, .json) - The variant catalog (31 pathogenic GRCh38 loci) is bundled inside the container at /usr/local/share/ExpansionHunter/variant_catalog/grch38/variant_catalog.json

Fix: If running manually, use the v5 syntax:

REF_FASTA=reference/GRCh38_no_alt_analysis_set.fasta   # see 00-reference-setup.md#the-reference-path-on-every-page
ExpansionHunter \
  --reads /genome/${SAMPLE}/aligned/${SAMPLE}_sorted.bam \
  --reference "/genome/${REF_FASTA}" \
  --variant-catalog /usr/local/share/ExpansionHunter/variant_catalog/grch38/variant_catalog.json \
  --output-prefix /genome/${SAMPLE}/expansion_hunter/${SAMPLE}_eh \
  --threads 4 \
  --sex male

Step 13: bcftools +split-vep and INFO/CSQ field parsing¶

Symptom: You're trying to extract VEP annotations from the annotated VCF using bcftools query or bcftools +split-vep, but the output is empty or garbled.

Cause: VEP stores all annotations in a single INFO/CSQ field as a pipe-delimited string. This is not a standard VCF INFO field — it is a compound annotation that requires special handling.

Common mistakes:

Using bcftools query -f '%INFO/CSQ' directly: This dumps the raw pipe-delimited string, which is unreadable. Use bcftools +split-vep instead.

Wrong column names in +split-vep: The column names in CSQ depend on your VEP command-line options. Check the VCF header for the actual field order:

source versions.env   # from the repository root
docker run --rm -v ${GENOME_DIR}:/genome "${BCFTOOLS_IMAGE}" \
  bcftools view -h /genome/${SAMPLE}/vep/${SAMPLE}_vep.vcf.gz | grep "^##INFO=<ID=CSQ"
# The "Format:" part shows the pipe-delimited column order

Forgetting to specify -f (format) in +split-vep: Without -f, it outputs all fields. Specify the ones you want, and choose how multiple transcripts are reported:

source versions.env   # from the repository root
docker run --rm -v ${GENOME_DIR}:/genome "${BCFTOOLS_IMAGE}" \
  bcftools +split-vep \
    /genome/${SAMPLE}/vep/${SAMPLE}_vep.vcf.gz \
    -f '%CHROM %POS %Consequence %SYMBOL %SIFT %PolyPhen %gnomADe_AF\n' \
    -s worst
# -s worst keeps only the most severe consequence per variant.
# -d selects nothing: it puts each kept consequence on its own line, so without -s
# it prints every transcript's consequence.

gnomAD field naming confusion: Depending on VEP version and cache, the gnomAD frequency field may be named gnomADe_AF, gnomAD_AF, AF, or MAX_AF. Always check the CSQ header first.

Step 13: VEP cache download failures¶

Symptom: VEP cache download times out, produces a partial file, or extraction fails.

Cause: The VEP cache is ~26 GB, hosted on Ensembl FTP servers that can be slow. The built-in INSTALL.pl downloader does not support resume and may fail silently.

Fix — manual download with resume:

mkdir -p ${GENOME_DIR}/vep_cache/tmp
cd ${GENOME_DIR}/vep_cache/tmp

# wget -c resumes interrupted downloads
# Release 116 matches the pinned VEP image (VEP_IMAGE in versions.env)
wget -c https://ftp.ensembl.org/pub/release-116/variation/indexed_vep_cache/homo_sapiens_vep_116_GRCh38.tar.gz

# Verify download size (~26 GB)
ls -lh homo_sapiens_vep_116_GRCh38.tar.gz

# Extract (takes 10-20 minutes, expands to ~30 GB)
cd ${GENOME_DIR}/vep_cache
tar xzf tmp/homo_sapiens_vep_116_GRCh38.tar.gz

# Verify extraction
ls ${GENOME_DIR}/vep_cache/homo_sapiens/116_GRCh38/
# Should contain info.txt, variation_set_*.gz, and many other files

Symptom: VEP INSTALL.pl fails with "Cannot open Local file" permission error.

Cause: The VEP container runs as a non-root user who cannot write to /opt/vep/.vep/tmp/.

Fix: Use --user root when running VEP and pre-create the temp directory. Or (recommended) download the cache manually as shown above.

Step 17: CPSR data directory or CLI issues¶

Symptom (PCGR 2.x): refdata_dir not found or similar errors about missing bundle paths.

Cause: PCGR 2.x uses a completely different CLI and volume mount structure from 1.x. The old --pcgr_dir flag no longer exists. You now need --refdata_dir and --vep_dir as separate arguments, each with its own Docker volume mount.

Fix: Ensure you have the correct data bundle and mount structure:

source versions.env   # from the repository root
# PCGR 2.x requires four separate volume mounts:
docker run --rm --user root \
  -v ${GENOME_DIR}/vep_cache:/mnt/.vep \
  -v ${GENOME_DIR}/pcgr_data/${PCGR_DATA_BUNDLE}:/mnt/bundle \
  -v ${GENOME_DIR}/${SAMPLE}/vcf:/mnt/inputs \
  -v ${GENOME_DIR}/${SAMPLE}/cpsr:/mnt/outputs \
  "${PCGR_IMAGE}" cpsr \
    --refdata_dir /mnt/bundle \
    --vep_dir /mnt/.vep \
    ...

If you have the old 1.x data bundle (pcgr.databundle.grch38.20220203.tgz), it is not compatible with PCGR 2.x. Download the new bundle:

cd ${GENOME_DIR}/pcgr_data
wget -c https://insilico.hpc.uio.no/pcgr/pcgr_ref_data.20260620.grch38.tgz
tar xzf pcgr_ref_data.20260620.grch38.tgz
mkdir -p 20260620 && mv data/ 20260620/

Additional CPSR issue -- wrong Docker image: CPSR does not have its own Docker image. It is bundled inside the PCGR image:

source versions.env   # from the repository root
# CORRECT:
docker run ... "${PCGR_IMAGE}" cpsr ...

# WRONG (there is no sigven/cpsr image):
docker run ... sigven/cpsr ...

Step 18: CNVpytor missing resources or empty output¶

Symptom: cnvpytor -his aborts with "Some reference genome resource files are missing", or the final output has 0 CNV calls.

Common causes: 1. GC/mask resources not staged. The 1.3.2 biocontainer ships without them and its -download is broken, so the pinned files must be present at ${GENOME_DIR}/reference/cnvpytor/ and bind-mounted (see 00-reference-setup.md). All seven files must exist or the resource check aborts. 2. .pytor not writable / data dir read-only. Add --user root; the step needs a writable container FS (docker). 3. BAM chromosome naming mismatch. CNVpytor inherits chromosome names from the BAM header — an hg38 BAM must use chr1-style names.

Diagnosis:

source versions.env   # from the repository root
# Confirm all 7 pinned resource files are present
ls -lh ${GENOME_DIR}/reference/cnvpytor/
# Check the .pytor size (should be several GB for 30X WGS)
ls -lh ${GENOME_DIR}/${SAMPLE}/cnvpytor/${SAMPLE}.pytor

# Re-verify the container data path for the pinned image (mount target)
docker run --rm "${CNVPYTOR_IMAGE}" \
  python -c 'import cnvpytor, os; print(os.path.dirname(cnvpytor.__file__) + "/data")'

Fix: Download the pinned resource files (docs/00-reference-setup.md), then re-run. Confirm the BAM path resolves to the containerized /genome/... path.

Step 19: Delly long runtime / high memory¶

Symptom: Delly has been running for many hours on a 30X genome.

Cause: Delly examines every read pair across the genome for split-read and paired-end SV evidence. On high-coverage samples or with slow storage, this takes longer.

Measured runtimes: in the pipeline (4 CPUs requested, Delly using about one), 21 minutes to 3 h 19 min over five runs on three 30-40x genomes, peak memory 1.2-3.1 GB (runtime). Higher coverage was not measured.

If Delly exceeds 12 hours: 1. Check docker stats to verify it is still using CPU (not stuck) 2. Confirm storage is not a bottleneck (NFS/SMB mounts are much slower) 3. Consider running only on specific chromosomes to reduce scope:

source versions.env   # from the repository root
REF_FASTA=reference/GRCh38_no_alt_analysis_set.fasta   # see 00-reference-setup.md#the-reference-path-on-every-page
# Delly on the sample's BAM without the exclude map step 19 passes
docker run --rm --user root \
  --cpus 4 --memory 8g \
  -v "${GENOME_DIR}:/genome" \
  "${DELLY_IMAGE}" \
  delly sr \
    -g "/genome/${REF_FASTA}" \
    -o /genome/${SAMPLE}/delly/${SAMPLE}_sv.bcf \
    /genome/${SAMPLE}/aligned/${SAMPLE}_sorted.bam
# Delly processes every contig of the reference; slowness usually comes
# from centromeres, telomeres and the unplaced scaffolds. Step 19 skips
# them with Delly's exclude map (-x reference/delly_human.hg38.excl.tsv,
# installed by setup.sh); add it here the same way.

Step 20: GATK Mutect2 mitochondrial mode¶

Problem: "Cannot create sequence dictionary" The script tries to create the reference's sequence dictionary (GRCh38_no_alt_analysis_set.dict) if it does not exist. If you get permission errors, create it manually:

source versions.env   # from the repository root
REF_FASTA=reference/GRCh38_no_alt_analysis_set.fasta   # see 00-reference-setup.md#the-reference-path-on-every-page
docker run --rm --user root \
  -v "${GENOME_DIR}:/genome" \
  "${GATK_IMAGE}" \
  gatk CreateSequenceDictionary \
    -R "/genome/${REF_FASTA}" \
    -O "/genome/${REF_FASTA%.fasta}.dict"

Problem: 0 mitochondrial variants called Possible causes: - BAM has no chrM reads (check with samtools idxstats | grep chrM) - Mitochondrial chromosome name mismatch (MT vs chrM). GRCh38 uses chrM.

Step 25: pgsc_calc killed for memory¶

Symptom: the PRS step stops after pgsc_calc's first task, FORMAT_SCOREFILES, with exit 137, or with concurrent.futures.process.BrokenProcessPool in pgsc_calc.log (the kernel killed one of its worker processes).

Cause: that task reads every scoring file whole. On the nine scores measured (the four lipid and blood-pressure scores added since hold about 1,200 rows together) it needs about 14 to 18 GB on one thread, more on more threads (step 25, Memory). Its container gets pgsc_calc's --max_memory.

Fix: 1. With run-all.sh or Nextflow, PRS asks for 24 GB and is retried once with twice that, both capped at --max_memory: pass --max_memory 24.GB or more when the machine has it. 2. With scripts/25-prs.sh, pgsc_calc gets three quarters of the RAM; set PGSC_MAX_MEMORY=24.GB (or as much as the machine can spare) and run the step again. 3. On a 16 GB machine, enable swap on the host: Docker then lets the container swap as much again as its limit, which is how a 16 GB GitHub runner finished it.

HLA Typing (Step 8): Known difficulties¶

HLA typing from WGS data is unreliable in Docker. The two main tools have unresolved issues:

HLA-LA: Requires a pre-serialized 40 GB graph. Most Docker images do not include it. The one image that does (jiachenzdocker/hla-la, pinned by digest in step 8) crashes during graph alignment. Status: UNSOLVED.

T1K: Works partially, but the coordinate file build step requires the full reference FASTA (not the .fai index). With correctly built coordinates, T1K can call some HLA alleles but may have ~50% unmapped alleles.

Recommendation: For clinical HLA typing, rely on dedicated lab assays. Short-read WGS is not ideal for this highly polymorphic region.

Performance Issues¶

Everything is slow on macOS Apple Silicon¶

Cause: All bioinformatics Docker images are amd64/x86_64. On Apple Silicon Macs (M1-M4), Docker Desktop runs them through Rosetta 2 emulation. This adds: - 2-5x CPU overhead (emulation is compute-expensive) - ~2x memory overhead (emulation runtime consumes extra RAM) - I/O overhead from the Docker Desktop VM

Expected slowdown by step:

We have not measured a run on Apple Silicon. Natively on Linux with 8 shards, DeepVariant alone took 8 h 50 min to 13 h 30 min in three measured 30-40x runs (runtime); under emulation every step takes longer.

Mitigation strategies: 1. Run fewer heavy steps at once. With run-all.sh, bound the total with an executor block set to Docker Desktop's CPUs and memory (a shared host), and pass THREADS and --max_memory no larger than the VM's (e.g. THREADS=6 ... --max_memory 14.GB); --max_cpus and --max_memory alone cap each task, not the total. 2. Lower the step scripts' limits. For step 3, THREADS=2 DV_MEM=12g instead of the default 8 CPUs and 32 GB (DeepVariant crashes on Mac). 3. Offload the heaviest steps. Run steps 2, 3, 18, 19 on a remote Linux machine and bring back the outputs. Everything after step 3 needs only the VCF or BAM. 4. Use a cloud instance. A Hetzner CCX33 has 8 vCPUs and 32 GB at ~$0.18/hr; plan for more than a day of run time on 8 vCPUs.

WSL2 file I/O is extremely slow¶

Symptom: Steps that read large files (alignment, DeepVariant, VEP) take 10-50x longer than expected.

Cause: If your data is on a Windows drive (/mnt/c/, /mnt/d/), WSL2 accesses it through the 9P filesystem protocol, which is catastrophically slow for large random-access files.

Fix: Move ALL genomics data to the Linux filesystem:

# Inside WSL2
mkdir -p ~/genome_data
# Move or copy data from /mnt/c/ to ~/genome_data/ (replace <windows-username>)
cp -r "/mnt/c/Users/<windows-username>/genome_data/"* ~/genome_data/
export GENOME_DIR=~/genome_data

Data on the Linux filesystem (~/, /home/, /tmp/) uses ext4 natively and performs at full speed.

Monitoring container resources¶

Use docker stats to see real-time resource usage:

# Live dashboard (updates every second)
docker stats

# One-time snapshot
docker stats --no-stream

# Output columns:
# CONTAINER  CPU%  MEM USAGE/LIMIT  NET I/O  BLOCK I/O

What to look for: - CPU at 0% for a long time: Container may be stuck or waiting for I/O - MEM USAGE near LIMIT: Container is about to be OOM-killed. Increase --memory. - NET I/O > 0: Container is downloading something (may be slow due to network) - BLOCK I/O very high: I/O-bound step. Faster storage would help.

When to reduce `--cpus` or `--memory`¶

With run-all.sh you do not edit the scripts' flags: THREADS=N caps every task's CPUs (--max_cpus), and --max_memory 24.GB after the sex caps every task's memory. They cap each task, not the run. Nextflow starts a task when its request fits in what is left of the machine's CPUs and RAM after the tasks it already started; it does not look at what other programs use. To bound the total, see a shared host.

Reduce --cpus when: - Running multiple steps in parallel on a machine with limited cores - Running on macOS with Rosetta 2 emulation (more shards = more overhead) - Your machine has fewer than 8 cores

Reduce --memory when: - Docker Desktop memory allocation is less than the script's --memory flag - Running multiple containers simultaneously - System becomes unresponsive during analysis

Safe minimum values per step:

Step	Min --cpus	Min --memory	Notes
2 (minimap2)	2	8g	Slower but works
3 (DeepVariant)	2	8g	`THREADS=2 DV_MEM=8g`; `THREADS` also sets the shards
4 (Manta)	2	4g
6 (ClinVar)	1	1g	Very light
7 (PharmCAT)	1	2g
9 (ExpansionHunter)	2	2g
13 (VEP)	2	4g	Set `--fork 2` to match
17 (CPSR)	2	4g
18 (CNVpytor)	4	8g
19 (Delly)	2	4g

Steps that can run in parallel vs must be sequential¶

SEQUENTIAL (must run in order):
  Step 2 (alignment) ──> Step 3 (variant calling)
  Step 4 (Manta) ──> Step 5 (AnnotSV)
  Step 4 (Manta) ──> Step 15 (duphold)

PARALLEL once their input exists (the BAM after step 2, the VCF after step 3):
  ┌─ Step 4 (Manta)         ← needs BAM
  ├─ Step 6 (ClinVar)       ← needs VCF
  ├─ Step 7 (PharmCAT)      ← needs VCF
  ├─ Step 9 (ExpansionHunter) ← needs BAM
  ├─ Step 10 (TelomereHunter) ← needs BAM
  ├─ Step 11 (ROH)          ← needs VCF
  ├─ Step 12 (haplogrep3)   ← needs VCF
  ├─ Step 13 (VEP)          ← needs VCF
  ├─ Step 14 (imputation)   ← needs VCF
  ├─ Step 16 (indexcov)      ← needs BAM index only
  ├─ Step 17 (CPSR)         ← needs VCF
  ├─ Step 18 (CNVpytor)     ← needs BAM
  ├─ Step 19 (Delly)        ← needs BAM
  └─ Step 20 (Mutect2 mito) ← needs BAM

RESOURCE CAUTION:
  Steps 3, 18, 19 are the most CPU/RAM intensive.
  Do not run more than 2 of these simultaneously on a 32 GB machine.

Output Issues¶

0-byte output files¶

Symptom: Output file exists but is 0 bytes.

Most common cause: The input path was wrong inside the Docker container. Docker mounts the host path to /genome, so all paths inside the container must start with /genome/, not the host path.

Diagnosis checklist: 1. Check that the input file exists at the expected path:

ls -la ${GENOME_DIR}/${SAMPLE}/vcf/${SAMPLE}.vcf.gz

2. Run the script with debug output:

bash -x scripts/06-clinvar-screen.sh your_name

3. Inspect Docker mount mapping. The script mounts ${GENOME_DIR}:/genome. Inside the container, ${GENOME_DIR}/sample/vcf/sample.vcf.gz becomes /genome/sample/vcf/sample.vcf.gz. 4. Check for bgzip/tabix path issues. The bcftools image (BCFTOOLS_IMAGE) does not include bgzip or tabix in $PATH. Use bcftools view -Oz -o instead of piping to bgzip. See lessons-learned.md.

Other causes: - Tool crashed silently before writing output (check exit code and stderr) - Disk full (check df -h) - Permission denied in output directory (add --user root)

Wrong number of variants (too few)¶

Symptom: DeepVariant VCF has far fewer variants than expected.

Expected variant counts for 30X WGS (GRCh38):

Metric	Expected Range	Notes
Total variants (SNP + indel)	4.5-5.5 million	Total rows in VCF
PASS variants	4.0-4.8 million	After DeepVariant quality filtering
SNPs	3.8-4.5 million	~85% of total
Indels	600K-900K	~15% of total
Heterozygous	2.5-3.2 million	~55-60% of total
Homozygous ALT	1.8-2.3 million	~40-45% of total

If total variants < 3 million: 1. Genome build mismatch. Your BAM may be hg19, but the reference is hg38. See Wrong genome build. 2. Low coverage. Check average depth:

source versions.env   # from the repository root
docker run --rm -v ${GENOME_DIR}:/genome "${SAMTOOLS_IMAGE}" \
  samtools depth -a /genome/${SAMPLE}/aligned/${SAMPLE}_sorted.bam | \
  awk '{sum+=$3; n++} END {print "Average depth:", sum/n}'

30X WGS should show average depth of 25-35. 3. BAM is mostly unmapped. Check alignment rate:

source versions.env   # from the repository root
docker run --rm -v ${GENOME_DIR}:/genome "${SAMTOOLS_IMAGE}" \
  samtools flagstat /genome/${SAMPLE}/aligned/${SAMPLE}_sorted.bam

Mapped percentage should be > 98%.

If total variants > 7 million: 1. Likely includes many false positives. Filter to PASS only:

source versions.env   # from the repository root
docker run --rm -v ${GENOME_DIR}:/genome "${BCFTOOLS_IMAGE}" \
  bcftools view -f PASS /genome/${SAMPLE}/vcf/${SAMPLE}.vcf.gz | \
  bcftools stats | grep "number of records"

2. Multi-allelic sites may be inflating the count. Normalize:

source versions.env   # from the repository root
REF_FASTA=reference/GRCh38_no_alt_analysis_set.fasta   # see 00-reference-setup.md#the-reference-path-on-every-page
docker run --rm -v ${GENOME_DIR}:/genome "${BCFTOOLS_IMAGE}" \
  bcftools norm -m -both \
    -f "/genome/${REF_FASTA}" \
    /genome/${SAMPLE}/vcf/${SAMPLE}.vcf.gz | bcftools stats | grep "number of records"

Wrong number of variants (too many ClinVar hits)¶

Symptom: ClinVar screen returns hundreds of "pathogenic" hits, which seems too many.

Cause: The ClinVar pathogenic filter includes CLNSIG~"Pathogenic", which matches variants where pathogenicity depends on context. Some variants are "Pathogenic" for one condition but "Benign" for another (compound CLNSIG fields like Pathogenic/Likely_benign).

Expected ClinVar hit counts:

Category	Expected Count
Total ClinVar matches (any significance)	50-150
Pathogenic + Likely pathogenic	0-10
True actionable findings	0-3

If you see more than ~15 pathogenic hits: The ClinVar VCF may not be filtered correctly, or chromosome naming may be mismatched (causing position-only matches without allele verification). Re-run the ClinVar screen from step 0 setup.

How to verify outputs are correct¶

Quick sanity checks for each major output:

source versions.env   # from the repository root
SAMPLE=your_name

# VCF: check variant count and type distribution
docker run --rm -v ${GENOME_DIR}:/genome "${BCFTOOLS_IMAGE}" \
  bcftools stats /genome/${SAMPLE}/vcf/${SAMPLE}.vcf.gz | grep "^SN"

# BAM: check alignment rate and depth
docker run --rm -v ${GENOME_DIR}:/genome "${SAMTOOLS_IMAGE}" \
  samtools flagstat /genome/${SAMPLE}/aligned/${SAMPLE}_sorted.bam

# Manta SV count (expect 7,000-9,000 total)
docker run --rm -v ${GENOME_DIR}:/genome "${BCFTOOLS_IMAGE}" \
  bcftools view /genome/${SAMPLE}/manta/results/variants/diploidSV.vcf.gz | grep -c -v "^#"

# PharmCAT: check the HTML report exists and has content
ls -la ${GENOME_DIR}/${SAMPLE}/vcf/${SAMPLE}.report.html
# Should be > 50 KB

# CPSR: check the HTML report exists
ls -la ${GENOME_DIR}/${SAMPLE}/cpsr/${SAMPLE}.cpsr.grch38.html
# Should be > 100 KB

# VEP: check annotated VCF exists and has annotations
gzip -dc ${GENOME_DIR}/${SAMPLE}/vep/${SAMPLE}_vep.vcf.gz | awk '!/^#/ && /CSQ=/ {n++} END {print n+0}'
# Should print a number above 0: records carry consequence annotations

Expected output sizes per step¶

Step	Output File(s)	Expected Size
2	`*_sorted.bam` + `.bai`	80-120 GB + 5-8 MB
3	`*.vcf.gz` + `.tbi`	80-200 MB + 1-2 MB
4	`diploidSV.vcf.gz`	1-5 MB
5	`*_sv_annotated.tsv`	25-35 MB
6	`*_clinvar_hits.vcf`	< 1 MB
7	`*.report.html`	50-200 KB
9	`_eh.json` + `_eh.vcf`	< 1 MB each
10	`*_summary.tsv` + plots	50-200 MB total
11	`*_roh.txt`	1-5 MB
12	`*_haplogroup.txt`	< 1 KB
13	`*_vep.vcf.gz` + `.tbi`	less than its 2-5 GB uncompressed size (the bgzipped size is not measured here)
17	`*.cpsr.grch38.html` + TSV	50-200 MB total
18	`*_cnvs.txt` + `.root`	Text: < 1 MB, ROOT: 5-15 GB
19	`*_sv.vcf.gz`	5-20 MB
20	`*_chrM_filtered.vcf.gz`	< 1 MB

If any output is significantly smaller than expected (especially 0 bytes), see 0-byte output files.

Getting Help¶

Before opening an issue¶

Check this guide first. Ctrl+F your error message.
Check docs/lessons-learned.md for known Docker image issues and tool quirks.
Run with debug output: bash -x scripts/XX-stepname.sh your_name 2>&1 | tee debug.log

Collect diagnostic info:

docker version
docker info | grep -E "Total Memory|CPUs|Storage Driver"
uname -a
df -h ${GENOME_DIR}

Opening a GitHub issue¶

File an issue at: github.com/GeiserX/OwnGenome/issues

Include: - Which step failed (step number and script name) - Full error message (copy-paste, not screenshot) - Your platform (Linux distro, macOS version, WSL2 version) - Docker version (docker version) - Available RAM and disk space - Input data type (FASTQ, BAM, or VCF) and approximate size - Whether you are using native Linux or Docker Desktop

Useful diagnostic commands¶

# System info
uname -a                           # OS and architecture
docker version                     # Docker version
docker info | head -30             # Docker config summary
docker system df                   # Docker disk usage

# Container debugging
docker ps -a --latest              # Last container status
docker logs <container_id>         # Container output
docker inspect <container_id>      # Full container metadata

# File validation
samtools quickcheck file.bam       # BAM integrity (via Docker)
gzip -t file.vcf.gz               # Gzip integrity
bcftools view -h file.vcf.gz      # VCF header readability

# Resource monitoring
docker stats --no-stream           # Current container resource usage
df -h                              # Disk space
free -h                            # RAM (Linux only)

Troubleshooting Guide¶

Table of Contents¶

Docker Issues¶

Container exits immediately with no output¶

Container exits with code 1 but no error message¶

"No such image" / "manifest for ... not found"¶

Images that were pulled are missing after a Docker cleanup¶

"Permission denied" writing to mounted volumes¶

Docker Desktop not enough memory (Mac/Windows)¶

Containers stuck / running forever¶

The run is silent and no container is running¶

The work disk is filling¶

"No space left on device" mid-analysis¶

Input Data Issues¶

Wrong genome build (hg19 vs hg38)¶

CRAM files instead of BAM¶

Paired FASTQ naming issues¶

Corrupt or truncated downloads¶

The BAM lost reads¶

BAM not sorted or not indexed¶

Per-Step Troubleshooting¶

Step 2: Alignment (minimap2 + samtools)¶

Step 3: DeepVariant and a GPU¶

Step 3: DeepVariant crashes on Mac (amd64 emulation)¶

Step 4: Manta produces no variants¶

Step 5: AnnotSV produces no output¶

Step 7: PharmCAT VCF format requirements¶

Step 9: ExpansionHunter fails immediately¶

Step 13: bcftools +split-vep and INFO/CSQ field parsing¶

Step 13: VEP cache download failures¶

Step 17: CPSR data directory or CLI issues¶

Step 18: CNVpytor missing resources or empty output¶

Step 19: Delly long runtime / high memory¶

Step 20: GATK Mutect2 mitochondrial mode¶

Step 25: pgsc_calc killed for memory¶

HLA Typing (Step 8): Known difficulties¶

Performance Issues¶

Everything is slow on macOS Apple Silicon¶

WSL2 file I/O is extremely slow¶

Monitoring container resources¶

When to reduce --cpus or --memory¶

Steps that can run in parallel vs must be sequential¶

Output Issues¶

0-byte output files¶

Wrong number of variants (too few)¶

Wrong number of variants (too many ClinVar hits)¶

How to verify outputs are correct¶

Expected output sizes per step¶

Getting Help¶

Before opening an issue¶

Opening a GitHub issue¶

Useful diagnostic commands¶

When to reduce `--cpus` or `--memory`¶