Skip to content

About

TopMed SV QC scripts

Resources

Stars

1 star

Watchers

4 watching

Forks

Latest commit

 

History

25 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

SEPARATE BY SVTYPE
==================
There should be separate DEL, DUP, and INV files. DUP files are a special case and will be added later.
To start, the pipeline will focus on DEL/INV

bash scripts/mk_separate_by_type_cmds.sh make the jobs and then submit them

EXTRACT NEW DUPS
================

bash scripts/mk_subsetted_dups_cmds.sh 

ESTIMATING PPV OF DATA
======================
I need the 'bench' samples vcfs

1) I'll need to extract just the replicate samples so I have an easy VCF to work with
	scripts/mk_extract_bench_samples_cmds.sh make the jobs and then submit them
	I'm running on a per-chrom bases so I need to consolidate manually with vcf-concat and create
	the validation/topmed.replicate.DEL.vcf.gz
3) The script to does the validation..
	bash scripts/mk_truvari_cmds.sh make the jobs and then submit them
	NOTE! 
	the COMPVCF and OUTDIR variables need to be manually set inside of the script
4) I then need to make the dataframe and the FP/TP consolidation of the results
	Make the summary dataframe for analysis
	python scripts/truvari_df_joiner.py validation/results/deletions/ validation/results/deletions/summary.jl

5) Make the consolidation, annotated TP/FPs from the 'defaults'
	python scripts/validation_consolidator.py validation/results/deletions/ \
			metadata/validation_replicate.manifest.txt \
			validation/results/annotated.deletions.vcf

6) And then I need to clean up the notebook so I can run one simple thing on the summary.jl and consolidated data frame

DOING THE PCA
=====================
Note - this doesn't work for the DUPs with filter values. So I manually concat and filter down to only DUPs.
Then skip to step #2

1) bash scripts/mk_vcf2pca_cmds.sh make the jobs and then submit them

This will setup pca_data/ directory with the per svtype files

2) Run pca pipeline on the h5 files

	python scripts/make_pca_data.py pca_data/deletions.jl pca_data/*DEL*h5
	python scripts/make_pca_plots.py metadata/ancestries.txt pca_data/inversions.jl pca_data/inversions.png

PCA FILTERING
=============
I think there might be samples I need to remove because they're looking weird in the PCAs?
PC2 <= -20
    2505    NWD895484
    2529    NWD593511
    2632    NWD134290
    3006    NWD594428
    3008    NWD989091
PC4 >= 500
    162     NWD207905
    179     NWD112322
    2505    NWD895484
    2529    NWD593511
    2632    NWD134290
    3006    NWD594428
    3008    NWD989091
Should look at the QC metrics on these...
For DUPs
>>> ancestries[pc1 > 150]
       NWDID          Ancestry
48220  NWD976804      AFR  < -- super outlier
234     NWD170197      EUR <-- mini outliers
1585    NWD910621      EUR
4458    NWD320728      EUR
4577    NWD592708      EUR
5686    NWD859633      EUR
27763   NWD786098      EUR
58463   NWD480965      EUR
58521   NWD369359      EUR
61519   NWD719720      EUR
127853  NWD884712      AFR <-- mini

DOING THE FST
=============
python scripts/make_fst_data.py metadata/indexed_ancestries.jl fst_data/inversions.jl pca_data/*INV*h5
python scripts/make_fst_plots.py fst_data/inversions.jl fst_data/inversions.png
	Note. Inversions fail...
	Traceback (most recent call last):
	  File "scripts/make_fst_data.py", line 58, in <module>
	    ac.append(garr.count_alleles(subpop=subpop))
	  File "/stornext/snfs5/next-gen/scratch/english/round2/pyenv/lib/python3.8/site-packages/allel/model/ndarray.py", line
	1835, in count_alleles
	    max_allele = self.max()
	  File "/stornext/snfs5/next-gen/scratch/english/round2/pyenv/lib/python3.8/site-packages/numpy/core/_methods.py", line
	40, in _amax
	    return umr_maximum(a, axis, None, out, keepdims, initial, where)
	ValueError: zero-size array to reduction operation maximum which has no identity
DELs:
    AFR AMR 0.08346761091158061
    AFR EAS 0.10828191976566769
    AFR EUR 0.0731332436710205
    AFR SAS 0.07136415180653874
    AMR EAS 0.0570303261965993
    AMR EUR 0.03520831082656243
    AMR SAS 0.03736846401732368
    EAS EUR 0.08069643282330462
    EAS SAS 0.05827352777745373
    EUR SAS 0.019150763825462696

ANNOTATING VARIANTS
===================
scripts/mk_annosv_cmds.sh  awef.vcf.gz aoba.vcf.gz to make the jobs and then submit them

Populates annosv/ directories

Build the VCFs
python scripts/put_annosv_in_vcf.py annosv/topmed.INV.vcf.gz/tmp_topmed.INV.vcf.gz.annotated.tsv call_only_vcfs/topmed.INV.vcf.gz \
	| bgzip > annosv/topmed.INV.anno.vcf.gz

Which I then can vcf2df
truvari vcf2df -i annosv/topmed.INV.anno.vcf.gz annosv/topmed.INV.anno.jl

MAKING THE OVERALL VARIANT QC
=============================
After I have the final, filtered set of variants, I then need to do the concat and restriction to the first sample.
Then I can run through annosv and make a variant summary
would be fine to concatenate by type at that point, also

I got a notebook that will work just fine for this

About

TopMed SV QC scripts

Resources

Stars

1 star

Watchers

4 watching

Forks

Releases

Packages

Contributors

Languages