syncing with master branch - #1
Merged
Merged
Conversation
* Barcode correction and demuxing don't necessarily need to be ran concurrently with annotation. Separate commands for both with options to also be performed concurrently. * Barcode correction can be ran without a need for a whitelist generated by corresponding short-read library. This makes barcode correction step to be ran once annotation is finished on the whole dataset. Concurrent annotation, barcode correction and demuxing can still be ran with a readily available whitelist. * Improved dynamic batching taking into consideration all the layers of the model being used. This prevents users from under or over utilizing the GPU memory * QC metrics automated and can be generated and visualized using qc-metrics.
New modules / commands
- evaluate_model / assess-model (scripts/evaluate_model.py, wrappers/evaluate_model_wrap.py, docs/webpages/model_training/assess_model.qmd): new model-assessment module with report generation and per-segment accuracy metrics.
- featurecounts (scripts/featurecounts_matrix.py, wrappers/featurecounts_wrap.py, docs/webpages/pipeline/featurecounts.qmd): new count-matrix generation step.
- container_runtime.py: container detection/runtime helpers.
Build / packaging
- Migrated setup.py → pyproject.toml (removed setup.py, MANIFEST.in).
- Added version stamping via __init__.py + utils/__init__.py, with new unit test tests/unit/test_init_version.py.
- environment.yml updates; conftest.py added.
Training / artifacts
- train_model: artifact versioning, cleaner training folder naming, new CLI help grouping, updated training_params.yaml, expanded docs (train_model.qmd, train_model_cli.qmd, read_simulation.qmd, simulate_data_cli.qmd, model_training.qmd).
- Added generic guidance that Conv+CRF (no BiLSTM) is often sufficient.
- Removed intro_errors.qmd.
Pipeline improvements
- Deduplication speed-up: the "duplicate marking speed up"
commit.
- QC metrics: boxplot corrections, MultiQC-compatible outputs, no-PyArrow refactor; wrapper updates.
- Preprocessing (scripts/preprocess_reads.py, wrappers/preprocess_wrap.py): reworked + documented.
- Barcode correction / demux: refactors in barcode_correction_wrap.py, correct_barcodes.py, demux_wrap.py; scripts/export_demux.py removed.
- Annotation: updates to annotate_new_data.py, annotate_reads.py, extract_annotated_seqs.py, annotate_reads_wrap.py.
- Barcode discovery updates (scripts/discover_barcodes.py, generate_whitelist_wrap.py).
Configuration
- utils/seq_orders.yaml: added segment/library definitions used by refactor.
- utils/training_params.yaml updated.
Tests
- New tests/unit/test_init_version.py.
- Major rewrite of tests/10x3p/test_pipeline.py for new features.
- Minor updates to other unit tests.
CI / docs
- .github/workflows/test.yml (model download + coverage adjustments), lint.yml tweak.
- Documentation restructured under docs/webpages/pipeline/ (new pages: preprocessing, annotation, barcode_demux, align_dedup, qc_metrics, split_bam, visualization, featurecounts) + pipeline_guide.qmd, quick_start.qmd, resource_requirements.qmd, usage.qmd, _quarto.yml, index.qmd.
pytest on push and PR to dev and annot_demux_refactor
10x3p_sc_ont_013 default model now
non-PEP440 tage stripped to clean version
New commands barcode_correct — standalone Levenshtein correction against a whitelist; resumable, optional inline demux via --run-demux. demux_reads — standalone FASTA/FASTQ export from annotations (demuxed or bulk). generate_whitelist — whitelist-free cell-barcode discovery via knee detection + deletion-neighborhood near-dup merging. qc_metrics — annotation + BAM-level QC, knee plots, boxplots, MultiQC TSVs, HTML report. ~3.9k lines, no PyArrow. assess_model — per-segment accuracy assessment for trained models with a generated report. featurecounts — gene-level count matrix from per-cell BAMs. Pipeline restructure You can run barcode_correct and demux_reads against existing annotations — useful for resuming, re-running with different thresholds, or the whitelist-free flow. annotate_reads and visualize accept preprocessed directories directly. User-configurable bin width in preprocessing Option to split concatenated reads during annotation. Checkpoint/resume across annotate, BC correct, and demux. Optional chunk cleanup. annotations_valid.parquet auto-removed once the corrected file exists. Barcode correction handles UMI-less protocols and arbitrary barcode-type combos. All FASTA/FASTQ outputs gzipped. Model / training REG and HYB paths dropped. CRF-only. 10x3p_sc_ont_013 is the new default. seq_orders and training_params moved from TSV → YAML with a cleaner schema. Dynamic batch sizing now accounts for all layers, not just conv. Training artifacts versioned and folder naming cleaned up. QC qc_metrics.py computes and visualizes multiple QC plots at various levels Dedup / BAM UMI dedup significantly faster. Minor tweaks to split_bam. Packaging & CI setup.py → pyproject.toml. Moved to setuptools-scm for versioning. Fixed the upstream CI breakage: tag_regex + git_describe_command so legacy non-PEP440 tags like v0.2.1_tf2.15.0 are stripped to 0.2.1. Added a Docker publish workflow. pytest now runs on push/PR to dev and annot_demux_refactor. New unit tests for checkpoint chunk size, _version, and the seq_orders YAML refactor. Integration tests expanded. Added container_runtime.py helper. Docs Quarto site reorganized into per-stage pages: preprocessing, annotation, barcode/demux, align/dedup, QC, split BAM, visualization, featurecounts. New model-assessment guide; model-training and read-simulation docs rewritten. Quick start, usage, install, resource-requirements pages refreshed.
AyushSemwal
merged commit Apr 17, 2026
b7c4c11
into
AyushSemwal:annot_demux_refactor
4 of 5 checks passed
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment Thanks for integrating Codecov - We've got you covered ☂️ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
syncing with master branch