Joint RPCA replication workflow

.github/workflows/basic_checks.yaml

@@ -49,7 +49,14 @@ jobs:
 
       - name: Build pkgdown
         run: |
-           PATH=$PATH:$HOME/bin/ Rscript -e 'pkgdown::build_site(".")'
+          PATH=$PATH:$HOME/bin/ Rscript -e 'pkgdown::build_site(".")'
+
+      - name: Add Joint-RPCA validation report
+        run: |
+          mkdir -p docs/filterRPCAInput_validation
+          cp \
+            vignettes/filterRPCAInput_validation/joint-rpca-mia-gemelli-comparison.html \
+            docs/filterRPCAInput_validation/index.html
 
       - name: Upload pkgdown artifact
         if: github.event_name == 'push'
@@ -143,4 +150,4 @@ jobs:
           push: ${{ github.event_name != 'pull_request' }}
           tags: |
             ${{ env.IMAGE }}:latest
-            ${{ env.IMAGE }}:${{ env.GIT_SHA }}
+            ${{ env.IMAGE }}:${{ env.GIT_SHA }}

vignettes/filterRPCAInput_validation/SETUP.md

@@ -0,0 +1,242 @@
+# Reproduce the mia–Gemelli Joint-RPCA validation
+
+This guide runs the numbered scripts and then renders one complete HTML report.
+You only need to enter the locations on your computer once in each environment.
+
+## Reproducibility specification
+
+| Item | Fixed value |
+|---|---|
+| mia version | `1.21.6` |
+| mia commit | `25fa7bb51792566131f2bd8d07b4ebd1fde54b38` |
+| mia data | Revised `ibdmdb` main MGX and MTX experiments |
+| Run configuration | `config.tsv` |
+| Feature prevalence | `0.10` in R = `10%` in Gemelli |
+| Components | `3` |
+| Maximum iterations | `5` |
+| Test samples | `10` |
+| Random seed | `42` |
+
+The R runner checks the mia version and commit. The Python runner checks the
+SHA-256 hashes of the inputs exported by R. Both implementations use the same
+saved training/test split.
+
+## Required software
+
+- Git
+- R and RStudio
+- Quarto, normally included with RStudio
+- Windows Subsystem for Linux (WSL)
+- Miniforge or Miniconda inside WSL
+
+## Project layout
+
+Extract or copy the supplied files into one folder. This folder is called
+`PROJECT_DIR` throughout this guide:
+
+```text
+<PROJECT_DIR>/
+├── config.tsv
+├── SETUP.md
+├── joint-rpca-mia-gemelli-comparison.qmd
+└── scripts/
+    ├── 00_check_gemelli_environment.py
+    ├── 01_run_mia_filter_joint_rpca.R
+    ├── 02_run_gemelli_filter_joint_rpca.py
+    └── 03_compare_mia_gemelli_corrected.R
+```
+
+The analysis creates `input/` and `results/` automatically.
+
+## Step 1: enter your Windows paths once
+
+Open PowerShell. Replace the three example paths below with locations on your
+computer. You should only need to edit these three lines:
+
+```powershell
+$PROJECT_DIR = "C:\path\to\filterRPCAInput_validation"
+$MIA_CLONE_DIR = "C:\path\to\your\mia"
+$MIA_VALIDATION_DIR = "C:\path\to\mia_validation_25fa7bb5"
+```
+
+- `PROJECT_DIR` is the folder containing this file and `config.tsv`.
+- `MIA_CLONE_DIR` is your existing Git clone of the mia repository.
+- `MIA_VALIDATION_DIR` is a new location for the clean, pinned worktree.
+
+Keep this PowerShell window open so the variables remain available.
+
+## Step 2: create a clean mia checkout
+
+Do not modify or discard changes in your normal mia checkout. Create a separate
+worktree at the pinned commit:
+
+```powershell
+Set-Location $MIA_CLONE_DIR
+git fetch upstream
+git worktree add --detach $MIA_VALIDATION_DIR 25fa7bb51792566131f2bd8d07b4ebd1fde54b38
+git -C $MIA_VALIDATION_DIR status --short
+git -C $MIA_VALIDATION_DIR rev-parse HEAD
+```
+
+The status command should print nothing. The final command should print the
+pinned commit shown above.
+
+## Step 3: prepare R and run mia
+
+Open RStudio. Enter your two relevant Windows paths once, using forward slashes
+in R. Replace only the first two path strings:
+
+```r
+project_dir <- normalizePath(
+    "C:/path/to/filterRPCAInput_validation",
+    winslash = "/"
+)
+mia_repo <- normalizePath(
+    "C:/path/to/mia_validation_25fa7bb5",
+    winslash = "/"
+)
+
+Sys.setenv(
+    MIA_GEMELLI_PROJECT = project_dir,
+    MIA_REPO = mia_repo
+)
+```
+
+Install the workflow packages and any missing dependencies of the pinned mia
+checkout. This is needed only once for the selected R library:
+
+```r
+install.packages(c("devtools", "digest", "knitr"))
+devtools::install_deps(mia_repo, dependencies = TRUE)
+```
+
+Run mia and export the common inputs:
+
+```r
+source(file.path(
+    project_dir,
+    "scripts",
+    "01_run_mia_filter_joint_rpca.R"
+))
+```
+
+A successful run ends with `mia run completed successfully.` and creates:
+
+```text
+input/frequency_010/
+results/frequency_010/mia/
+```
+
+## Step 4: prepare Gemelli in WSL
+
+From PowerShell, enter WSL:
+
+```powershell
+wsl
+```
+
+Windows and WSL name the same folder differently. For example, Windows drive
+`C:` is `/mnt/c` in WSL, while drive `D:` is `/mnt/d`. Set the WSL form of your
+project path once, replacing only this line:
+
+```bash
+PROJECT_DIR_WSL="/mnt/c/path/to/filterRPCAInput_validation"
+cd "$PROJECT_DIR_WSL"
+```
+
+Enable Conda and activate the environment:
+
+```bash
+source ~/miniforge3/etc/profile.d/conda.sh
+conda activate mia-gemelli
+```
+
+If the environment does not exist, create it and install the pinned Gemelli
+source instead:
+
+```bash
+conda create -n mia-gemelli python=3.10 pip -y
+conda activate mia-gemelli
+python -m pip install \
+  "gemelli @ git+https://github.com/biocore/gemelli.git@c53c9ee958948683bc4216cf5fa447e2c3ac4806"
+```
+
+Run the environment/input check and Gemelli:
+
+```bash
+python scripts/00_check_gemelli_environment.py --project-dir .
+python scripts/02_run_gemelli_filter_joint_rpca.py --project-dir .
+```
+
+Run the second command only after the first passes. A successful run creates
+`results/frequency_010/gemelli/`. Then leave WSL with `exit`.
+
+## Step 5: compare the implementations
+
+Return to the same RStudio session, where `project_dir` is already defined:
+
+```r
+source(file.path(
+    project_dir,
+    "scripts",
+    "03_compare_mia_gemelli_corrected.R"
+))
+```
+
+A successful run ends with `Comparison completed successfully.` and creates
+`results/frequency_010/comparison/`.
+
+## Step 6: render the complete report
+
+Open `joint-rpca-mia-gemelli-comparison.qmd` in RStudio and select **Render**.
+The report reads all validated result tables and figures; it does not repeat the
+Joint-RPCA computation.
+
+Alternatively, in the original PowerShell window:
+
+```powershell
+Set-Location $PROJECT_DIR
+quarto render joint-rpca-mia-gemelli-comparison.qmd
+```
+
+The output is `joint-rpca-mia-gemelli-comparison.html`. Tables and figures are
+embedded in this standalone file.
+
+## Correct execution order
+
+```text
+01 mia/R
+  → 00 environment and input check
+  → 02 Gemelli/Python
+  → 03 comparison/R
+  → render the QMD
+```
+
+## Common mistakes
+
+### Mixing Windows and WSL paths
+
+Use a Windows path in PowerShell and RStudio. Use `/mnt/<drive-letter>/...` in
+WSL. Do not use a WSL path in PowerShell or a Windows path inside WSL.
+
+### Opening a new terminal or R session
+
+PowerShell variables last only for the current PowerShell session. R variables
+last only for the current R session. If you open a new session, repeat only the
+short path-definition block for that environment.
+
+### Rendering before comparison
+
+The QMD stops if required tables or figures are missing. Complete scripts 01,
+02, and 03 before rendering.
+
+### Using the wrong prevalence unit
+
+Do not change `0.10` to `10` in `config.tsv`. R uses the proportion `0.10`; the
+Python runner converts it to Gemelli's `10%` representation.
+
+### Using nonzero count thresholds with IBDMDB
+
+The revised IBDMDB inputs are abundance-like and non-integer. This workflow
+permits prevalence filtering but stops if nonzero count thresholds are used.
+Validate count thresholds separately with an integer-count dataset.

vignettes/filterRPCAInput_validation/config.tsv

@@ -0,0 +1,2 @@
+run_label	data_variant	min_sample_count	min_feature_count	min_feature_frequency	n_components	max_iterations	n_test_samples	seed	expected_mia_version	expected_mia_commit
+frequency_010	main	0	0	0.10	3	5	10	42	1.21.6	25fa7bb51792566131f2bd8d07b4ebd1fde54b38