TileOmics: Self-Supervised Learning for Spatial RNA Expression Prediction

A comprehensive computational pipeline for analyzing histomorphological diversity and predicting spatial RNA expression across multiple human tissue types using self-supervised vision transformers and tile-based spatial analysis.

Paper:

---

Overview

TileOmics is a modular bioinformatics pipeline that integrates:

• Whole Slide Image (WSI) Processing: Automated segmentation and tiling of tissue images
• Self-Supervised Feature Extraction: Vision Transformer-based morphological feature learning
• Spatial Transcriptomics Integration: Mapping predicted tile-level RNA expression to spatial coordinates (Visium)
• Differential Expression Analysis: Gene expression pattern identification and enrichment analysis
• Image-Derived Phenotypes: Quantitative morphological measurements from tissue histology

---

Directory Structure

TileOmics/
├── preprocessing/                      # WSI preprocessing pipeline
│   ├── segmentation_patching/         # Tissue segmentation & tiling
│   │   ├── segmentation.py
│   │   ├── tiling.py
│   │   └── config.yaml
│   ├── features_extraction/           # ViT-based feature extraction
│   │   ├── extract_features.py
│   │   ├── extract_features_parallel_gpu.py
│   │   ├── target_feature_extraction.py
│   │   └── config files
│   └── dataset.py, preprocessing_utils.py
│
├── clustering_approach/               # Spatial correlation & analysis
│   ├── spatial_corr_visium_*.py      # Tile-to-Visium mapping
│   ├── main_grid*.py, vit_grid*.py  # Model architectures
│   ├── visualize*.py                 # Visualization scripts
│   ├── run_train*.sh                 # Training configurations
│   ├── run_visium*.sh                # Visium analysis workflows
│   ├── visual*.sh                    # Result visualization
│   └── he2rna.py, model.py
│
├── tiles_classification/              # Tile-level classification
│   ├── multiclass_tissue_segmentation.py
│   ├── fine_grained_multiclass_segmentation.py
│   └── define_clusters_kNN.py
│
├── differential_expression_analysis/  # DGE and enrichment
│   └── Analysis scripts and utilities
│
├── image_derived_phenotypes/          # Morphometric analysis
│   ├── compute_IDPs.py
│   └── compute_pivot_coordinates.py
│
├── utils/                             # Utility functions
│   └── Helper modules for data processing
│
├── preprocessing/                     # Helper modules
├── splits/                            # Data split definitions
├── training/                          # Training utilities
└── root level scripts:
    ├── train.py                       # Main training script
    ├── inference.py                   # Inference pipeline
    ├── main_vit.py                    # ViT model training
    ├── tformer_lin.py                # Transformer architecture
    ├── heatmaps.py                    # Heatmap generation
    ├── SSES_enrichment_analysis.py   # Enrichment analysis
    └── organize_bulk_rna_seq_data.py # Data organization

---

Quick Start

Installation

1. Clone the repository:

   git clone <repository-url>
   cd TileOmics_Clean

2. Install dependencies:

   # Using pip
   pip install -r requirements.txt
   
   # Or using conda
   conda env create -f convocals_py38_env.yaml
   conda activate convocals

3. Verify installation:

   python -c "import torch; print('PyTorch version:', torch.__version__)"

---

Usage Workflows

1. WSI Preprocessing (Segmentation & Tiling)

Extract tissue regions and create tiles from whole slide images:

cd preprocessing/segmentation_patching
python segmentation.py
# Configure parameters in config.yaml

python tiling.py
# Generates .h5 files with tile coordinates

Configuration: preprocessing/segmentation_patching/config.yaml

• Image input paths
• Tile size (e.g., 128x128, 256x256)
• Overlap parameters
• Output formats

---

2. Feature Extraction

Extract morphological features using self-supervised Vision Transformer:

cd preprocessing/features_extraction
python extract_features.py --config config_exfea.yaml

# Or for parallel GPU processing:
python extract_features_parallel_gpu.py --config config_exfea.yaml

Outputs:

• .pt files: Feature tensors (K, 384) where K = number of tiles
• .h5 files: Coordinates and metadata

---

3. Model Training

Train models to predict RNA expression from tile features:

# Using Grid-ViT architecture
bash clustering_approach/run_train_grid_filtered_noRibo_noAltChr.sh

# Using ClusterViT architecture
bash clustering_approach/run_train_clustervit_uni2h2048_500cluster_8h.sh

# Custom training
python train.py --config config.yaml

Key Scripts:

• clustering_approach/main_grid*.py: Grid-based ViT models
• clustering_approach/vit_grid*.py: Architecture variants
• train.py: Main training entry point

---

4. Spatial Correlation Analysis

Map tile-level predictions to Visium spots and compute correlations:

cd clustering_approach

python spatial_corr_visium_from_tiles_align_roi_cropping.py \
  --adata_path <path-to-adata> \
  --sample_id <sample-id> \
  --tiles_csv <tiles-coordinates.csv> \
  --training_csv <expression-matrix.csv> \
  --genes "CD68,ACTA2,CD34,GYPC" \
  --anchors "CD68,ACTA2" \
  --auto_orient \
  --clip_to_hull

Parameters:

• Orientation matching (anchor genes)
• Fine-tuning rotation/scale/translation
• Aggregation methods (gaussian, uniform, power, max)
• Optional ROI cropping and hull clipping

---

5. Visualization

Generate spatial maps and expression overlays:

# For HE2RNA predictions
bash clustering_approach/run_visium_he2rna_<sample>.sh

# For grid-based predictions
bash clustering_approach/visual_<sample>_big_tif_hvg_usergenes_gridD.sh

# Custom visualization
python clustering_approach/visualize_resume_2options_he2rna_big_tif.py \
  --data_config <config> \
  --output_dir <output-dir>

---

6. Differential Expression & Enrichment

Identify significant gene expression patterns:

python SSES_enrichment_analysis.py

# Generates:
# - Top differentially expressed genes
# - Gene set enrichment analysis (GSEA)
# - Functional annotation

---

7. Image-Derived Phenotypes (IDPs)

Compute morphometric features from tissue images:

python image_derived_phenotypes/compute_IDPs.py \
  --wsi_path <path-to-slides> \
  --output_dir <output-dir>

---

File Formats

Input Data

• WSI: .ndpi, .tif, .svs (supported by OpenSlide)
• Coordinates: .csv with x,y tile positions
• Expression: .csv with genes as columns, spots/tiles as rows
• Metadata: .pkl (AnnData objects for consistency)

Output Data

• Features: .pt (PyTorch tensors)
• Predictions: .csv (spot-level predictions)
• Visualizations: .png (spatial overlays, heatmaps)
• Results: .json (metadata), .pkl (full results)

Key Data Files Included

• macrophage_subtypes_all.mac_spec.arch.markers.tsv: Macrophage marker gene signatures
• organized_normalized_counts_with_symbols_feature_id.csv: Reference gene expression matrix

---

Model Architectures

Vision Transformer (ViT-S)

• Pre-trained on 1.7M histology patches from 23 GTEx tissues
• Feature dimension: 384-D
• Self-supervised training (DINO)

Grid-ViT

• Tile-based spatial modeling with attention
• Variants with/without multi-head attention
• Positional encoding options (learned, fixed)
• Query aggregation methods (attention pooling, max pooling)

ClusterViT

• Clustering-aware attention mechanism
• Configurable number of clusters (e.g., 500)
• Enhanced feature aggregation

---

Configuration Files

Preprocessing Config

preprocessing/segmentation_patching/config.yaml:

• Input slide paths
• Segmentation thresholds
• Tile dimensions and overlap
• Output directory structure

Feature Extraction Config

preprocessing/features_extraction/config_exfea.yaml:

• Model checkpoint paths
• Batch size, device (GPU/CPU)
• Feature output format

Training Config

Create custom config with:

model:
  arch: grid_vit_16h
  num_heads: 16
  embedding_dim: 384
  
training:
  batch_size: 32
  epochs: 100
  learning_rate: 1e-4
  
data:
  train_splits: [0, 1, 2]
  val_splits: [3]
  test_splits: [4]

---

Performance Benchmarks

Expected performance varies by model and dataset:

• Pearson correlation (tile → Visium predictions): 0.3-0.6 depending on gene
• Top-1000 genes: Higher correlation for highly expressed genes
• Grid-ViT vs ClusterViT: ClusterViT typically 2-5% improvement

See clustering_approach/ for detailed result files:

• *_Test_Results.csv: Per-fold performance metrics
• *_Test_Metrics.csv: Aggregated metrics
• Summary_Comparison*.csv: Model comparison tables

---

Advanced Features

Tile Filtering

• Remove white/empty patches
• Expression-based filtering
• Library size normalization

Spatial Alignment

• Automatic orientation detection using anchor genes
• Fine-tuning with ICP-like optimization
• Hull-based ROI clipping

Multi-Scale Analysis

• Feature aggregation at different scales
• Neighbor-based feature expansion
• Hierarchical clustering

Batch Processing

• GPU-accelerated feature extraction
• Parallel slide processing
• Distributed training (optional)

---

Dependencies

Key Python packages:

torch==1.12.1
torchvision==0.13.1
numpy==1.21.6
pandas==1.2.4
scikit-learn==1.3.0
scipy==1.4.1
h5py==3.8.0
opencv-python==4.7.0.72
Pillow==10.0.0
PyYAML==6.0.1
tensorboardX==2.6.2.2
tqdm==4.64.0

See requirements.txt for complete list with versions.

---

Troubleshooting

Common Issues

1. Out of Memory during feature extraction

   • Reduce batch size in config
   • Use extract_features_parallel_cpu.py for CPU processing
   • Process slides in smaller chunks

2. Coordinate mismatch in spatial correlation

   • Verify tile coordinate CSV format (should match output of tiling.py)
   • Check image orientation (may need flip/rotation)
   • Use --auto_orient flag for automatic detection

3. Missing dependencies

   pip install -r requirements.txt --upgrade

4. GPU not detected

   python -c "import torch; print(torch.cuda.is_available())"

---

Project Structure by Analysis Type

For WSI Analysis Only

1. Run preprocessing workflow
2. Extract tile-level features
3. Analyze tile statistics

For Visium Integration

1. Complete preprocessing + feature extraction
2. Run spatial correlation analysis
3. Visualize predictions on tissue

For RNA Prediction

1. Prepare training splits
2. Train tile-to-RNA model
3. Generate spot-level predictions
4. Evaluate against ground truth

For Enrichment Analysis

1. Identify differentially expressed genes
2. Run GSEA with marker gene signatures
3. Map genes to pathways and biological processes

---

Output Interpretation

Correlation Matrices

• <sample>_spot_correlations.csv: Pearson/Spearman correlations between predictions and ground truth
• <sample>_pearson_bar.png: Visualization of top genes by correlation

Prediction Files

• <sample>_<gene>_spot_preds.csv: Predicted expression at each Visium spot
• maps_<sample>_<gene>.png: Spatial map showing predictions

Metadata

• <sample>_meta.json: Alignment parameters, gene list, normalization info
• alignment_<sample>.png: Diagnostic image showing tile-to-Visium alignment

---

Citation

If you use TileOmics in your research, please cite:

@preprint{cisternino2023selfsupervised,
  title={Self-supervised learning for characterising histomorphological diversity and spatial RNA expression prediction across 23 human tissue types},
  author={Cisternino, Francesco and Ometto, Sara and Chatterjee, Soumick and Giacopuzzi, Edoardo and Levine, Adam P. and Glastonbury, Craig A.},
  journal={bioRxiv},
  year={2023},
  doi={10.1101/2023.08.22.554251}
}

---

License

See LICENSE file for details.

---

Support & Contribution

For questions, bug reports, or feature requests, please open an issue in the repository.

Contributions are welcome! Please follow the code style and include tests for new features.

---

Authors & Acknowledgments

• Primary Authors: Francesco Cisternino, Sara Ometto, Craig A. Glastonbury
• Computational Contributors: Soumick Chatterjee, Edoardo Giacopuzzi
• Supervision: Adam P. Levine, Craig A. Glastonbury

---

Last Updated: June 2026
Repository Version: 1.0