AI-powered histopathological analysis of Oral Squamous Cell Carcinoma from H&E-stained biopsy slides
Problem · Solution · Features · Architecture · Datasets · Quick Start · Training · Roadmap · License
📸 → 🧠 → 📊 — From microscope photo to structured clinical report
Oral Squamous Cell Carcinoma (OSCC) accounts for 90% of oral cancers, with ~377,000 new cases annually worldwide. Survival rates exceed 80% when detected early but plummet below 30% in late stages. The challenge:
- Pathologist shortage: In low/middle-income countries, the pathologist-to-patient ratio can be 1:1,000,000+
- Delayed diagnoses: Biopsies often take 2-4 weeks for processing and review
- Subjective grading: Inter-pathologist agreement on tumor grading is moderate (κ ≈ 0.5-0.7)
OralPath is a non-commercial research project that builds a mobile-first AI diagnostic assistant to address these gaps.
⚠️ This is NOT a replacement for a pathologist. It is a decision-support tool for resource-limited settings where dedicated oral pathology departments are unavailable. All outputs must be reviewed by a qualified pathologist before any clinical decision-making.
A multi-stage AI pipeline that classifies and grades OSCC from H&E-stained biopsy slides photographed through a standard microscope:
| Stage | Task | Classes | Target |
|---|---|---|---|
| Stage 1 → | Binary Detection | Normal / OSCC | Sensitivity ≥ 0.95 |
| Stage 2 → | Grading | Normal / OSMF / WD / MD / PD | Macro-F1 ≥ 0.65 |
| Stage 3 (v1.2) → | Segmentation | Epithelium / Stroma / TILs / Collagen | mIoU ≥ 0.85 |
Results are surfaced through an Android app (Jetpack Compose) with structured reports, confidence scores, and full offline capability via TFLite.
🔬 AI Pipeline
- Frozen foundation backbones — UNI (ViT-L/14), CTransPath, or EfficientNetB3 extract rich features without fine-tuning
- Multi-source training — Combines 5 public datasets (~2,084 cases) for robust generalization
- Multiple Instance Learning (MIL) — Whole-slide classification from patch embeddings using Attention-Top-K pooling
- Stain normalization — Macenko & Reinhard algorithms correct cross-lab H&E staining variation
- Ordinal classification loss — Exploits natural ordering of WD < MD < PD grades
- Focal loss + weighted sampling — Handles severe class imbalance in rare grades
📱 Android App
- Jetpack Compose UI — Modern, declarative UI with Material 3 theming
- 5-class result display — Predicted class badge, confidence score, probability distribution bars
- CameraX integration (planned) — Capture slide photos directly from app
- Room/SQLite storage (planned) — Patient case history locally
- PDF report export (planned) — Structured clinical report with disclaimer
- TFLite INT8 on-device (planned) — Offline inference without connectivity
🚀 Training Infrastructure
- Google Colab — VS Code extension for GPU training with automatic dataset setup
- Kaggle Kernels — Pre-submitted kernels with bundled data; GPU probe detects T4/P100
- Lightning AI Studio — Persistent cloud GPU with multi-session resumable training
- PowerShell automation — One-command scripts for submit/monitor/download across all runtimes
- Resumable checkpoints — Training can pause/resume across Colab session limits
┌─────────────────────────────────────────────────────────────┐
│ 📱 Android App │
│ Kotlin · Jetpack Compose · Coil · Room · CameraX │
│ ┌────────┐ ┌──────────┐ ┌────────┐ ┌─────────────┐ │
│ │Camera │──▶│View/Edit │──▶│Infer │──▶│Results + PDF│ │
│ │Capture │ │Slide Img │ │(API/TFL)│ │Report │ │
│ └────────┘ └──────────┘ └────────┘ └─────────────┘ │
│ │ │
└──────────────────────────────────────┼──────────────────────┘
│ REST / ONNX / TFLite
┌──────────────────────────────────────┼──────────────────────┐
│ 🧠 Inference Server │ │
│ FastAPI · ONNX Runtime · Python │ │
│ ┌──────────────┐ ┌──────────────────────────┐ │
│ │ Preprocess │──▶│ Stage 1 → Stage 2 → │ │
│ │(Normalize · │ │ Detection + Grading │ │
│ │ Resize · │ │ Ensemble │ │
│ │ Patch Extract)│ └──────────────────────────┘ │
│ └──────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
┌──────────────────────────────────────┼──────────────────────┐
│ 🏋️ Training Pipeline │ │
│ PyTorch · timm · transformers │ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Stage 1 │ │Stage 2 │ │Stage 3 │ │
│ │Detection │ │Grading │ │Segment │ (v1.2) │
│ └──────────┘ └──────────┘ └──────────┘ │
│ Colab · Kaggle · Lightning AI · Local GPU │
└─────────────────────────────────────────────────────────────┘
📁 oralpath/
├── 📱 android/ # Kotlin Android app (Jetpack Compose)
│ ├── app/src/main/java/com/oralpath/
│ │ ├── MainActivity.kt
│ │ ├── ui/result/ # Level 1 result screen
│ │ └── ui/theme/ # Material 3 color/typography
│ ├── build.gradle.kts
│ └── settings.gradle.kts
│
├── 🧠 model/ # Python ML pipeline
│ ├── config.py # Centralized paths and hyperparameters
│ ├── data/
│ │ ├── datasets/ # Multi-source dataset registry (5 sources)
│ │ │ ├── orchid_source.py, gdc_source.py
│ │ │ ├── multi_oscc_source.py, ndb_ufes_source.py
│ │ │ ├── wsi_patch_extraction.py
│ │ │ └── orchestrator.py # Unified CLI entry point
│ │ ├── preprocessing/ # Dataset loader + zip-backed I/O
│ │ └── qa/ # Split builders, stain norm, integrity checks
│ ├── training/
│ │ ├── stage1_detection/ # Binary OSCC/Normal classifier
│ │ ├── stage2_grading/ # 5-class grading with resumable training
│ │ └── mil/ # MIL production pipeline (Attention-Top-K)
│ ├── evaluation/ # Metrics + case-level evaluation
│ ├── inference/ # CLI inference wrapper (JSON contract)
│ ├── kaggle/ # Kaggle kernel source code + bundles
│ ├── external/ # CTransPath, ORCHID, OralPatho references
│ └── notebooks/ # Colab bootstrap notebook
│
├── 📄 docs/ # Comprehensive documentation
├── 📜 scripts/ # PowerShell + Python automation
├── 🧪 tests/ # pytest test suite
└── 📦 external/ # Git submodules (OralPatho reference)
OralPath combines 5 public datasets totaling ~2,084 cases — one of the most comprehensive OSCC training collections for a research project:
| Dataset | Type | Size | Grade Labels | Use | License |
|---|---|---|---|---|---|
| Kaggle OSCC | Patches | 1,224 (230 patients) | Normal / OSCC | Stage 1 binary | CC BY 4.0 |
| ORCHID | Patches | 23,000+ | Normal / OSMF / WD / MD / PD | Stage 2 grading | CC BY 4.0 |
| TCGA-OSCC | WSIs | ~257 | Grade info | MIL (Stage 2+) | GDC Open |
| CPTAC-OSCC | WSIs | ~165 | Grade info | MIL (Stage 2+) | GDC Open |
| Multi-OSCC | WSIs | 1,325 | Grade info | MIL (Stage 2+) | CC BY-SA 4.0 |
| NDB-UFES | Patches | 3,763 | Tissue type | Segmentation (v1.2) | CC BY 4.0 |
| Model | Role | License |
|---|---|---|
| UNI (Mahmood Lab, ViT-L/14) | Primary backbone | CC BY-NC-ND 4.0 |
| CTransPath | Secondary backbone | GPLv3-NC |
| CONCH (Mahmood Lab) | Research benchmark | CC BY-NC-ND 4.0 |
| EfficientNetB3 | Fallback / benchmark | Apache 2.0 |
- Python 3.10+
- Git
- Android Studio (for mobile app development)
# Clone the repo
git clone https://github.com/ORION2809/OSCC.git
cd oralpath
# Create and activate virtual environment
python -m venv .venv
# Windows:
.venv\Scripts\activate
# macOS/Linux:
# source .venv/bin/activate
# Install dependencies
pip install -r requirements.txt# Unified dataset download + manifest generation
python scripts/download_datasets.py
# Verify manifests (dry run — no extraction needed)
python model/data/preprocessing/dataset_loader.pypython model/inference/stage2_predict.py \
--image path/to/slide_image.png \
--checkpoint path/to/stage2_checkpoint.pt# Open android/ in Android Studio
# Sync Gradle → Select device/emulator → Run# Stage 1 — Binary Detection (OSCC vs Normal)
python model/training/stage1_detection/train.py \
--config model/training/stage1_detection/config.yaml
# Stage 2 — 5-Class Grading
python model/training/stage2_grading/train.py \
--config model/training/stage2_grading/config.yaml# 1. Install the "Google Colab" VS Code extension
# 2. Open model/notebooks/oralpath_colab_bootstrap.ipynb
# 3. Click "Connect to Colab Runtime" — select a GPU backend
# 4. Run cells to mount Drive, clone repo, install deps, launch training📖 See docs/COLAB_SETUP.md for the full VS Code + Colab workflow.
# Submit Stage 2 training kernel
.\scripts\run_kaggle_stage2.ps1
# Monitor progress
.\scripts\kaggle_stage2_status.ps1
# Download results when complete
.\scripts\download_kaggle_stage2_outputs.ps1⚡ Lightning AI Studio (Recommended for Production)
# Set your SSH target
$env:LIGHTNING_SSH_TARGET = "s_xxx@ssh.lightning.ai"
# Launch production MIL training
.\scripts\run_lightning_mil_production.ps1
# Monitor
.\scripts\lightning_mil_status.ps1📖 See docs/LIGHTNING_TRAINING_HANDOFF.md for the canonical training guide.
The most advanced experiment in the project uses Multiple Instance Learning for case-level classification:
- Embedding Extraction — UNI backbone produces 1024-dim patch embeddings from 5 data sources
- Top-K Attention Pooling — Learns to weight the most diagnostically-relevant patches
- Focal Loss — Handles class imbalance across grades
- 5-Fold Stratified Cross-Validation — Per-source disaggregated metrics track generalization
# Extract embeddings
python model/training/mil/extract_embeddings.py
# Train MIL model
python model/training/mil/train_mil_v2.py \
--config model/training/mil/config.lightning_mil_production.yaml# Run the full test suite
pytest tests/ -v
# Run specific test modules
pytest tests/test_multi_source_pipeline.py -v
pytest tests/test_data_qa.py -v
pytest tests/test_mil_port.py -v| Component | Status | Notes |
|---|---|---|
| Stage 1 (Binary Detection) | ✅ Prototype | Trained as background artifact |
| Stage 2 (Grading) | 🔄 Iterating | Run B: macro-F1 0.48; targeting ≥ 0.65 |
| Multi-source MIL | 🟢 Pipeline ready | Awaiting production run on Lightning AI |
| Stage 3 (Segmentation) | ⏳ Deferred | Target v1.2 |
| Android App | 🟡 Level 1 skeleton | Mock result screen complete |
| Inference API | 📝 Spec complete | JSON contract defined in docs |
| ONNX/TFLite Export | ⏳ Planned | |
| Kaggle Integration | GPU detection bug; Lightning AI preferred |
| Class | Precision | Recall | F1-Score |
|---|---|---|---|
| Normal | 0.65 | 0.74 | 0.70 |
| OSMF | 0.41 | 0.48 | 0.44 |
| WD-OSCC | 0.47 | 0.57 | 0.51 |
| MD-OSCC | 0.24 | 0.12 | 0.16 |
| PD-OSCC | 0.56 | 0.52 | 0.54 |
| Macro Avg | 0.47 | 0.49 | 0.48 |
Current focus: Improving MD-OSCC recall via focal loss, weighted sampling, and ordinal loss.
v0.1 ── Research scaffold (this release)
│ • Repository structure, model interfaces, Android skeleton
│ • Multi-source dataset pipeline (5 sources)
│ • MIL production pipeline (Attention-Top-K)
│ • Kaggle + Colab + Lightning AI training automation
│
v1 ──── Detection + grading pilot
│ • Stage 1 binary model (sensitivity ≥ 0.95)
│ • Stage 2 grading model (macro-F1 ≥ 0.65)
│ • Android: camera capture, inference display, case history
│ • ONNX export + FastAPI inference server
│
v1.2 ── Segmentation
│ • Stage 3: MobileViT for tissue component ID
│ • TILs analysis, tumor-stroma ratio
│
v2 ──── Clinical pilot
• Multi-centric validation study
• Regulatory pathway assessment (IRB, CE/FDA)
• On-device TFLite INT8 inference
| Document | Description |
|---|---|
ARCHITECTURE.md |
System architecture, inference contract JSON, API spec |
VISION.md |
Product vision, problem statement, target users, success metrics |
MODEL_CARD.md |
Model descriptions, intended use, limitations, fairness |
DATASETS.md |
Complete dataset inventory, licenses, download instructions |
IMPLEMENTATION_PLAN.md |
Workstreams, acceptance gates, timeline |
COLAB_SETUP.md |
VS Code + Colab training workflow |
KAGGLE_STAGE2_TRAINING.md |
Kaggle kernel submission runbook |
LIGHTNING_TRAINING_HANDOFF.md |
Canonical Lightning AI training guide |
ORALPATHO_ADAPTATION.md |
Notes on adapting OralPatho reference architecture |
DATA_QA_REBUILD_REPORT.md |
Dataset quality assurance and rebuild |
This is a non-commercial research project. Contributions are welcome in the spirit of open science:
- Fork the repo
- Create a feature branch:
git checkout -b feature/my-idea - Commit your changes:
git commit -m 'Add my idea' - Push:
git push origin feature/my-idea - Open a Pull Request
Areas where help is especially valuable:
- Improving MD-OSCC grading recall
- Adding new dataset sources
- Android UI development
- ONNX/TFLite export and optimization
- Documentation and test coverage
- OralPatho — Reference architecture adapted for MIL (MIT license) — Repository
- Mahmood Lab — UNI and CONCH foundation models — UNI | CONCH
- CTransPath — Transformer-based pathological feature extractor — Repository
- Dataset authors — Kaggle OSCC (Tabassum et al.), ORCHID (NishaChaudhary23), TCGA, CPTAC, Multi-OSCC (Cavalcante et al.), NDB-UFES
- Google Colab, Kaggle, and Lightning AI for GPU compute resources
- Application code: MIT or Apache 2.0 (to be finalized)
- Models & datasets: Varies per component — see
docs/DATASETS.mdfor per-component licensing - Project status: Non-commercial research. NC-licensed models (UNI, CONCH, CTransPath) are used under their research terms
This application is intended for RESEARCH USE ONLY.
It does not provide a definitive medical diagnosis. The software is not cleared or approved by the FDA, CE, or any other regulatory body for clinical use. All outputs must be reviewed by a qualified pathologist before any clinical decision-making. The developers assume no liability for any clinical decisions made based on this software.