# Patient-level multimodal foundation models for treatment selection

Source: https://onco.cc/ideas/idea-multimodal-foundation-model/  
OnCo record `idea-multimodal-foundation-model` (Idea). Data CC BY-NC 4.0, attribute "Data from OnCo (onco.cc)"; commercial use needs a licence.

## TL;DR

Train one AI on scans, pathology slides, genomics and treatment outcomes pooled across patients, including completed phase 3 trials, so it can predict which treatment will work for a new patient. Pathology and radiology models already exist separately; combining them with genomic and trial outcome data is the untested step.

## Summary

Train one AI on scans, slides, genomics and outcomes pooled across patients, including completed phase 3 trials, so that it can predict, for a new patient, which treatment will work. Pathology and radiology foundation models exist separately; combining them with genomic and clinical data, as Tempus AI, Owkin and CanSim-style efforts are doing, is the next step, and ArteraAI showed that single-modality models can be predictive. Adding outcomes from randomised trials would allow causal treatment-effect estimation, for example TROP2 ADC versus chemotherapy. The test is to train on completed phase 3 datasets with sponsors, validate on held-out trials and then run a prospective biomarker-stratified trial. With preclinical evidence, it addresses the bottlenecks Data silos and AI that is built but not validated or deployed.

## Fields

- Kind: Idea
- Last checked: 2026-09-04
- Hypothesis: A multimodal model trained on trial cohorts predicts benefit from a specific therapy (e.g., TROP2 ADC vs chemotherapy) with clinically useful discrimination beyond current biomarkers.
- Rationale: ArteraAI showed single-modality models can be predictive; adding modalities and outcomes from randomised trials allows causal treatment-effect estimation.
- Proposed test: Train on completed phase 3 datasets (with sponsors), validate on held-out trials; prospective biomarker-stratified trial.
- Maturity: preclinical-evidence

## Sources

- Chen et al., Towards a general-purpose foundation model for computational pathology (Nature Medicine 2024): https://doi.org/10.1038/s41591-024-02857-3

## Connected records

- technologies: [AI trial matching & clinical decision support](https://onco.cc/technologies/ai-trial-matching/), [Digital pathology & AI](https://onco.cc/technologies/digital-pathology-ai/), [Pathology & radiology foundation models](https://onco.cc/technologies/pathology-foundation-model/)
- drugs: [ArteraAI Breast](https://onco.cc/drugs/artera-ai-breast/)
- companies: [Artera](https://onco.cc/companies/artera/), [Owkin](https://onco.cc/companies/owkin/), [Tempus AI](https://onco.cc/companies/tempus/)
- key papers: [AlphaFold 2: predicting protein structures to near-experimental accuracy](https://onco.cc/key-papers/paper-alphafold2-jumper-nature-2021/), [Towards a general-purpose foundation model for computational pathology](https://onco.cc/key-papers/paper-chen-nat-med/)
- bottlenecks: [AI that is built but not validated or deployed](https://onco.cc/bottlenecks/b-ai-validation/), [Data silos](https://onco.cc/bottlenecks/b-data-silos/), [No one can predict who responds to immunotherapy](https://onco.cc/bottlenecks/b-immunotherapy-response/)
- ideas: [A federated learning consortium of cancer centres that jointly own the models](https://onco.cc/ideas/idea-fund-federated-learning-consortium/), [A pre-competitive consortium to train a shared multimodal cancer foundation model](https://onco.cc/ideas/idea-data-precompetitive-cancer-foundation-model/), [A registry of external validation datasets for cancer AI models, with mandatory reporting](https://onco.cc/ideas/idea-tr2-ai-external-validation-registry/), [Digital twins for treatment selection, validated by predicting before observing](https://onco.cc/ideas/idea-data-digital-twin-predict-then-observe/), [Pool every immunotherapy trial's biomarker data into one commons](https://onco.cc/ideas/idea-bio2-io-biomarker-data-commons/), [Whole-patient digital twins validated in prospective randomised trials](https://onco.cc/ideas/idea-moon-validated-digital-twins/)
- roadmaps: [AI in oncology roadmap: pattern readers → foundation models → agents in the workflow](https://onco.cc/roadmaps/ai-oncology-roadmap/), [AI in the oncology clinic: from narrow cleared tools to multimodal decision support](https://onco.cc/roadmaps/ai-oncology-clinic/), [Virtual cell roadmap: from bulk omics to a predictive model of a cancer cell](https://onco.cc/roadmaps/virtual-cell/)
- terms: [Cancer AI vocabulary (CanSim terms map)](https://onco.cc/terms/cancer-ai-vocabulary/), [Foundation model](https://onco.cc/terms/foundation-model/)

---
JSON: https://onco.cc/api/v1/entities/idea-multimodal-foundation-model.json