# Multimodal fusion (early, late, modality dropout)

Source: https://onco.cc/terms/multimodal-fusion/  
OnCo record `multimodal-fusion` (Term). Data CC BY-NC 4.0, attribute "Data from OnCo (onco.cc)"; commercial use needs a licence.

## TL;DR

Multimodal fusion combines several kinds of data about one patient (slides, expression, mutations, clinical variables) in one model; modality dropout randomly hides modalities during training so the model still works when some are missing.

## Summary

Multimodal learning integrates and processes multiple data types, or modalities, in one deep model (Wikipedia). Patient data is rarely complete, so fusion models mask absent modalities in attention and use modality dropout, an application of dropout regularisation (Wikipedia), to stay robust. PORPOISE fused histology and genomics for pan-cancer prognosis; the recurring finding is that when one modality (expression) already carries the signal, fusion adds little, and the honest test is each modality alone against the combination.

## Fields

- Kind: Term
- Last checked: 2026-09-24
- Also known as: multimodal fusion; multimodal model; multimodal learning; modality fusion; early fusion; late fusion; modality dropout; attention masking over modalities; missing modalities
- Tags: cansim-terms

## Notes

- Listed in the CanSim terms map 1.0.0 (docs/onco/terms.json, generated 2026-09-24), CC BY 4.0, attribution: CanSim project, an open, public-data-first cancer foundation-model programme; CanSim page path /terms/multimodal-fusion.

## Sources

- Wikipedia: https://en.wikipedia.org/wiki/Multimodal_learning
- Wikipedia: dropout: https://en.wikipedia.org/wiki/Dropout_(neural_networks)
- Chen et al., PORPOISE: pan-cancer integrative histology-genomic analysis via multimodal deep learning (Cancer Cell 2022): https://doi.org/10.1016/j.ccell.2022.07.004
- Wikipedia: https://en.wikipedia.org/wiki/Multimodal_learning

## Connected records

- terms: [Ablation study and multi-task heads](https://onco.cc/terms/ablation-study/), [Cancer AI vocabulary (CanSim terms map)](https://onco.cc/terms/cancer-ai-vocabulary/), [Contrastive learning (InfoNCE)](https://onco.cc/terms/contrastive-learning/), [Principal component analysis (PCA) as a feature compressor](https://onco.cc/terms/pca/), [Transformer and attention](https://onco.cc/terms/transformer-architecture/)

---
JSON: https://onco.cc/api/v1/entities/multimodal-fusion.json