{"entity":{"id":"transformer-architecture","kind":"term","name":"Transformer and attention","aka":["transformer architecture","transformer model","transformer encoder","attention mechanism","self-attention","multi-head attention","attention weights","fusion transformer","Perceiver"],"tldr":"A transformer is a neural network that turns its input into a sequence of tokens and lets every token weigh every other through attention, the architecture behind language models and most new biology models.","summary":"The transformer is a family of neural network architectures based on multi-head attention: input data is converted to tokens, each token to a vector, and attention determines the importance of each token relative to the others (Wikipedia). Genes, tiles, cells or whole modalities can be tokens, and a special summary token (CLS) can stand for the patient. The Perceiver variant routes inputs through a small latent array so that arbitrary and very long inputs stay affordable. Attention weights are often read as explanations, which they only loosely are.","asOf":"2026-09-24","wikipedia":"https://en.wikipedia.org/wiki/Transformer_(deep_learning)","links":[{"label":"Wikipedia: attention (machine learning)","url":"https://en.wikipedia.org/wiki/Attention_(machine_learning)"},{"label":"Wikipedia: Perceiver","url":"https://en.wikipedia.org/wiki/Perceiver"},{"label":"Wikipedia","url":"https://en.wikipedia.org/wiki/Transformer_(deep_learning)"}],"tags":["cansim-terms"],"related":["cancer-ai-vocabulary"],"cancers":[],"sections":[],"technologies":[],"targets":[],"drugs":[],"companies":[],"institutions":[],"pathways":[],"terms":["tokenisation","embedding","abmil","foundation-model"],"trials":[],"people":[],"bottlenecks":[],"keyPapers":[],"journals":[],"dependsOn":[],"notes":["Listed in the CanSim terms map 1.0.0 (docs/onco/terms.json, generated 2026-09-24), CC BY 4.0, attribution: CanSim project, an open, public-data-first cancer foundation-model programme; CanSim page path /terms/transformer."],"provenance":{"editedBy":"OnCo CanSim terms wave (Wikipedia summaries, standards and project pages, GDC and FDA pages, Europe PMC)","editedOn":"2026-09-24","note":"CanSim terms map 1.0.0 (docs/onco/terms.json, generated 2026-09-24), CC BY 4.0, attribution: CanSim project, an open, public-data-first cancer foundation-model programme"},"category":"Methods and models"},"route":"/terms/transformer-architecture/","neighbours":{"term":[{"id":"abmil","kind":"term","name":"Attention-based multiple-instance learning (ABMIL, CLAM)","route":"/terms/abmil/"},{"id":"autoregressive-modelling","kind":"term","name":"Autoregressive (next-token) modelling","route":"/terms/autoregressive-modelling/"},{"id":"cancer-ai-vocabulary","kind":"term","name":"Cancer AI vocabulary (CanSim terms map)","route":"/terms/cancer-ai-vocabulary/"},{"id":"embedding","kind":"term","name":"Embedding (learned representation)","route":"/terms/embedding/"},{"id":"foundation-model","kind":"term","name":"Foundation model","route":"/terms/foundation-model/"},{"id":"multimodal-fusion","kind":"term","name":"Multimodal fusion (early, late, modality dropout)","route":"/terms/multimodal-fusion/"},{"id":"tokenisation","kind":"term","name":"Tokenisation (genes, tiles and sequence as tokens)","route":"/terms/tokenisation/"}]}}