# Transformer and attention

Source: https://onco.cc/terms/transformer-architecture/  
OnCo record `transformer-architecture` (Term). Data CC BY-NC 4.0, attribute "Data from OnCo (onco.cc)"; commercial use needs a licence.

## TL;DR

A transformer is a neural network that turns its input into a sequence of tokens and lets every token weigh every other through attention, the architecture behind language models and most new biology models.

## Summary

The transformer is a family of neural network architectures based on multi-head attention: input data is converted to tokens, each token to a vector, and attention determines the importance of each token relative to the others (Wikipedia). Genes, tiles, cells or whole modalities can be tokens, and a special summary token (CLS) can stand for the patient. The Perceiver variant routes inputs through a small latent array so that arbitrary and very long inputs stay affordable. Attention weights are often read as explanations, which they only loosely are.

## Fields

- Kind: Term
- Last checked: 2026-09-24
- Also known as: transformer architecture; transformer model; transformer encoder; attention mechanism; self-attention; multi-head attention; attention weights; fusion transformer; Perceiver
- Tags: cansim-terms

## Notes

- Listed in the CanSim terms map 1.0.0 (docs/onco/terms.json, generated 2026-09-24), CC BY 4.0, attribution: CanSim project, an open, public-data-first cancer foundation-model programme; CanSim page path /terms/transformer.

## Sources

- Wikipedia: https://en.wikipedia.org/wiki/Transformer_(deep_learning)
- Wikipedia: attention (machine learning): https://en.wikipedia.org/wiki/Attention_(machine_learning)
- Wikipedia: Perceiver: https://en.wikipedia.org/wiki/Perceiver
- Wikipedia: https://en.wikipedia.org/wiki/Transformer_(deep_learning)

## Connected records

- terms: [Attention-based multiple-instance learning (ABMIL, CLAM)](https://onco.cc/terms/abmil/), [Autoregressive (next-token) modelling](https://onco.cc/terms/autoregressive-modelling/), [Cancer AI vocabulary (CanSim terms map)](https://onco.cc/terms/cancer-ai-vocabulary/), [Embedding (learned representation)](https://onco.cc/terms/embedding/), [Foundation model](https://onco.cc/terms/foundation-model/), [Multimodal fusion (early, late, modality dropout)](https://onco.cc/terms/multimodal-fusion/), [Tokenisation (genes, tiles and sequence as tokens)](https://onco.cc/terms/tokenisation/)

---
JSON: https://onco.cc/api/v1/entities/transformer-architecture.json