Loading
A transformer is a neural network that turns its input into a sequence of tokens and lets every token weigh every other through attention, the architecture behind language models and most new biology models.
The transformer is a family of neural network architectures based on multi-head attention: input data is converted to tokens, each token to a vector, and attention determines the importance of each token relative to the others (Wikipedia). Genes, tiles, cells or whole modalities can be tokens, and a special summary token (CLS) can stand for the patient. The Perceiver variant routes inputs through a small latent array so that arbitrary and very long inputs stay affordable. Attention weights are often read as explanations, which they only loosely are.
Shares Multimodal fusion (early, late, modality dropout), Embedding (learned representation), Cancer AI vocabulary (CanSim terms map) and the tag cansim-terms.
Shares Autoregressive (next-token) modelling, Embedding (learned representation), Cancer AI vocabulary (CanSim terms map) and the tag cansim-terms.
Shares Multimodal fusion (early, late, modality dropout), Embedding (learned representation), Cancer AI vocabulary (CanSim terms map) and the tag cansim-terms.
Shares Tokenisation (genes, tiles and sequence as tokens), Autoregressive (next-token) modelling, Cancer AI vocabulary (CanSim terms map) and the tag cansim-terms.
Shares Attention-based multiple-instance learning (ABMIL, CLAM), Foundation model, Cancer AI vocabulary (CanSim terms map) and the tag cansim-terms.
Shares Embedding (learned representation), Foundation model, Cancer AI vocabulary (CanSim terms map) and the tag cansim-terms.
Shares Tokenisation (genes, tiles and sequence as tokens), Foundation model, Cancer AI vocabulary (CanSim terms map) and the tag cansim-terms.
Shares Multimodal fusion (early, late, modality dropout), Cancer AI vocabulary (CanSim terms map) and the tag cansim-terms.