# Tokenisation (genes, tiles and sequence as tokens)

Source: https://onco.cc/terms/tokenisation/  
OnCo record `tokenisation` (Term). Data CC BY-NC 4.0, attribute "Data from OnCo (onco.cc)"; commercial use needs a licence.

## TL;DR

Tokenisation is how raw input is chopped into the discrete pieces a transformer reads: words into subwords, a slide into tiles, an expression profile into genes ranked or binned by level.

## Summary

Lexical tokenisation converts text into meaningful tokens of defined categories (Wikipedia); language models use subword schemes such as byte-pair encoding. Biology models must invent their tokens: expression can be tokenised by ranking the genes of a cell (Geneformer), keeping the top N, or binning values; DNA by nucleotides or k-mers. Grounding a gene token in its sequence or protein embedding (as UCE does with ESM2) rather than an arbitrary ID lets a model generalise to unseen genes and species.

## Fields

- Kind: Term
- Last checked: 2026-09-24
- Also known as: tokenisation; tokenization; gene tokenisation; gene tokens; rank-based tokenisation; value binning; byte-pair encoding; universal tokenisation; semantic token grounding
- Tags: cansim-terms

## Notes

- Listed in the CanSim terms map 1.0.0 (docs/onco/terms.json, generated 2026-09-24), CC BY 4.0, attribution: CanSim project, an open, public-data-first cancer foundation-model programme; CanSim page path /terms/tokenisation.

## Sources

- Wikipedia: https://en.wikipedia.org/wiki/Lexical_analysis
- Wikipedia: byte-pair encoding: https://en.wikipedia.org/wiki/Byte-pair_encoding
- Wikipedia: https://en.wikipedia.org/wiki/Lexical_analysis

## Connected records

- terms: [Cancer AI vocabulary (CanSim terms map)](https://onco.cc/terms/cancer-ai-vocabulary/), [Embedding (learned representation)](https://onco.cc/terms/embedding/), [Genomic and protein language models: Evo 2, Enformer, ESM](https://onco.cc/terms/genomic-and-protein-language-models/), [Masked autoencoders and masked gene modelling](https://onco.cc/terms/masked-modelling/), [Single-cell and transcriptome foundation models: UCE, GeneCompass, BulkFormer, BulkRNABert](https://onco.cc/terms/single-cell-foundation-models/), [Transformer and attention](https://onco.cc/terms/transformer-architecture/)

---
JSON: https://onco.cc/api/v1/entities/tokenisation.json