ideasIdea
Score every model system on how well it predicted real trial results
No one keeps score of which laboratory models actually predicted what happened in patients. A public scoreboard would show which models to trust.
Protein structure prediction improved rapidly once CASP created a blinded, periodic benchmark. An oncology equivalent would take drugs with known but embargoed clinical outcomes, ask model owners (organoids, PDX, chips, in silico) to submit blinded predictions of response rate or ranking, and publish accuracy by model class. Over time this creates evidence for which systems merit regulatory and investment weight.
Hypothesis
Blinded benchmarking reveals large and reproducible differences between model classes in predicting clinical response rates, and participation improves accuracy across rounds.
Rationale
Community benchmarks with held-out truth transformed structural biology and machine learning; oncology model validation is currently self-reported and non-comparable.
What would test it
Run a first round with ten agents whose phase 2 results are complete but unpublished or paywalled, and publish accuracy metrics per submitted model class.
Maturity
speculative
Who has to act
data
Cost to try
Small (under $1M)
Years to first evidence
3
Bottlenecks it attacks
- Preclinical models that do not predict people · Nine in ten cancer drugs that work in mice fail in humans. Our models are the reason.
- AI that is built but not validated or deployed · Thousands of cancer AI models are published; a handful are in clinical use, and fewer have shown they help patients.
- Preclinical results do not reproduce · Fewer than half of landmark cancer biology findings reproduce when someone else tries.