# Sequestered, prospectively collected benchmark datasets that no one can train on

Source: https://onco.cc/ideas/idea-data-sequestered-prospective-benchmarks/  
OnCo record `idea-data-sequestered-prospective-benchmarks` (Idea). Data CC BY-NC 4.0, attribute "Data from OnCo (onco.cc)"; commercial use needs a licence.

## TL;DR

Keep test datasets locked away and collect them going forward, so AI claims are checked on data the developers have never seen and could not have memorised.

## Summary

Public benchmarks leak into training sets and go stale; retrospective validation flatters models. The proposal is a set of sequestered evaluation datasets for key cancer AI tasks (mammography, lung nodules, prostate biopsy, HER2 scoring, ctDNA calls), collected prospectively from multiple sites and countries, held by a neutral body, with evaluation only via submission of the model or an API, and results published. NIST's face recognition testing and the MICCAI challenge model are precedents.

## Fields

- Kind: Idea
- Last checked: 2026-09-08
- Hypothesis: Performance on sequestered prospective data will be materially lower than published performance for most models, and public reporting will shift developers toward robust training and honest claims.
- Rationale: In face recognition, NIST's sequestered testing became the de facto standard buyers rely on; in medical imaging, external test sets consistently reveal performance drops that published papers omit.
- Proposed test: Stand up two sequestered benchmarks; evaluate all willing vendors; publish results alongside their published claims; repeat annually to measure whether the gap narrows.
- Maturity: early-clinical
- Actor: research

## Sources

- NIST FRTE (face recognition evaluation): https://www.nist.gov/programs-projects/face-technology-evaluations-frtefate

## Connected records

- fronts: [AI & Computation](https://onco.cc/fronts/ai-computation/)
- technologies: [AI in radiology](https://onco.cc/technologies/radiology-ai-screening/), [Digital pathology & AI](https://onco.cc/technologies/digital-pathology-ai/)
- terms: [Circulating tumour DNA (ctDNA)](https://onco.cc/terms/ctdna/)
- bottlenecks: [AI that is built but not validated or deployed](https://onco.cc/bottlenecks/b-ai-validation/)
- ideas: [A neutral public evaluator for cancer AI, on the model of NIST](https://onco.cc/ideas/idea-data-neutral-ai-evaluator/), [External validation at five or more sites in two countries before clearance](https://onco.cc/ideas/idea-data-multisite-validation-precondition/), [Red-team programmes that attack cancer AI before patients do](https://onco.cc/ideas/idea-data-ai-red-team-programme/)

---
JSON: https://onco.cc/api/v1/entities/idea-data-sequestered-prospective-benchmarks.json