ideasIdea
Red-team programmes that attack cancer AI before patients do
Pay independent experts to try to break cancer AI tools with unusual images, rare cases, bad scans and data shifts, and publish what breaks them.
Robustness of medical AI to artefacts, rare presentations, adversarial inputs and distribution shift is poorly characterised. The proposal funds standing red teams (imaging physicists, pathologists, security researchers) that stress-test cleared and pre-clearance cancer AI with curated adversarial and edge-case corpora, publish failure modes in a common taxonomy, and feed results to the registry and developers, as is done for cybersecurity and increasingly for general-purpose AI.
Hypothesis
Red-teaming will uncover clinically relevant failure modes in most tested models that were not disclosed in their validation, and disclosure will lead to fixes or labelling changes.
Rationale
Every mature safety-critical field uses adversarial testing; medical AI relies on developers' own validation, which is structurally blind to what the developers did not think of.
What would test it
Red-team ten cancer AI tools over one year; publish findings; track developer responses and label changes within a further year.
Maturity
speculative
Who has to act
research
Cost to try
Small (under $1M)
Years to first evidence
1
Bottlenecks it attacks
- AI that is built but not validated or deployed · Thousands of cancer AI models are published; a handful are in clinical use, and fewer have shown they help patients.