# Red-team programmes that attack cancer AI before patients do

Source: https://onco.cc/ideas/idea-data-ai-red-team-programme/  
OnCo record `idea-data-ai-red-team-programme` (Idea). Data CC BY-NC 4.0, attribute "Data from OnCo (onco.cc)"; commercial use needs a licence.

## TL;DR

Pay independent experts to try to break cancer AI tools with unusual images, rare cases, bad scans and data shifts, and publish what breaks them.

## Summary

Robustness of medical AI to artefacts, rare presentations, adversarial inputs and distribution shift is poorly characterised. The proposal funds standing red teams (imaging physicists, pathologists, security researchers) that stress-test cleared and pre-clearance cancer AI with curated adversarial and edge-case corpora, publish failure modes in a common taxonomy, and feed results to the registry and developers, as is done for cybersecurity and increasingly for general-purpose AI.

## Fields

- Kind: Idea
- Last checked: 2026-09-08
- Hypothesis: Red-teaming will uncover clinically relevant failure modes in most tested models that were not disclosed in their validation, and disclosure will lead to fixes or labelling changes.
- Rationale: Every mature safety-critical field uses adversarial testing; medical AI relies on developers' own validation, which is structurally blind to what the developers did not think of.
- Proposed test: Red-team ten cancer AI tools over one year; publish findings; track developer responses and label changes within a further year.
- Maturity: speculative
- Actor: research

## Sources

- Bottleneck evidence (AI that is built but not validated or deployed): Wu et al., How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals (Nature Medicine 2021): https://doi.org/10.1038/s41591-021-01312-x

## Connected records

- ideas: [Sequestered, prospectively collected benchmark datasets that no one can train on](https://onco.cc/ideas/idea-data-sequestered-prospective-benchmarks/)
- fronts: [AI & Computation](https://onco.cc/fronts/ai-computation/)
- bottlenecks: [AI that is built but not validated or deployed](https://onco.cc/bottlenecks/b-ai-validation/)
- key papers: [How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals](https://onco.cc/key-papers/paper-wu-nat-med/)

---
JSON: https://onco.cc/api/v1/entities/idea-data-ai-red-team-programme.json