# AI that is built but not validated or deployed

Source: https://onco.cc/bottlenecks/b-ai-validation/  
OnCo record `b-ai-validation` (Bottleneck). Data CC BY-NC 4.0, attribute "Data from OnCo (onco.cc)"; commercial use needs a licence.

## TL;DR

Thousands of cancer AI models are published; a handful are in clinical use, and fewer have shown they help patients.

## Summary

Machine learning models for cancer detection, pathology, prognosis and treatment selection are published by the thousand, but almost all are evaluated retrospectively on data from the institution that built them. Of AI-enabled devices cleared by the FDA up to 2020, nearly all were evaluated only retrospectively and most on a single site, and among deep-learning studies comparing AI with clinicians, only a handful were prospective and two were randomised. Retrospective accuracy is not clinical benefit: models drift as scanners, populations and practice change, integration into workflow is costly, liability is unresolved, and reimbursement rarely exists. The MASAI trial of AI-supported mammography screening is one of the first randomised demonstrations that an AI tool can safely change a cancer pathway. Prospective and randomised evaluation, reporting standards, post-market monitoring for drift, and regulatory pathways for models that keep learning are the requirements for AI to move from papers into care.

## Fields

- Kind: Bottleneck
- Last checked: 2026-09-08
- Stage: data-knowledge
- Severity: major
- Metrics: FDA-approved AI medical devices (to 2020) evaluated only retrospectively: 126 of 130 (Wu et al., Nature Medicine 2021); Deep-learning studies comparing AI with clinicians that were randomised trials (systematic review, 2010-2019): 2 of 83 (with 9 prospective non-randomised) (Nagendran et al., BMJ 2020); Screen-reading workload reduction with AI-supported mammography screening at equal or higher cancer detection (MASAI, randomised, 80,033 women): 44% fewer reads (Lång et al., Lancet Oncology 2023)
- Causes: Retrospective single-site accuracy is cheap to produce and enough to publish, so prospective validation is rarely done.; Models trained on one population and scanner degrade on others (dataset shift) and are not monitored after deployment.; Regulatory clearance has not required evidence of clinical benefit or multi-site validation.; Workflow integration, IT security review and liability allocation cost more than the model.; No reimbursement code exists for most AI outputs, so hospitals have no business case.; Locked models cannot legally be updated without re-clearance, so they age in place.

## Sources

- Wu et al., How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals (Nature Medicine 2021): https://doi.org/10.1038/s41591-021-01312-x
- Nagendran et al., Artificial intelligence versus clinicians: systematic review (BMJ 2020): https://doi.org/10.1136/bmj.m689
- Lång et al., AI-supported mammography screening (MASAI, Lancet Oncology 2023): https://doi.org/10.1016/S1470-2045(23)00298-X
- FDA, Artificial intelligence-enabled medical devices: https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-enabled-medical-devices

## Connected records

- ideas: [A dedicated fund for randomised trials of cancer AI with patient outcomes](https://onco.cc/ideas/idea-data-prospective-ai-trials-fund/), [A federated learning consortium of cancer centres that jointly own the models](https://onco.cc/ideas/idea-fund-federated-learning-consortium/), [A liability framework for clinical AI: safe harbour for clinicians, liability for makers](https://onco.cc/ideas/idea-data-ai-liability-safe-harbour/), [A mandatory silent (shadow) trial before any cancer AI goes live](https://onco.cc/ideas/idea-data-silent-trial-before-deployment/), [A monthly-updated benchmark for AI answers to oncology questions with citation accuracy](https://onco.cc/ideas/idea-data-living-llm-oncology-benchmark/), [A neutral public evaluator for cancer AI, on the model of NIST](https://onco.cc/ideas/idea-data-neutral-ai-evaluator/), [A pre-competitive consortium to train a shared multimodal cancer foundation model](https://onco.cc/ideas/idea-data-precompetitive-cancer-foundation-model/), [A public API serving the current standard of care for any cancer, stage and biomarker](https://onco.cc/ideas/idea-data-standard-of-care-api/), [A public benchmark and audit of chatbot answers to cancer questions](https://onco.cc/ideas/idea-moon-ai-cancer-answer-audit/), [A public registry of every AI model used in cancer care](https://onco.cc/ideas/idea-data-clinical-ai-model-registry/), [A randomised trial of AI scribes in oncology clinics measuring errors and time](https://onco.cc/ideas/idea-data-llm-documentation-rct-oncology/), [A randomised trial of AI-generated treatment recommendations versus tumour boards](https://onco.cc/ideas/idea-data-ai-vs-tumour-board-rct/), [A registry of external validation datasets for cancer AI models, with mandatory reporting](https://onco.cc/ideas/idea-tr2-ai-external-validation-registry/), [A regulatory sandbox for continuously learning cancer AI](https://onco.cc/ideas/idea-data-regulatory-sandbox-adaptive-ai/), [A shared AI review assistant that maps one dossier to every regulator's questions](https://onco.cc/ideas/idea-reg-shared-ai-review-assistant/), [A standard evaluation pathway for AI-assisted pathology, from reader study to deployment](https://onco.cc/ideas/idea-data-ai-pathology-evaluation-standard/), [A standard for monitoring AI performance drift with pause thresholds](https://onco.cc/ideas/idea-data-drift-monitoring-standard/), [A virtual cancer cell that predicts what a drug will do before you test it](https://onco.cc/ideas/idea-bio1-virtual-cell-perturbation/), [AI clears the normal lung screening scans so radiologists read only the suspicious ones](https://onco.cc/ideas/idea-prev-ldct-ai-negative-triage/), [AI malignancy scores to end repeat scans and biopsies for benign lung nodules](https://onco.cc/ideas/idea-prev-lung-nodule-ai-discharge/), [AI quantification of HER2-low and HER2-ultralow](https://onco.cc/ideas/idea-ai-her2-low-scoring/), [AI second reads to stop borderline lesions being upgraded to cancer](https://onco.cc/ideas/idea-prev-pathology-ai-borderline-anchor/), [AI-assisted central imaging reads to cut endpoint cost and variability](https://onco.cc/ideas/idea-tr1-ai-central-imaging-reads/), [AI-designed proteins that grip the floppy parts of cancer drivers](https://onco.cc/ideas/idea-bio1-ai-binders-disordered-regions/), [AI-first reading for high-volume common cancer diagnoses, pathologist for the exceptions](https://onco.cc/ideas/idea-acc-ai-first-pathology-common-cases/), [An independent evaluation unit for surgical robots and AI, paid on evidence](https://onco.cc/ideas/idea-fund-surgical-ai-robotics-evaluation/), [An open foundation model of the cancer cell trained on perturbation data](https://onco.cc/ideas/idea-data-open-cell-foundation-model/), [Autonomous closed-loop adaptive therapy driven by blood tests and evolutionary models](https://onco.cc/ideas/idea-moon-closed-loop-adaptive-therapy/), [Continuous prospective validation for every oncology AI tool after deployment](https://onco.cc/ideas/idea-moon-continuous-ai-validation-registry/), [Decision support that cites the exact trial and guideline line it relies on](https://onco.cc/ideas/idea-data-provenance-first-decision-support/), [Digital batch records and AI process control to halve cell therapy batch failures](https://onco.cc/ideas/idea-reg-ai-process-control-cell-manufacturing/), [Digital twins for treatment selection, validated by predicting before observing](https://onco.cc/ideas/idea-data-digital-twin-predict-then-observe/), [Digitise the nation's pathology slides and link them to outcomes](https://onco.cc/ideas/idea-data-national-slide-archive/), [Double oncology capacity in low-resource settings with task-shifting and AI decision support](https://onco.cc/ideas/idea-moon-task-shifting-ai-oncology-capacity/), [Every AI output logged in the record with input hash, version and clinician response](https://onco.cc/ideas/idea-data-ai-audit-trail-in-ehr/), [External validation at five or more sites in two countries before clearance](https://onco.cc/ideas/idea-data-multisite-validation-precondition/), [Federated training of pathology and radiology models across hospitals](https://onco.cc/ideas/idea-data-federated-learning-imaging/), [Forecast the next resistance mutation like the weather](https://onco.cc/ideas/idea-bio1-evolution-forecasting/), [In silico trials to prioritise combinations, scored against later real trials](https://onco.cc/ideas/idea-data-in-silico-trials-calibrated/), [Judge skin cancer AI by the thick melanomas it prevents, not the thin ones it finds](https://onco.cc/ideas/idea-prev-melanoma-ai-thick-melanoma-metric/), [Mandatory post-market performance reporting for cancer AI](https://onco.cc/ideas/idea-data-ai-post-market-performance-reporting/), [Mandatory subgroup performance reporting for cancer AI](https://onco.cc/ideas/idea-data-subgroup-performance-reporting-mandate/), [No clinical claims for imaging-derived biomarkers without phantom and standards compliance](https://onco.cc/ideas/idea-tr2-radiomics-ibsi-mandate/), [One certified open-source de-identification pipeline for scans and slides](https://onco.cc/ideas/idea-data-open-deidentification-pipeline/), [Pathologist assistants plus AI triage to multiply pathologist capacity](https://onco.cc/ideas/idea-acc-pathologist-assistants-and-ai-triage/), [Patient-level multimodal foundation models for treatment selection](https://onco.cc/ideas/idea-multimodal-foundation-model/), [Patients told which AI is used in their care, in plain language](https://onco.cc/ideas/idea-data-patient-facing-model-cards/), [Pay for cancer AI only when it has outcome evidence, then pay properly](https://onco.cc/ideas/idea-data-ai-reimbursement-tied-to-outcomes/), [Pool every immunotherapy trial's biomarker data into one commons](https://onco.cc/ideas/idea-bio2-io-biomarker-data-commons/), [Pre-agreed update rules so an MCED test is not obsolete when its trial reads out](https://onco.cc/ideas/idea-prev-mced-change-control-plan/), [Pre-registered, publicly scored AI ranking of repurposing candidates for cancer](https://onco.cc/ideas/idea-reg-preregistered-ai-repurposing-scoring/), [Public gold-standard datasets for validating every cancer biomarker test](https://onco.cc/ideas/idea-tr2-open-cdx-validation-sets/), [Publicly funded cancer AI must release open weights and model cards](https://onco.cc/ideas/idea-data-open-weights-for-public-funded-ai/), [Publish a per-lesion miss rate for every prostate MRI service before it is allowed to guide focal treatment](https://onco.cc/ideas/idea-prostate-per-lesion-mri-audit-before-focal-treatment/), [Read muscle loss automatically from scans patients already have](https://onco.cc/ideas/idea-bio2-ai-sarcopenia-from-ct/), [Red-team programmes that attack cancer AI before patients do](https://onco.cc/ideas/idea-data-ai-red-team-programme/), [Require stage-shift or interval-cancer endpoints for AI in cancer screening](https://onco.cc/ideas/idea-data-ai-screening-endpoints/), [Rules for retiring cancer AI when performance drops or the standard of care moves](https://onco.cc/ideas/idea-data-ai-decommissioning-rules/), [Score every model system on how well it predicted real trial results](https://onco.cc/ideas/idea-bio1-model-predictivity-benchmark/), [Sequestered, prospectively collected benchmark datasets that no one can train on](https://onco.cc/ideas/idea-data-sequestered-prospective-benchmarks/), [Turn the map of immune cells inside a tumour into a standardised test](https://onco.cc/ideas/idea-bio2-spatial-signature-cdx/), [Validate and reimburse AI contouring and planning to expand radiotherapy capacity](https://onco.cc/ideas/idea-fund-rt-planning-ai-capacity/), [Version control and locked reference sets for AI algorithms used as companion diagnostics](https://onco.cc/ideas/idea-tr2-ai-cdx-change-control/), [Whole-patient digital twins validated in prospective randomised trials](https://onco.cc/ideas/idea-moon-validated-digital-twins/)
- collections: [FDA Oncology Approvals (OCE) & Novel Drug Approvals](https://onco.cc/collections/fda-approvals/)
- cancers: [Colorectal cancer](https://onco.cc/cancers/colorectal/), [HR-positive / HER2-negative breast cancer](https://onco.cc/cancers/breast-hr-positive/), [Melanoma](https://onco.cc/cancers/melanoma/), [Non-small-cell lung cancer](https://onco.cc/cancers/nsclc/), [Prostate cancer](https://onco.cc/cancers/prostate/)
- fronts: [AI & Computation](https://onco.cc/fronts/ai-computation/)
- technologies: [AI in radiology](https://onco.cc/technologies/radiology-ai-screening/), [AI trial matching & clinical decision support](https://onco.cc/technologies/ai-trial-matching/), [AI-driven drug & target discovery](https://onco.cc/technologies/ai-drug-design/), [Dermoscopy, total-body photography & AI skin analysis](https://onco.cc/technologies/dermoscopy-ai/), [Digital pathology & AI](https://onco.cc/technologies/digital-pathology-ai/), [Mammography & tomosynthesis](https://onco.cc/technologies/mammography/), [Pathology & radiology foundation models](https://onco.cc/technologies/pathology-foundation-model/)
- drugs: [ArteraAI Breast](https://onco.cc/drugs/artera-ai-breast/), [ArteraAI Prostate](https://onco.cc/drugs/artera-ai-prostate/)
- companies: [Aidoc](https://onco.cc/companies/aidoc/), [Artera](https://onco.cc/companies/artera/), [Lunit](https://onco.cc/companies/lunit/), [Owkin](https://onco.cc/companies/owkin/), [Paige AI](https://onco.cc/companies/paige/), [PathAI](https://onco.cc/companies/pathai/), [Tempus AI](https://onco.cc/companies/tempus/)
- key papers: [AlphaFold 2: predicting protein structures to near-experimental accuracy](https://onco.cc/key-papers/paper-alphafold2-jumper-nature-2021/), [Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies](https://onco.cc/key-papers/paper-nagendran-bmj/), [How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals](https://onco.cc/key-papers/paper-wu-nat-med/), [MASAI: AI-supported mammography screening finds more cancers with half the radiologist workload](https://onco.cc/key-papers/paper-masai-lancet-oncol-2023/)
- terms: [Whole-mount pathology](https://onco.cc/terms/whole-mount-pathology/)
- institutions: [Medicines and Healthcare products Regulatory Agency](https://onco.cc/institutions/mhra/)
- roadmaps: [AI in oncology roadmap: pattern readers → foundation models → agents in the workflow](https://onco.cc/roadmaps/ai-oncology-roadmap/)

---
JSON: https://onco.cc/api/v1/entities/b-ai-validation.json