# A standard for monitoring AI performance drift with pause thresholds

Source: https://onco.cc/ideas/idea-data-drift-monitoring-standard/  
OnCo record `idea-data-drift-monitoring-standard` (Idea). Data CC BY-NC 4.0, attribute "Data from OnCo (onco.cc)"; commercial use needs a licence.

## TL;DR

Set common rules for how hospitals check that an AI tool still works as the scanners, patients and practices around it change, and when it must be switched off.

## Summary

Model performance shifts when scanners, staining protocols, populations or clinical practice change. Few deployments monitor this. The proposal is a technical standard: a per-site reference dataset re-scored monthly, input distribution monitoring, calibration and subgroup checks, pre-specified thresholds for alert and pause, and a documented recalibration or retraining pathway, integrated with the vendor's change control plan and reported to the registry.

## Fields

- Kind: Idea
- Last checked: 2026-09-08
- Hypothesis: Sites following the drift standard will detect degradation months earlier than unmonitored sites and avoid patient harm events attributable to silent drift.
- Rationale: Industrial machine learning monitors drift as routine engineering practice; healthcare deployments largely do not, and the few audits done have found drift within a year or two of deployment.
- Proposed test: Implement the standard at ten sites running the same pathology or radiology model; compare detected drift events and time to detection with ten unmonitored sites over two years.
- Maturity: early-clinical
- Actor: engineering

## Sources

- Bottleneck evidence (AI that is built but not validated or deployed): Wu et al., How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals (Nature Medicine 2021): https://doi.org/10.1038/s41591-021-01312-x

## Connected records

- ideas: [A mandatory silent (shadow) trial before any cancer AI goes live](https://onco.cc/ideas/idea-data-silent-trial-before-deployment/), [Mandatory post-market performance reporting for cancer AI](https://onco.cc/ideas/idea-data-ai-post-market-performance-reporting/), [Rules for retiring cancer AI when performance drops or the standard of care moves](https://onco.cc/ideas/idea-data-ai-decommissioning-rules/)
- fronts: [AI & Computation](https://onco.cc/fronts/ai-computation/)
- bottlenecks: [AI that is built but not validated or deployed](https://onco.cc/bottlenecks/b-ai-validation/)
- key papers: [How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals](https://onco.cc/key-papers/paper-wu-nat-med/)

---
JSON: https://onco.cc/api/v1/entities/idea-data-drift-monitoring-standard.json