Overfitting is when a model learns the chance patterns and quirks of the data used to build it, so it looks accurate there and does worse in new patients.
When a study measures many candidate features in few patients, some features will separate good from poor outcomes by chance alone, and a signature chosen and scored in the same patients overstates how well it predicts. Michiels and colleagues reanalysed the seven largest published microarray studies of cancer prognosis using many random training and validation splits: the lists of predictive genes depended strongly on which patients were in the training set, the published results were over-optimistic because validation had been inadequate, and five of the seven studies did not classify patients better than chance. Venet and colleagues compared 47 published breast cancer outcome signatures with signatures made of random genes: 28 of them (60 per cent) were not significantly better predictors than random signatures of the same size, and more than 90 per cent of random signatures of more than 100 genes were significant outcome predictors, in part because much of the breast cancer transcriptome tracks proliferation. An outcome association alone does not establish that the biology named by a signature causes the outcome. Overfitting is not the same as bias in the data: a model can also be wrong because its development patients differ from the patients it is used on, and both lower real-world accuracy. Three habits hide the problem: scoring a model on the patients that built it, adjusting a threshold after seeing results, and relying on one favourable split. Repeated internal splits reduce optimism but do not test a new hospital, assay batch or population. The defence is a model locked before testing, evaluated in independent patients and compared with a simple baseline.
Shares Batch effects and harmonisation, External validation.
Shares Data leakage in model evaluation, External validation.
Shares Data leakage in model evaluation, External validation.
Shares Batch effects and harmonisation, External validation.
Shares Batch effects and harmonisation, Data leakage in model evaluation.
Shares Data leakage in model evaluation, External validation.
Shares Data leakage in model evaluation, External validation.
Every source this page cites, grouped by who published it. Grouping is read from the web address, not assigned by hand.
A source that has moved or gone is reported by the weekly link check. /audit/