Clinical machine learning models often report near-perfect accuracies (98-99%) in literature. However, a rigorous audit reveals that data leakage (such as performing SMOTE oversampling before cross-validation splitting) frequently manufactures these metrics.
The Leakage Experiment#
We tested a Stacking Ensemble under two conditions: (1) Correct validation where SMOTE lives inside each fold, and (2) Leaky validation where SMOTE occurs beforehand. The leaky setup inflated accuracy by 3.2% and F1-score by 9.0%, showing that model reporting must detail validations to be trusted.