In the fourth week of the second semester I built a churn prediction model and reached 99.2% accuracy. I sent the result to my instructor expecting praise. Her reply was a single question.
The question that brought it all down
She wrote: what are the ten most important features in your model? I opened the report and found, in first place, a column called subscription cancellation date.
The model was predicting a customer's churn from the date they churned. It had learned nothing — it was reading the answer off the question paper.
Data leakage is not a rare mistake
I assumed it was a beginner's slip I would outgrow. But the next semester we found that nearly half the cohort had hit some form of leakage.
Sometimes it is obvious, as in my case; sometimes it hides in a normalisation step applied to the full dataset before splitting — and that kind you only catch if you are looking for it.
What changed afterwards
I now start every project with one question before any code: will this column genuinely be available at the moment of prediction in the real world? A column unavailable at decision time has no place in the model, however much accuracy it adds.
And I became suspicious of every high score. Now when I see 99% I do not celebrate — I look for the mistake first, and celebrate only if I cannot find one.
