Data Drift: Keep Models Healthy in Prod

Data Drift - Why Models Quietly Die in Production 📉

The scary part. Your model launches at 95% accuracy. Six months later it's making terrible calls - and throwing zero errors. No crash, no alert, no red light. It just quietly got worse while everyone assumed it was fine. This is the silent killer of production ML, and the only defence is monitoring.

🧠 The mental model: a photograph of the world

A trained model is a photograph of the world at training time. It assumes the future looks like that photo. But the world keeps moving - a new market regime, a product launch, a pandemic - and the photo doesn't. Drift = how far reality has walked away from the photo.

Two flavours of drift: Data (covariate) drift - the inputs' distribution changes (your customers got younger; volatility spiked). Concept drift - the relationship between input and output changes (the same inputs now mean something different). Both quietly break your model.

Detecting it: Population Stability Index (PSI)

PSI compares a feature's distribution now vs at training, bucket by bucket:

PSI = Σ over buckets of  (current%  −  reference%) × ln(current% / reference%)

Read it with this industry-standard table:

The MLOps loop (this is what "production ML" actually means)

train → deploy → MONITOR (drift, accuracy) → alert → retrain → redeploy → ...

Building the model is ~20% of the job. Monitoring it and retraining when it drifts is the other 80% - that's the difference between a notebook and a real system.

🔧 Try it for real

The exercise below computes PSI on two distributions. After it passes, change current to [0.25, 0.25, 0.25, 0.25] (identical to reference) and re-run - PSI drops to ~0 ("stable"). Then make it more lopsided and watch PSI climb. That hands-on intuition is what you'll remember.

Your Task

Given a reference (training) distribution and the current (live) one as binned shares, compute the PSI with NumPy and flag drift when it exceeds 0.2.