Model drift

What model drift is

Model drift means a model's accuracy declines over time even though nothing about the model itself was changed. The cause lies outside: the world the data came from has moved, the model has not. A model that learned in spring which customers return faces different buying behaviour in winter.

This is not a defect of the method but a property. Every model describes a state that held at the time of training. Without monitoring the decay stays invisible, because the model keeps producing output, only worse output.

Which forms are distinguished

• Data shift: The input values change. A new ad network brings a younger audience and suddenly the features look different from the training data.
• Concept shift: The inputs stay the same but their meaning changes. A discount used to produce more purchases, in a high inflation market it no longer does.
• Feedback loop: The model influences the data it learns from. A recommendation engine shows only items it already knows and thereby narrows its own learning material.
• Season and one-off: Distortions from seasons, holidays or a single campaign. Not permanent decay, but indistinguishable from it in a year on year comparison.

How drift is detected

Two routes are common. The first is direct accuracy tracking: once the actual outcomes exist, compare how well the model predicted them. That only works when the outcome becomes known quickly, for instance with a returns forecast, not with a model for payment defaults beyond 90 days.

The second route is watching the inputs. When the distribution of features shifts, for instance the share of new customers or the device mix, drift is likely even without known outcomes. This route is faster because it needs no waiting time. Both routes together work better than either alone.

What helps

Three measures form the core. First, one fixed control metric per model, measured on a fixed cadence, plus a defined threshold that triggers a review. A model without a threshold has no alarm. Second, a fixed refresh cycle, meaning retraining on current data at regular intervals, so drift does not have to be detected before it is fixed.

Third, a simple fallback. If a model performs badly, it must be switchable off without stopping the shop. For a price optimiser that means returning to fixed rules, for a recommendation returning to the best sellers. Without a fallback in place, an outage has to be sat out, and that makes a bad model more expensive than a missing one.