MLOpsLab

Article #14

Production ML Monitoring

Model Drift Detection:
Stop Silent Failures Before They Kill Your Model

Your model shipped. Now it’s slowly dying — and you don’t know it. Learn to detect data drift, concept drift, and prediction drift using Evidently AI, FastAPI, and Python before the damage is done.

Evidently AI
FastAPI
Python 3.12+
PSI · KS Test

Reason ML projects fail

3×

Types of drift to monitor

Steps to full monitoring

Table of Contents

What Is Model Drift Detection?

Model drift detection is the practice of monitoring machine learning models in production to identify when they start degrading due to changes in real-world data. Without proper detection, the model that worked perfectly at deployment starts making worse predictions — often silently, without any errors or alerts.

This is the #1 reason ML projects fail in production. By the time you notice the problem, you’ve already lost revenue, damaged user trust, or made critical bad decisions. Implementing drift detection is the only way to catch these issues early.

📉

The Silent Killer: Most teams don’t monitor drift. They only notice when a stakeholder complains. By then, the model has been wrong for weeks — sometimes months. Model drift detection would have flagged this on day one.

Three Types of Drift You Must Monitor

📊

Data Drift

Input feature distributions change over time. Your model sees data it was never trained on.

🔄

Concept Drift

The relationship between inputs and outputs changes. What the model learned is no longer valid.

🎯

Prediction Drift

The distribution of model outputs changes over time — a leading indicator that something upstream has shifted.

Real-World Examples

Data DriftA fraud detection model trained on 2024 transaction patterns encounters completely different spending behavior in 2026. Accuracy silently collapses.
Concept DriftA house price model trained pre-COVID fails badly after remote work permanently changes housing demand dynamics.
Prediction DriftA recommendation model starts surfacing entirely different categories as user behavior shifts post-product-change.

⚠

The hard truth: If you’re not implementing drift detection, you’re flying blind. Your model is degrading right now — you just don’t know it.

Why Model Drift Detection Matters

💰

Revenue Loss

👥

User Trust

⚖️

Compliance

🔧

Engineering

Bad recommendations, wrong pricing, failed fraud detection — all translate directly to lost money. Users notice when your model is wrong before you do. Regulated industries (finance, healthcare) require model monitoring by law. Without drift detection, you waste days debugging “why did the model get worse?” with zero data to guide you.

✅

Business case: One drift detection system can save months of engineering time and prevent millions in revenue loss. The ROI is not even close.

Statistical Methods for Drift Detection

Metric

PSI

Population Stability Index — measures distribution shift between two samples.

PSI < 0.1 → stable
PSI > 0.25 → retrain

Test

KS Test

Kolmogorov-Smirnov test. Low p-value signals drift.

p < 0.05 → drift detected

Visual

Distribution Plots

Plot feature distributions over time. Look for shifts.

> 15–20% shift → investigate

📊

Quick rule of thumb: If any feature’s distribution changes by more than 15–20%, investigate immediately. Don’t wait for accuracy to drop. That’s the power of drift detection.

Tools for Model Drift Detection

Evidently AI

Open Source

Python library that generates drift reports, data quality dashboards, and model performance metrics. Free, self-hosted.

WhyLabs

SaaS

Managed platform with a free tier. Monitors drift, data quality, and performance out of the box.

Prometheus + Grafana

Infrastructure

Self-hosted monitoring stack. Track drift metrics as time-series. Alert when drift scores cross thresholds.

MLflow

Open Source

Track drift scores as metrics alongside experiments. Trigger external alerts when scores exceed thresholds.

Step-by-Step Tutorial

Step 01Install Evidently AI

pip install evidently

1	pip install evidently

Step 02Log Predictions from Your FastAPI Endpoint

Add prediction logging to your /predict endpoint. Every prediction gets stored to a JSONL file for later drift analysis.

# Add this to your FastAPI /predict endpoint
import json
from datetime import datetime

def log_prediction(features: dict, prediction: int, probability: float):
    log_entry = {
        "timestamp": datetime.utcnow().isoformat(),
        "features": features,
        "prediction": prediction,
        "probability": probability
    }
    with open("predictions.jsonl", "a") as f:
        f.write(json.dumps(log_entry) + "\n")

# Add this to your FastAPI /predict endpoint

import json

from datetime import datetime

def log_prediction(features: dict, prediction: int, probability: float):

log_entry = {

"timestamp": datetime.utcnow().isoformat(),

"features": features,

"prediction": prediction,

"probability": probability

}

with open("predictions.jsonl", "a") as f:

f.write(json.dumps(log_entry) + "\n")

Step 03Load Your Reference (Training) Distribution

import pandas as pd
from evidently import ColumnMapping

reference_data = pd.read_csv("data/training_features.csv")
reference_data["prediction"] = training_predictions
reference_data["probability"] = training_probabilities

import pandas as pd

from evidently import ColumnMapping

reference_data = pd.read_csv("data/training_features.csv")

reference_data["prediction"] = training_predictions

reference_data["probability"] = training_probabilities

Step 04Run the Drift Detection Report

Evidently’s DataDriftPreset automatically runs PSI and KS tests across all features and produces a visual HTML report.

from evidently.report import Report
from evidently.metric_preset import DataDriftPreset

current_data = pd.read_json("predictions.jsonl", lines=True)

data_drift_report = Report(metrics=[DataDriftPreset()])
data_drift_report.run(
    reference_data=reference_data,
    current_data=current_data
)

data_drift_report.save_html("drift_report.html")

from evidently.report import Report

from evidently.metric_preset import DataDriftPreset

current_data = pd.read_json("predictions.jsonl", lines=True)

data_drift_report = Report(metrics=[DataDriftPreset()])

data_drift_report.run(

reference_data=reference_data,

current_data=current_data

)

data_drift_report.save_html("drift_report.html")

Step 05Configure Alerts (Slack / PagerDuty)

Don’t just generate reports — trigger alerts automatically. Connect to Slack, email, or PagerDuty so your team is notified the moment drift appears.

from evidently.metrics import DatasetDriftMetric

drift_metric = DatasetDriftMetric()
drift_metric.reference = reference_data
drift_metric.current   = current_data

if drift_metric.get_result().drift_detected:
    print("⚠ DRIFT DETECTED — investigate immediately")

    import requests
    requests.post(
        "https://hooks.slack.com/services/YOUR_WEBHOOK",
        json={"text": "🚨 Model drift detected in production!"}
    )

from evidently.metrics import DatasetDriftMetric

drift_metric = DatasetDriftMetric()

drift_metric.reference = reference_data

drift_metric.current = current_data

if drift_metric.get_result().drift_detected:

print("⚠ DRIFT DETECTED — investigate immediately")

import requests

requests.post(

"https://hooks.slack.com/services/YOUR_WEBHOOK",

json={"text": "🚨 Model drift detected in production!"}

)

Step 06Automate with Cron or Airflow

Drift detection should run on a schedule, not manually. Add it to cron for simplicity or to an Airflow DAG if you need orchestration and retry logic.

# Run drift detection every day at 9:00 AM
# Add with: crontab -e
0 9 * * * python3 /opt/ml/drift_detection.py

# Run drift detection every day at 9:00 AM

# Add with: crontab -e

0 9 * * * python3 /opt/ml/drift_detection.py

✦

Pro tip on frequency: Run drift detection daily for revenue-critical models, weekly for others. The cost of a single missed drift event vastly outweighs the cost of running checks regularly.

When Drift Is Detected — What To Do

Identify the drifting features

Open the Evidently report and look at which specific features are flagged. Sort by drift score descending.

Diagnose the root cause

Is it seasonal? A data pipeline bug? A real-world behavioral shift? Drift detection tells you what, not why.

Trigger retraining if drift is confirmed

If drift is real and significant, retrain on newer labeled data. Don’t retrain blindly — confirm you have sufficient new data first.

Recalibrate your thresholds

Update your alert thresholds based on what you learned. Some drift may be acceptable for your use case.

Document the incident

Add it to your model’s changelog. Include what drifted, why, and how you fixed it.

🔁

Retraining strategy: Don’t retrain reflexively. Only retrain when drift is confirmed AND you have sufficient new labeled data. Retraining on insufficient data can make things worse.

Continue Learning About MLOps

Article #8

mlopslab.org · Model Drift Detection Tutorial · 2026 · Evidently AI · FastAPI · Python

Model Drift Detection Tutorial: How to Monitor ML Models in Production (2026)

Model Drift Detection:
Stop Silent Failures Before They Kill Your Model

What Is Model Drift Detection?

Three Types of Drift You Must Monitor

Data Drift

Concept Drift

Prediction Drift

Real-World Examples

Why Model Drift Detection Matters

Statistical Methods for Drift Detection

PSI

KS Test

Distribution Plots

Tools for Model Drift Detection

Step-by-Step Tutorial

When Drift Is Detected — What To Do

Continue Learning About MLOps

Kubeflow vs Airflow for ML

MLflow Tutorial

MLOps Roadmap 2026

ML Pipeline Tutorial

Model Drift Detection:Stop Silent Failures Before They Kill Your Model

What Is Model Drift Detection?

Three Types of Drift You Must Monitor

Data Drift

Concept Drift

Prediction Drift

Real-World Examples

Why Model Drift Detection Matters

Statistical Methods for Drift Detection

PSI

KS Test

Distribution Plots

Tools for Model Drift Detection

Step-by-Step Tutorial

When Drift Is Detected — What To Do

Continue Learning About MLOps

Kubeflow vs Airflow for ML

MLflow Tutorial

MLOps Roadmap 2026

ML Pipeline Tutorial

Model Drift Detection:
Stop Silent Failures Before They Kill Your Model