After Launch: Monitoring and Drift
The world keeps changing after the model is fitted
Your rent estimator worked on last quarter’s flats. A new neighbourhood opens, the listing site starts accepting area in a different unit, or the mix of expensive flats increases. The service can still return 200 while its estimates become less useful.
Monitoring collects evidence about service health, data quality, prediction behaviour, and actual outcomes. It gives you a reason to investigate a change instead of assuming yesterday’s evaluation is enough.
1. Identify predictions
Request ID + time + model version
Each prediction remains linked to its original release.
2. Observe immediately
Latency, errors, input quality, prediction distribution
These signals can arrive before the actual rent is known.
3. Join outcomes later
Prediction ID → actual rent → versioned MAE
Report outcome coverage and delay alongside model-quality metrics.
4. Investigate and act
Repair data, roll back, collect labels, or validate a new model
A shift is a reason to investigate, not an automatic retraining command.
Example input shift: larger flats appear more often
Share of flats above 1,000 square feet: 10% in the reference sample, 40% now. This chart alone does not establish accuracy loss.
Watch several layers separately
Key Points
- Service health: latency, errors, traffic, memory, and availability.
- Input quality: missing fields, invalid values, unknown categories, and unit changes.
- Prediction behaviour: score distribution, class decisions, or extreme numerical estimates.
- Outcome quality: error metrics on examples whose actual outcomes are now known.
- Important groups: whether one location, price range, or customer type behaves differently.
A server can be healthy with a weak model. A good model can sit behind a broken endpoint. Separating these layers helps you change the right thing.
A feature shift is a measurable observation
Suppose 10% of flats in the reference period were larger than 1,000 square feet. In the latest sample, 40% exceed that size. The input distribution changed. That is a signal to inspect, even before final rents are available.
1reference = [500] * 9 + [1500]
2current = [500] * 6 + [1500] * 4
3
4def large_flat_share(values):
5 return sum(value > 1000 for value in values) / len(values)
6
7old_share = large_flat_share(reference)
8new_share = large_flat_share(current)
9print(f"Reference share: {old_share:.2f}")
10print(f"Current share: {new_share:.2f}")
11print(f"Change: {(new_share - old_share) * 100:.0f} percentage points")Reference share: 0.10 Current share: 0.40 Change: 30 percentage points
The samples are tiny to make the calculation visible. In practice, choose representative windows, require enough rows, and compare against seasonal patterns. A different share does not by itself prove that prediction error rose.
Data drift and concept drift are different
Data drift means the input distribution changes: perhaps larger flats appear more often. Concept drift means the relationship between features and the target changes: perhaps comparable flats now rent for more because local demand increased.
You may see the first kind from inputs alone. Establishing the second usually needs outcome evidence or a domain investigation. A system that accepts a new unit can also cause an input shift; that is a pipeline bug to repair, not automatically a reason to retrain.
Key Points
- A distribution shift is an observation, not a performance verdict.
- Performance can deteriorate without a large visible feature shift.
- Class proportions can change too, affecting threshold behaviour and precision.
- Check representation and data quality before blaming the model.
Actual outcomes arrive later
At prediction time, the eventual agreed rent may not be known. Store a prediction ID, time, model version, prediction, and enough context for evaluation. When the actual rent arrives, join it to the original record.
1predictions = [
2 {"id": "p1", "version": "rent-v1", "rent": 25000},
3 {"id": "p2", "version": "rent-v1", "rent": 30000},
4 {"id": "p3", "version": "rent-v2", "rent": 35000},
5]
6outcomes = {"p1": 27000, "p2": 29000}
7
8matched = [row for row in predictions if row["id"] in outcomes]
9mae = sum(abs(outcomes[row["id"]] - row["rent"]) for row in matched) / len(matched)
10print("Predictions with outcomes:", len(matched))
11print(f"Outcome coverage: {len(matched) / len(predictions):.2f}")
12print(f"Matched rent-v1 MAE: ₹{mae:,.0f}")Predictions with outcomes: 2 Outcome coverage: 0.67 Matched rent-v1 MAE: ₹1,500
This MAE describes two labelled predictions from rent-v1 only. The third has no outcome yet. Always track coverage and delay; if only certain listings receive labels, the measured error may not represent all predictions.
Versions and cohorts keep comparisons meaningful
A cohort is a group chosen for analysis, such as listings predicted in the same week or neighbourhood. Compare metrics by prediction time, model version, and relevant cohorts. Mixing releases or comparing different customer mixes can hide what changed.
Key Points
- Keep target and metric definitions stable across comparison windows.
- Report row counts and outcome coverage beside the metric.
- Compare groups that matter to the decision, not only the overall average.
- A rollout can change the data that each model receives, so do not call every score difference a model improvement.
Tail latency can hide behind a quick average
If most requests take 20 milliseconds but a few take hundreds, the mean can look acceptable while users wait. A percentile describes how much of the sample is at or below a latency value.
1import math
2
3latencies_ms = [20, 20, 21, 22, 23, 24, 25, 26, 30, 400]
4sorted_latencies = sorted(latencies_ms)
5rank = math.ceil(0.95 * len(sorted_latencies))
6p95 = sorted_latencies[rank - 1]
7mean = sum(latencies_ms) / len(latencies_ms)
8
9print(f"Mean latency: {mean:.1f} ms")
10print(f"Nearest-rank p95: {p95} ms")Mean latency: 61.1 ms Nearest-rank p95: 400 ms
p95 means the 95th percentile. This example uses the nearest-rank convention, which selects an observed sample value. Other libraries interpolate, so record the metric convention. Ten requests teach the calculation but are too few to estimate a reliable production tail.
An alert needs an investigation and an action
An alert that simply says “drift detected” leaves the team guessing. Include the changed feature or metric, baseline and current window, sample count, affected model version, and the owner who investigates.
Key Points
- First check upstream data, units, schema, and recent release changes.
- Inspect affected examples and relevant groups.
- Choose whether to repair data, roll back, adjust a threshold, collect labels, or retrain.
- Define alert duration and minimum volume so a tiny noisy sample does not trigger constant interruptions.
Automatic retraining can absorb corrupted data or reproduce the same failure. A new candidate still needs clean training data, validation, a controlled release, and evidence that it improves the actual decision.
Where people get confused
Key Points
- A 200 status is an operational result, not an accuracy metric.
- Input drift is not automatically concept drift or proven model failure.
- Metrics from only the labelled subset need coverage and sampling context.
- Retraining is one response, not the default fix for every alert.
- Monitoring the whole population can hide a failing subgroup.
Quick check
How does data drift differ from concept drift?
Data drift is a change in the input distribution. Concept drift is a change in the relationship between features and the outcome.
Why keep a prediction ID and model version?
They let delayed outcomes join back to the original prediction and identify which fitted release produced it.
Why report outcome coverage beside an error metric?
The metric may describe only a partial or biased subset. Coverage and label delay help show how representative the result is.
Why not retrain automatically whenever an input distribution changes?
The cause may be a pipeline bug, seasonality, or an unsupported population. Retraining may not repair it and can learn from corrupted data.
From a notebook to a maintained prediction system
Key Points
- Save the fitted pipeline with its input contract, environment, and version.
- Serve it through a validated interface and test real prediction behaviour.
- Package and release it with readiness checks, load evidence, and rollback.
- Monitor service health, inputs, outcomes, and important groups separately.
- Investigate changes and validate new candidates before promoting them.
This completes the numbered ML Learn chapters. You can now frame a problem, prepare data, fit and evaluate models, inspect unlabelled structure, and explain how predictions are shipped and maintained. The optional next bridge connects Logistic Regression to a single neuron and then to neural networks.
Theia
Can you explain this concept?
Practise After Launch: Monitoring and Drift with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.