Overfitting and Underfitting: When Generalisation Breaks
A model must work beyond its study table
A model learns from examples, but the people who use it usually care about new examples. If a rent model only works for the flats it saw during training, it has not learned a useful pattern. It has remembered part of the study table.
Generalisation means performing reasonably on new data drawn from the situation where the model will be used. Training performance is useful evidence, but it is not the final answer.
Key Points
- Training data is used to learn parameters.
- Evaluation data is kept separate to estimate behaviour on new rows.
- A generalising model captures signal that continues beyond the training examples.
- Noise, unusual cases, and changes in the real world limit how good any model can be.
Tip:A low training error is a description of the fit to known rows. It becomes evidence about the future only when the evaluation design supports that conclusion.
Underfitting is too little learning
Underfitting happens when the model is too simple, too constrained, or not trained enough to capture a useful pattern. A straight line trying to describe a strong curve may miss the structure in both the training rows and the new rows.
The model has high error on the training set and high error on the validation or test set. More data may help, but first ask whether the features and model family can express the relationship at all.
Too simple
High bias, underfitting
The dashed line is the real pattern. A straight line cannot follow a hump, and more rows will not change that.
Too eager
High variance, overfitting
It hits every point exactly, so it zigzags away from the dashed line to reach the noise. It has learned this sample, not the pattern.
Key Points
- Training error is high.
- Held-out error is also high and often similar to training error.
- Useful responses can include better features, a more suitable model, or more training.
- Making the model more complex is not always the right response if the data or target is noisy.
Overfitting is too much memorisation
Overfitting happens when a model follows accidental details in the training data. A high-degree polynomial may bend around every observed point, including measurement noise, and then make poor predictions between or beyond those points.
The usual symptom is a large gap: training error is low while validation or test error is much higher. The model has enough capacity to fit the examples, but not enough discipline to retain only the pattern that generalises.
Key Points
- Training error is low.
- Held-out error is noticeably higher.
- The model may be too flexible for the amount or quality of data.
- Useful responses can include simpler features, regularisation, more representative data, or a better validation design.
Read the train and test pattern together
Do not diagnose a model from one score alone. Put training and held-out metrics side by side. The size of the gap and the absolute level of both scores tell different parts of the story.
1models = {
2 "too_simple": {"train_mae": 18, "test_mae": 20},
3 "useful": {"train_mae": 8, "test_mae": 10},
4 "too_flexible": {"train_mae": 2, "test_mae": 27},
5}
6
7for name, scores in models.items():
8 gap = scores["test_mae"] - scores["train_mae"]
9 print(f"{name}: train={scores['train_mae']}, test={scores['test_mae']}, gap={gap}")too_simple: train=18, test=20, gap=2 useful: train=8, test=10, gap=2 too_flexible: train=2, test=27, gap=25
The useful model has lower held-out error than the too-simple model without the large generalisation gap of the too-flexible model. The exact acceptable gap depends on the problem, data size, and consequences of errors.
Capacity is a useful mental model
Model capacity describes how many different patterns a model can represent. A one-parameter rule has less capacity than a polynomial with many powers or a tree allowed to split until every row has its own branch.
Low capacity can miss real structure. Very high capacity can absorb noise. The goal is not the smallest model or the most powerful model. It is enough capacity to capture useful signal while retaining performance on new data.
Key Points
- More parameters or deeper structures often increase capacity.
- More capacity can reduce training error.
- More capacity can increase variance and overfitting when data is limited.
- Capacity is only one factor. Features, labels, noise, and the evaluation split matter too.
A learning curve gives more evidence
A learning curve compares training and validation error as the amount of training data grows. It can reveal whether more representative examples might help or whether the model family is still too simple.
Key Points
- High training and validation error that converge can suggest underfitting or high noise.
- Low training error with high validation error can suggest overfitting.
- A validation curve that improves with more data suggests additional representative data may help.
- Learning curves do not replace checking the data split and target definition.
Underfitting
Both stay high, together
Almost no gap, and neither curve is still falling. It is equally poor on everything, so give it more capacity, not more data.
Overfitting
They separate, then one turns
Training loss keeps falling while validation bottoms out and climbs again. The dotted line is where you should have stopped.
1curve = [
2 {"rows": 20, "train_mae": 3, "validation_mae": 24},
3 {"rows": 80, "train_mae": 5, "validation_mae": 14},
4 {"rows": 200, "train_mae": 7, "validation_mae": 10},
5]
6
7for point in curve:
8 print(
9 f"rows={point['rows']}: "
10 f"train={point['train_mae']}, "
11 f"validation={point['validation_mae']}"
12 )rows=20: train=3, validation=24 rows=80: train=5, validation=14 rows=200: train=7, validation=10
When the model is already flexible and the validation score improves as more data arrives, collecting more varied, correctly labelled examples may be more useful than adding another layer of complexity.
Do not confuse overfitting with every failure
A high test error can come from overfitting, but it can also come from a distribution change, bad labels, missing features, data leakage in the wrong direction, or a test set that does not represent the deployment situation.
Key Points
- Check whether training and test rows come from the same intended population.
- Check whether the target was measured consistently.
- Inspect residuals and important subgroups, not only an overall average.
- Check for features that were unavailable when the prediction would have been made.
- Treat overfitting as a hypothesis to investigate, not a label to attach automatically.
A practical diagnosis sequence
Key Points
- 1. Establish a simple baseline and record train and held-out metrics.
- 2. Confirm that the split reflects how future data arrives.
- 3. Compare training error with held-out error.
- 4. Inspect learning curves, residuals, and important groups.
- 5. Change one thing at a time: features, model capacity, regularisation, or data.
- 6. Re-evaluate on a clean holdout instead of celebrating a training improvement.
This sequence keeps model changes connected to evidence. It also prevents “make the model bigger” from becoming the default answer to every disappointing score.
Where people get confused
Key Points
- "Low training error means the model is good." It means the model fits the training rows well. Generalisation still needs evidence.
- "Overfitting means the model is complex." Complexity can contribute, but leakage, duplicate rows, and a tiny dataset can also create a misleading fit.
- "Underfitting means we need more data." More data may help, but a model with the wrong features or family may remain too simple.
- "A train-test gap is always bad." Some gap is normal. The question is whether its size is acceptable for the decision.
- "A test score is the truth." It is evidence from a particular sample and evaluation design.
Quick check
What is the usual sign of overfitting in train and held-out metrics?
Training error is low but held-out error is much higher. The gap suggests the model fitted details that did not generalise.
What is a common sign of underfitting?
Both training and held-out errors are high and often similar. The model may be too simple, insufficiently trained, or missing useful features.
Why should you inspect the data split before changing the model?
A misleading split or a distribution change can make a model appear to overfit or fail. Model changes cannot repair an evaluation design that does not represent deployment.
What does a learning curve compare?
It compares training and validation performance as the amount of training data grows, helping show whether more data or a different capacity may help.
Recap
Key Points
- Generalisation is performance on new, relevant examples.
- Underfitting means the model misses useful structure in both training and held-out data.
- Overfitting means the model fits training details that do not continue to held-out data.
- Compare train and held-out metrics, learning curves, residuals, and data quality together.
- The next lesson separates two sources of error: bias and variance.
Theia
Can you explain this concept?
Practise Overfitting and Underfitting: When Generalisation Breaks with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.