Bias vs Variance: Two Ways a Model Can Miss
One bad score can have different causes
Two models can have the same validation error for completely different reasons. One may be consistently too simple. Another may be very sensitive to which training rows it received. Bias and variance give us language for separating these patterns.
Key Points
- Bias is error from assumptions that are too restrictive or systematically miss the relationship.
- Variance is sensitivity to the particular training sample.
- The terms describe sources of prediction error, not moral qualities or fairness by themselves.
- The same model family can show different bias and variance on different datasets.
Tip:Think of bias as a model repeatedly missing in a similar way, and variance as a model changing its answer too much when the training sample changes.
High bias looks like a model that never learns enough
A high-bias model makes strong simplifying assumptions. A straight line used for a curved relationship may miss the pattern in a consistent direction. More training rows do not magically give that model the ability to draw a bend.
Key Points
- Training error is often high.
- Validation error is also high and not dramatically higher than training error.
- A richer feature set or model family may reduce the systematic miss.
- Noise in the target can place a lower limit on achievable error.
High variance looks like a model that changes its mind
A high-variance model reacts strongly to the particular rows used for training. If one training sample contains a few unusual flats, a flexible model may bend towards them. A different sample may produce a noticeably different curve.
Key Points
- Training error is often low.
- Validation error is higher and can move a lot across different splits.
- More representative data, simpler structure, or regularisation may stabilise the model.
- Variance is especially concerning when the dataset is small compared with model capacity.
See the train and validation pattern
1patterns = {
2 "high_bias": (18, 20),
3 "balanced": (8, 10),
4 "high_variance": (2, 27),
5}
6
7for name, (train_error, validation_error) in patterns.items():
8 gap = validation_error - train_error
9 print(f"{name}: train={train_error}, validation={validation_error}, gap={gap}")high_bias: train=18, validation=20, gap=2 balanced: train=8, validation=10, gap=2 high_variance: train=2, validation=27, gap=25
The numbers are a diagnostic sketch, not a universal threshold. High bias is suggested by high errors on both sets. High variance is suggested by a low training error and a much higher validation error.
The error tradeoff changes with capacity
As capacity increases from very low, bias often falls because the model can express more of the signal. After a point, variance can rise because the model starts following sample-specific noise. Validation error can therefore decrease, reach a useful region, and then increase.
Key Points
- Very simple model: high bias, lower variance.
- Moderately flexible model: lower bias with manageable variance.
- Very flexible model: very low training error but potentially high variance.
- The best capacity is selected by performance on data not used to fit the parameters.
This is the bias-variance tradeoff. It is not a command to choose the middle-sized model blindly. It is a reason to measure how error changes as capacity, data, and regularisation change.
Noise creates irreducible error
Some target variation cannot be predicted from the features available at decision time. Two flats with identical recorded features may have different realised rents because of an unrecorded view, negotiation, or temporary event. That uncertainty is often called irreducible error.
Key Points
- Reducing bias and variance cannot remove information that is absent from the inputs.
- Better labels and better features can lower the practical error floor.
- A model that predicts every noisy target perfectly on training data may be memorising noise.
- Communicate uncertainty and typical error instead of promising perfect prediction.
Different interventions target different problems
Key Points
- High bias: consider useful features, a more expressive model, or enough training.
- High variance: consider more representative data, a simpler model, regularisation, or stronger validation.
- Both high: inspect target quality, feature signal, and whether the problem is predictable at all.
- A validation design that is too optimistic: fix the split or leakage before tuning the model.
The same action can have side effects. Increasing capacity may reduce bias while increasing variance. Regularisation may reduce variance while increasing bias. Measure the result rather than assuming one intervention is always good.
Bias and variance are not fairness by themselves
In interviews, bias can mean systematic prediction error. In fairness discussions, bias can refer to unequal outcomes or treatment across groups. The words are related to systematic problems but are not interchangeable.
Key Points
- Use “statistical bias” when discussing systematic error from model assumptions.
- Measure errors by relevant groups when fairness matters.
- A model can have low overall variance and still perform inconsistently for one group.
- A good bias-variance diagnosis does not replace a fairness analysis.
Use repeated samples to see variance
One train-validation split can hide instability. Repeating the split, or using cross-validation later in this chapter, lets you see whether the model’s score and parameters change substantially across samples.
1validation_errors = [10, 12, 11, 29, 8]
2
3average_error = sum(validation_errors) / len(validation_errors)
4spread = max(validation_errors) - min(validation_errors)
5
6print(f"Average validation error: {average_error:.1f}")
7print(f"Best-to-worst spread: {spread}")Average validation error: 14.0 Best-to-worst spread: 21
An average alone hides the unusually poor split. Report variation or the individual fold scores when stability matters.
Where people get confused
Key Points
- "High bias means the model is unfair." In this lesson, bias means systematic prediction error. Fairness needs group-specific analysis.
- "High variance means a large variance in the target." It means the fitted model is sensitive to the training sample.
- "More data always fixes bias." More data can reduce variance, but it cannot give a model a missing ability to represent the pattern.
- "The best model has zero bias and zero variance." Noise and finite data make that goal unrealistic.
- "One split proves stability." Repeated samples or cross-validation provide stronger evidence.
Quick check
How does high bias usually appear in train and validation errors?
Both errors are high and often fairly close. The model is systematically missing useful structure rather than only memorising the training rows.
How does high variance usually appear?
Training error is low while validation error is much higher or changes substantially across different samples.
Why can adding capacity reduce one kind of error while increasing another?
A more flexible model can express more of the signal, reducing bias, but it also has more ways to follow sample-specific noise, increasing variance.
Why inspect several validation splits?
A single split may be unusually easy or hard. Several splits show the average performance and how sensitive the model is to the sample.
Recap
Key Points
- Bias is systematic error from assumptions that are too restrictive.
- Variance is sensitivity to the particular training sample.
- Capacity, data size, noise, and regularisation influence the tradeoff.
- Use train and validation patterns plus repeated evaluation to diagnose the problem.
- The next lesson introduces Ridge and Lasso as tools that can control variance.
Theia
Can you explain this concept?
Practise Bias vs Variance: Two Ways a Model Can Miss with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.