Gradient Boosting: Add Small Trees One Correction at a Time
Many weak corrections can make a strong model
Gradient Boosting builds an ensemble sequentially. The first small tree makes a rough prediction. The next tree focuses on the current errors, and later trees add more corrections. The final model is the sum of many small contributions.
Key Points
- Boosting is sequential; Random Forest trees are usually trained independently.
- Each new learner tries to improve the current ensemble.
- Small trees and a learning rate control how quickly the model changes.
- The method can achieve strong accuracy but needs careful tuning.
Start with a baseline prediction
For binary classification, a boosting model may begin with a constant score or probability. The first tree then finds regions where the baseline is systematically wrong. Later trees refine those regions.
Key Points
- The baseline is not discarded; new learners add to it.
- The training loss supplies a direction for correction.
- Classification boosting commonly works with a probability loss such as log loss.
- The correction is not simply “memorise every wrong row”.
Add predictions sequentially
1prediction = 0.40
2corrections = [0.20, -0.05, 0.03]
3learning_rate = 0.50
4
5for correction in corrections:
6 prediction += learning_rate * correction
7 print(f"prediction={prediction:.3f}")prediction=0.500 prediction=0.475 prediction=0.490
The learning rate shrinks each correction. A smaller rate usually needs more trees, but can make the path more gradual and sometimes generalise better.
The residual idea becomes a gradient idea
For squared-error regression, the next tree can fit residuals. For classification and other losses, the next tree follows the negative gradient of the loss. This is why the method is called Gradient Boosting.
Key Points
- A gradient gives the direction that reduces the current loss.
- The next tree approximates a correction function.
- The loss determines what “error” the learner focuses on.
- You do not need to implement the full optimiser to reason about the sequential process.
Learning rate and tree count trade off
Key Points
- High learning rate with few trees can move quickly and overfit.
- Low learning rate with many trees can make smaller corrections.
- Too many trees can still overfit.
- Validation curves help select the number of boosting rounds.
- Early stopping can stop when validation performance stops improving.
Tree depth controls the correction shape
Shallow base trees make simple corrections. Deeper trees can model interactions and detailed regions, but they also increase capacity and overfitting risk.
Key Points
- Depth one creates a stump-like correction.
- More depth can capture feature interactions.
- Subsampling rows or features can add useful randomness.
- Tune depth, learning rate, number of trees, and regularisation together.
Watch validation performance as trees are added
1validation_f1 = [0.62, 0.71, 0.78, 0.80, 0.79]
2best_round = max(range(len(validation_f1)), key=validation_f1.__getitem__) + 1
3
4print("Best number of rounds:", best_round)
5print(f"Best validation F1: {validation_f1[best_round - 1]:.2f}")Best number of rounds: 4 Best validation F1: 0.80
The fifth round improves the training fit in a real system but lowers the validation F1 in this sketch. Early stopping would prefer the fourth round.
Boosting needs disciplined evaluation
Key Points
- Use a validation or cross-validation process for settings.
- Keep preprocessing and feature generation inside the proper folds.
- Choose the metric and threshold for the decision.
- Check calibration if probabilities drive action.
- Inspect error groups and leakage, especially when the score is surprisingly strong.
Gradient Boosting is powerful enough to exploit small quirks in the data. Its power increases the value of the evaluation discipline from Chapter 5.
Where people get confused
Key Points
- “Boosting trains trees in parallel.” The standard process is sequential because each tree responds to the current ensemble.
- “A weak tree is a bad model.” Weak learners are deliberately small so their corrections can combine.
- “More rounds always help.” Validation performance can peak and then decline.
- “Gradient Boosting is the same as Gradient Descent.” Both use a loss direction, but boosting adds learners while Gradient Descent updates parameters.
- “Feature importance proves causation.” It does not.
Quick check
How does Gradient Boosting build an ensemble?
It adds learners sequentially, with each new small tree focusing on the direction that improves the current loss.
What does the learning rate control?
It shrinks each new learner’s contribution. Lower rates make smaller corrections and often require more trees.
Why can early stopping help?
Validation performance can stop improving even while training fit continues to improve. Early stopping limits later corrections that may follow noise.
How is Gradient Boosting different from Random Forest?
Random Forest trees are usually trained independently and combined. Gradient Boosting trains learners sequentially so later learners correct the current ensemble.
Recap
Key Points
- Gradient Boosting adds small trees sequentially.
- The loss supplies the correction direction.
- Learning rate, tree count, depth, and subsampling control the tradeoff.
- Validation and early stopping are important because boosting can overfit.
- The final lesson compares a margin-based classifier, Support Vector Machines.
Theia
Can you explain this concept?
Practise Gradient Boosting: Add Small Trees One Correction at a Time with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.