Decision Trees: Ask a Sequence of Useful Questions
A tree turns features into questions
A Decision Tree predicts by asking a sequence of feature questions. “Is the order value above ₹1,000?” sends a row down one branch. Another question continues the path until a leaf gives a class prediction.
Key Points
- Root is the first split.
- Internal nodes ask questions.
- Branches represent answers or ranges.
- Leaves store a prediction or class distribution.
- The path can be explained row by row.
A split should make child groups more pure
A classification tree searches for splits that make the resulting child groups contain more similar labels. Gini impurity and entropy are common measures. A pure node has one class; a mixed node contains several classes.
Gini impurity for class probabilities pᵢ is 1 − Σpᵢ². A node with 80% safe and 20% spam has lower impurity than a node with 50% safe and 50% spam.
1def gini(counts):
2 total = sum(counts)
3 probabilities = [count / total for count in counts]
4 return 1 - sum(probability ** 2 for probability in probabilities)
5
6print(f"Pure node: {gini([10, 0]):.2f}")
7print(f"Mixed node: {gini([5, 5]):.2f}")Pure node: 0.00 Mixed node: 0.50
A leaf predicts from its training labels
If a leaf contains 8 safe and 2 spam training rows, its majority class is safe. A probability-style output might be 0.2 spam before smoothing or calibration. The leaf can be easy to explain, but a small leaf may be unstable.
Key Points
- Leaf prediction is based on rows reaching that leaf.
- A minimum leaf size can reduce tiny, noisy leaves.
- A probability from class frequency is not automatically calibrated.
- The same feature can appear at different depths.
Depth controls capacity
A shallow tree asks only a few questions and may underfit. An unrestricted tree can keep splitting until leaves describe individual training rows, creating very low training error and high variance.
Key Points
- Maximum depth limits the number of decision steps.
- Minimum samples per split or leaf prevents fragile branches.
- Pruning removes branches that do not improve validation performance.
- Choose complexity with validation evidence.
Trace one path
1def approve_order(value, previous_orders):
2 if value < 500:
3 return "review"
4 if previous_orders >= 3:
5 return "approve"
6 return "review"
7
8print(approve_order(700, 4))
9print(approve_order(700, 1))approve review
This hand-written rule is not a trained tree, but it shows the path idea. A learned tree chooses thresholds from data and can contain many branches.
Trees handle feature shapes naturally
Key Points
- Trees can represent nonlinear boundaries through repeated splits.
- They do not require feature scaling for ordinary threshold splits.
- They can use numeric and encoded categorical features.
- Missing-value handling and category encoding still need an explicit pipeline.
- Very high-cardinality identifiers can create memorising splits.
A split is useful only if it generalises
A tree can find a threshold that separates the training labels by chance. Check validation performance, leaf sizes, feature availability, and whether the split represents a meaningful future rule.
Key Points
- Inspect train-validation gaps.
- Limit depth or leaf size when variance is high.
- Look for leakage in features that create suspiciously perfect leaves.
- Do not interpret a split as causal evidence automatically.
Where people get confused
Key Points
- “Trees always need scaling.” Threshold trees usually do not, though preprocessing may still be needed for other reasons.
- “A tree is automatically interpretable.” A very deep tree can be impossible for a person to follow.
- “The root split is the cause.” It is the first split selected by the training objective.
- “Pure leaves mean a good model.” They may mean the tree memorised the training rows.
- “A tree cannot model curves.” Repeated axis-aligned splits can approximate nonlinear regions.
Quick check
What does a Decision Tree internal node do?
It asks a feature-based question, such as whether a value is below a threshold, and sends the row down a branch.
What does impurity measure?
It measures how mixed the class labels are in a node. A pure node has one class and low impurity.
Why can an unrestricted tree overfit?
It can keep creating small branches that match accidental details or individual training rows, producing low training error but poor generalisation.
What are common ways to control tree complexity?
Limit depth, require minimum samples for splits or leaves, prune branches, and select settings with validation data.
Recap
Key Points
- Decision Trees classify by following feature questions to leaves.
- Impurity measures help choose useful splits.
- Depth and leaf-size settings control variance.
- Trees capture nonlinear boundaries but can memorise noise.
- The next lesson combines many trees in a Random Forest.
Theia
Can you explain this concept?
Practise Decision Trees: Ask a Sequence of Useful Questions with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.