Class Imbalance: When the Rare Case Matters Most
A 99 percent score can still miss every alert
Class imbalance means one class appears much more often than another. If only 1% of transactions are fraud, a model that always predicts “not fraud” gets 99% accuracy and catches zero fraud.
Key Points
- The majority class dominates the row count.
- The minority class may carry the important or expensive cases.
- Accuracy can reward a model for ignoring the minority class.
- Start with class counts and a majority baseline.
What the headline says
99.9% accurate
99.9%
accuracy
Out of 10,000 transactions it got 9,990 right. Reported on its own, this number ends the conversation.
What the matrix says
It caught nothing
caught
0
false alarm
0
missed
10
correctly cleared
9,990
Ten frauds, none found. A model that answers no to everything scores 99.9% here, and this is that model.
Tip:Rare does not mean unimportant. The positive class should be chosen from the decision, not from which label has more rows.
Measure the baseline before the model
1actual = [0] * 95 + [1] * 5
2always_negative = [0] * len(actual)
3
4accuracy = sum(real == guess for real, guess in zip(actual, always_negative)) / len(actual)
5recall = sum(real == 1 and guess == 1 for real, guess in zip(actual, always_negative)) / sum(actual)
6
7print(f"Majority baseline accuracy: {accuracy:.2f}")
8print(f"Majority baseline recall: {recall:.2f}")Majority baseline accuracy: 0.95 Majority baseline recall: 0.00
The baseline is not a useless calculation. It tells us how much the model must improve beyond class frequency, especially for the class we care about.
Look beyond one overall score
Key Points
- Confusion matrix shows which class is being ignored.
- Precision and recall reveal positive-class behaviour.
- Balanced accuracy averages recall across classes.
- Per-class metrics matter in multiclass problems.
- PR curves can be more informative than ROC curves when positives are rare.
No metric fixes an unclear decision. Decide whether missing a rare positive, reviewing a false alarm, or spending a limited intervention budget is the main concern.
Thresholds are a practical control
A lower probability threshold marks more cases as positive. This can improve recall for the rare class, but it increases the review queue and may reduce precision. Choose the threshold on validation data using a realistic workload or cost constraint.
Key Points
- The threshold can change without retraining the model.
- A review team may have a maximum daily capacity.
- A safety process may require a minimum recall.
- Report the threshold with the metric values.
Class weights change the learning emphasis
Many algorithms allow class weights. A larger weight for the minority class makes errors on its examples count more during training. This can improve minority recall, but it may also reduce precision or probability calibration.
Key Points
- Class weighting changes the objective, not the underlying labels.
- Select weights with validation evidence.
- Do not judge class weighting from training accuracy alone.
- Record the weights as part of the model configuration.
Resampling must happen inside training data
Oversampling duplicates or synthesises minority examples. Undersampling removes some majority examples. Either can help training, but the validation and test sets should keep the real deployment class distribution unless the evaluation question explicitly says otherwise.
Key Points
- Never oversample before splitting; duplicates can cross the boundary.
- Apply resampling only to each training fold.
- Report results on untouched data.
- Synthetic examples still need a valid feature and label story.
Compare metrics at one operating point
1models = [
2 {"name": "always_negative", "precision": 0.00, "recall": 0.00, "review_rate": 0.00},
3 {"name": "balanced_threshold", "precision": 0.42, "recall": 0.80, "review_rate": 0.10},
4 {"name": "cautious_threshold", "precision": 0.75, "recall": 0.45, "review_rate": 0.03},
5]
6
7for model in models:
8 print(
9 f"{model['name']}: precision={model['precision']:.2f}, "
10 f"recall={model['recall']:.2f}, review_rate={model['review_rate']:.2f}"
11 )always_negative: precision=0.00, recall=0.00, review_rate=0.00 balanced_threshold: precision=0.42, recall=0.80, review_rate=0.10 cautious_threshold: precision=0.75, recall=0.45, review_rate=0.03
The right row depends on the cost of misses, false alarms, and review capacity. The table makes those tradeoffs visible instead of hiding them inside accuracy.
Split and monitor the rare class carefully
A random split can accidentally place very few positives in validation or test. Stratified splitting aims to preserve class proportions, while time or group constraints may require a different design. Always inspect the actual counts in each split.
Key Points
- Confirm every evaluation split contains enough minority examples.
- Use confidence intervals or repeated evaluation when rare counts are small.
- Check whether the class rate changes over time.
- A shift in prevalence can change precision even when recall stays similar.
Where people get confused
Key Points
- “High accuracy means the rare class is handled.” It may mean the model always predicts the majority class.
- “Oversampling creates new information.” It changes training emphasis; it does not create guaranteed new signal.
- “Class weights solve imbalance.” They change the objective but do not fix poor features, labels, or threshold choice.
- “The test set should be balanced to make evaluation fair.” It should represent the evaluation question; changing prevalence changes the meaning of precision.
- “A rare class is always positive.” Positive is the class selected for the decision.
Quick check
Why can a majority-class baseline have high accuracy?
When one class dominates, predicting it for every row is correct often. That does not show the model finds the minority class.
What can a lower threshold do in an imbalanced problem?
It can increase the number of positive predictions and often improve recall, while increasing review volume and potentially reducing precision.
Where should oversampling be applied?
Only inside the training portion or training fold. Validation and test rows should remain untouched so the score reflects the intended data distribution.
Why inspect class counts in every split?
A rare class can be nearly absent from a split, making its metrics unstable or misleading.
Recap
Key Points
- Imbalance can make accuracy look strong while the minority class is ignored.
- Use baselines, confusion matrices, per-class metrics, and realistic thresholds.
- Class weights and resampling change training emphasis and must be validated.
- The next lesson ranks thresholds across all operating points with ROC curves and AUC.
Theia
Can you explain this concept?
Practise Class Imbalance: When the Rare Case Matters Most with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.