Logistic Regression: Turning Scores into Probabilities
A line can score a case, but a probability is more useful
A linear model can calculate a weighted score. For classification, we want a value between 0 and 1 that can be read as an estimated probability for the positive class. Logistic Regression applies a smooth function to the score.
Key Points
- The linear score is z = w₁x₁ + ... + b.
- The sigmoid maps any real score to a value between 0 and 1.
- The probability can be converted to a class using a threshold.
- The weights still describe a direction in feature space, but probability interpretation needs care.
Linear alone
Walks straight off the scale
Nothing stops it predicting 1.4 or minus 0.2, and neither is a probability.
Plus a sigmoid
Cannot leave 0 and 1
Same linear part underneath. The curve crosses 0.5 exactly where the line crossed zero, and that crossing is your decision boundary.
The sigmoid squeezes scores into a range
The sigmoid formula is σ(z) = 1 ÷ (1 + e⁻ᶻ). z is the linear score, e is the natural exponential base, and σ(z) is the resulting probability. Large positive scores move close to 1; large negative scores move close to 0.
1import math
2
3def sigmoid(score):
4 return 1 / (1 + math.exp(-score))
5
6for score in [-3, 0, 3]:
7 print(f"score={score}, probability={sigmoid(score):.3f}")score=-3, probability=0.047 score=0, probability=0.500 score=3, probability=0.953
A score of zero maps to 0.5. The sigmoid is monotonic, so a larger score always produces a larger positive-class probability.
A classification prediction has a threshold
With a threshold of 0.5, probabilities below 0.5 become class 0 and probabilities at or above 0.5 become class 1. The threshold is not a law of nature. A fraud team may choose a lower threshold to catch more suspicious payments.
Key Points
- Lower threshold: more positive predictions, usually more recall and more false positives.
- Higher threshold: fewer positive predictions, usually more precision and more false negatives.
- Choose the threshold on validation data, not by repeatedly checking the final test set.
- The threshold and the model parameters are separate decisions.
The loss teaches probabilities to respect labels
Logistic Regression commonly uses log loss, also called binary cross-entropy. For a true label y and predicted positive probability p, the loss is −[y log(p) + (1 − y) log(1 − p)]. The formula strongly penalises confident wrong predictions.
Key Points
- If y = 1, high p has low loss and low p has high loss.
- If y = 0, low p has low loss and high p has high loss.
- The loss uses probabilities, not only the final thresholded labels.
- A model can have the same accuracy as another while having better or worse probability quality.
Trace one probability by hand
Suppose z = 2. The sigmoid is 1 ÷ (1 + e⁻²), approximately 0.881. With threshold 0.5, the predicted class is 1. The probability says the model is confident, but confidence is only useful when calibration and evaluation support it.
1import math
2
3score = 2
4probability = 1 / (1 + math.exp(-score))
5label = int(probability >= 0.5)
6
7print(f"Positive probability: {probability:.3f}")
8print("Predicted class:", label)Positive probability: 0.881 Predicted class: 1
Weights describe log-odds, not direct probability points
In Logistic Regression, a weight changes the linear score z. The score is also called log-odds: log(p ÷ (1 − p)). A one-unit feature change adds its weight to log-odds while other features stay fixed in the model.
Key Points
- A weight does not mean “probability increases by exactly 0.2” everywhere.
- The same score change has a larger probability effect near 0.5 than near 0.99.
- Exponentiating a weight gives an odds ratio when the feature unit and model assumptions support that interpretation.
- Correlation and confounding still limit causal conclusions.
Train, validate, then choose the threshold
Key Points
- Fit parameters on training data.
- Use validation data to compare model settings and thresholds.
- Inspect confusion-matrix metrics and probability quality.
- Use a final untouched test set after the process is locked.
- Check performance for groups and operating ranges that matter.
Logistic Regression is a strong baseline because it is fast, interpretable relative to many models, and often gives a useful linear decision boundary. It cannot represent every curved boundary without feature engineering.
Where people get confused
Key Points
- “Logistic Regression predicts a continuous number.” It predicts class probabilities, then often class labels.
- “A 0.8 probability means the event must happen.” It is an estimate, not a guarantee.
- “The 0.5 threshold is mandatory.” It is a default decision rule that can be changed.
- “A large weight means a fixed probability increase.” Weights change score or log-odds, while probability effects depend on the current score.
- “Accuracy trains the model.” Logistic Regression usually fits a probability loss such as log loss.
Quick check
What does the sigmoid function do?
It maps any real-valued linear score to a number between 0 and 1, which can be interpreted as a positive-class probability under the model.
What changes when the classification threshold is lowered?
More examples are labelled positive. This often catches more positives but can also create more false positives.
Why does log loss punish confident wrong predictions?
The logarithm makes a probability near zero very costly when the true label is one, and makes a probability near one very costly when the true label is zero.
What does a Logistic Regression weight directly change?
It changes the linear score and therefore the log-odds for a one-unit feature change while other features stay fixed in the model. The probability change is not constant across all scores.
Recap
Key Points
- Logistic Regression applies a sigmoid to a weighted score.
- The output probability and classification threshold are separate.
- Log loss rewards honest probabilities and penalises confident mistakes.
- The next lesson counts correct and incorrect class decisions with a confusion matrix.
Theia
Can you explain this concept?
Practise Logistic Regression: Turning Scores into Probabilities with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.