Skip to content
Kuza Ndoto Academy

Logistic Regression

IntermediateChapter 3 · Page 10 of 264 min readUpdated 15 Nov 2026

Despite its name, logistic regression is a classification algorithm. It predicts the probability that an example belongs to a class (pass or fail, spam or not, default or repay) and is one of the most widely used models in banking, health and marketing because it's simple and explainable.

From a line to a probability

Linear regression outputs any number, from minus infinity to plus infinity, but a probability must lie between 0 and 1. Logistic regression first computes a linear score, exactly like linear regression:

z = w₁x₁ + w₂x₂ + … + b

Then it squashes z through the sigmoid (logistic) function:

σ(z) = 1 / (1 + e−z)

Large positive z gives a probability near 1; large negative z gives a probability near 0; z = 0 gives exactly 0.5.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

for z in [-2, 0, 2]:
    print(z, round(sigmoid(z), 3))
-2 0.119 0 0.5 2 0.881

A worked example

Suppose a model predicting whether a student passes has learned z = 1.5 × hours − 6.

  • 2 hours: z = 3 − 6 = −3, so P(pass) = 1 / (1 + e3) ≈ 0.047.
  • 4 hours: z = 6 − 6 = 0, so P(pass) = 0.5. This is the decision boundary.
  • 6 hours: z = 9 − 6 = 3, so P(pass) ≈ 0.953.
012345678 00.51 Hours studied P(pass) threshold 0.5 decision boundary (4 h) predict fail predict pass
Figure 1. The sigmoid for z = 1.5 × hours − 6. The marked points are 2 hours (0.047), 4 hours (0.5) and 6 hours (0.953).

To turn a probability into a class, use a threshold, normally 0.5: predict "pass" if P ≥ 0.5. You can move the threshold. A hospital screening test might flag anyone above 0.2, accepting more false alarms in exchange for missing fewer sick patients.

Coefficients have a useful reading in terms of odds (P divided by 1 − P): each extra hour multiplies the odds of passing by e1.5 ≈ 4.48.

How it is trained

Logistic regression is trained by minimising log loss (cross-entropy). For each example the loss is −log(probability given to the correct class). Predicting 0.9 for a student who really passed costs −log(0.9) ≈ 0.105, a small penalty. Predicting 0.1 for that student costs −log(0.1) ≈ 2.303, a big penalty. Confident wrong answers are punished hardest. There's no closed-form formula, so an optimiser adjusts the weights step by step, in the spirit of gradient descent.

Logistic regression in scikit-learn

from sklearn.linear_model import LogisticRegression

hours = [[1], [2], [3], [4], [5], [6], [7], [8]]
passed = [0, 0, 0, 0, 1, 1, 1, 1]

model = LogisticRegression()
model.fit(hours, passed)

print(model.predict([[1.5], [7.5]]))
print(model.predict_proba([[4.5]]).round(2))

The first line prints [0 1]: fail for 1.5 hours, pass for 7.5 hours. The second prints the probabilities of [fail, pass] for 4.5 hours. Because this data is perfectly symmetric around 4.5 hours, you should see both values very close to 0.5, i.e. [[0.5 0.5]]. The exact probabilities further from the boundary depend on regularisation (the C parameter), so we don't quote them.

Note

scikit-learn's LogisticRegression applies L2 regularisation by default, which keeps the weights from growing huge. Smaller C means stronger regularisation. Because of this, scale your features (see Feature Scaling) so the penalty treats them equally.

More than two classes

Logistic regression also handles several classes (for example, three iris species). With the default solver, scikit-learn fits a multinomial model that uses the softmax function, a generalisation of the sigmoid that gives one probability per class, all adding up to 1. You use it exactly the same way.

Strengths and limits

  • Strengths: fast, outputs probabilities, coefficients are interpretable, a strong baseline for classification.
  • Limits: the decision boundary is a straight line (a flat plane with many features). Curved boundaries need engineered features or a different model such as a decision tree.
Common mistake

Judging a logistic regression only by accuracy on imbalanced data. If only 2% of loans default, a model that always predicts "repay" is 98% accurate and completely useless. Look at precision, recall and the confusion matrix (see Evaluation Metrics), and consider adjusting the threshold or using class_weight="balanced".

Key points

  • Logistic regression is a classifier: a linear score passed through the sigmoid gives a probability.
  • A threshold (default 0.5) turns probabilities into classes; you can move it.
  • It's trained by minimising log loss, which punishes confident mistakes.
  • It's fast, explainable and a great first model for yes/no problems.
Up next · Supervised LearningDecision TreesModels that ask a series of questions.
→

Tip: use the ← and → keys to move between pages.