Home / Glossary / Logistic Regression
StatisticsLogistic Regression
By Sitraka Forler · Lecturer, Durham Business SchoolUpdated 23 August 2026 About this site
Classification model that turns a linear score into a probability through the sigmoid.
Logistic regression models the log-odds of an event as a linear function of the features, then squashes that score through the sigmoid so the output lies between 0 and 1. It is fitted by maximum likelihood (minimising log-loss), not by least squares, and each coefficient becomes an odds ratio once exponentiated. In finance it is the standard baseline for default prediction and up/down market classification.
The intuition
A straight line can output any number, but a probability must sit between 0 and 1. Logistic regression keeps the linear score z = β₀ + β₁x and pushes it through the sigmoid σ(z) = 1 / (1 + e^(−z)), an S-curve that maps −∞ to 0, 0 to 0.5 and +∞ to 1.
Undo the sigmoid and the model is linear again, in log-odds: log(p / (1 − p)) = β₀ + β₁x. So e^(β₁) is an odds ratio: each extra unit of x multiplies the odds of the event by e^(β₁). That is the sentence to write in a report.
There is no closed-form solution. The coefficients maximise the likelihood of the observed 0/1 labels, which is the same as minimising log-loss (cross-entropy). That loss is convex, so gradient descent or Newton's method finds the unique optimum.
Formula / theory
P(y=1 | x) = σ(β₀ + β₁x) = 1 / (1 + e^(−(β₀ + β₁x))) ; log(p / (1 − p)) = β₀ + β₁x
In Python
from sklearn.linear_model import LogisticRegression clf = LogisticRegression().fit(X, y) clf.predict_proba(X)[:, 1] # probability of class 1 import numpy as np; np.exp(clf.coef_) # odds ratios
From score to probability, by hand
- Model: β₀ = −1, β₁ = 0.5, and a borrower with feature x = 4 (say, debt-to-income in tens of percent).
- Linear score: z = −1 + 0.5 × 4 = 1.
- Probability: p = 1 / (1 + e^(−1)) = 1 / (1 + 0.3679) = 1 / 1.3679 ≈ 0.731.
- Odds: p / (1 − p) = 0.731 / 0.269 ≈ 2.72 = e^1. Log-odds = 1 = z, as it must be.
- Decision boundary: p = 0.5 exactly when z = 0, i.e. x = 2. Odds ratio per unit of x: e^(0.5) ≈ 1.649, so each extra unit of x multiplies the odds of default by about 1.65.
The model is a line in log-odds space and an S-curve in probability space. Threshold at 0.5 only if a false positive costs the same as a false negative; in credit and fraud it never does.
Common pitfalls
- Judging by accuracy on imbalanced data: with 2 % defaults, predicting 'no default' for everyone scores 98 %. Use precision, recall, ROC-AUC and a confusion matrix weighted by real costs.
- Reading coefficients as probabilities. They act on log-odds; exponentiate for odds ratios, or compute marginal effects at a given x.
- Perfect separation: if a feature splits the classes perfectly, the likelihood has no maximum and coefficients run off to infinity. Regularise (sklearn's default C = 1 already does) or drop the feature.
- Forgetting to scale features before a regularised fit, so the penalty hits large-unit features hardest.
- Trusting predicted probabilities without a calibration check (reliability curve): a model that ranks well can still be badly calibrated.
Frequently asked questions
Why is it called regression if it classifies?
Because it regresses the log-odds of the class on the features. Classification only happens when you choose a threshold on the predicted probability.
Logistic regression or a random forest?
Start with logistic regression: it is fast, interpretable, usually well calibrated and a strong baseline on tabular data. Move to trees or boosting when interactions and non-linearities matter and you can measure the gain out of sample.
What is the loss function?
Log-loss (binary cross-entropy): −[y log p + (1 − y) log(1 − p)], averaged over observations. Minimising it is equivalent to maximising the likelihood.
Can it handle more than two classes?
Yes. Multinomial (softmax) logistic regression generalises the sigmoid to K classes; sklearn's LogisticRegression does this automatically when y has more than two labels.
Test yourself
Further reading
Where to go deeper. Free means a full, legal copy is online.
Chapter 4 for logistic regression, odds and the comparison with LDA.
Section 4.4 for the maximum-likelihood fit and the Newton (IRLS) algorithm.
Chapter 4 for the practical scikit-learn workflow and regularisation.
Free companion notebooks