Home / Glossary / Linear Regression
StatisticsLinear Regression
By Sitraka Forler · Lecturer, Durham Business SchoolUpdated 23 August 2026 About this site
Fit a straight line (or hyperplane) that minimises the sum of squared errors: ordinary least squares (OLS).
Ordinary least squares picks the coefficients that make the squared gap between predictions and observed values as small as possible. It is the workhorse of econometrics and the baseline every machine-learning model must beat. A coefficient reads as: one more unit of x moves the prediction by β, holding the other features fixed. Whether that is causal depends on the design, not on the fit.
The intuition
Least squares is geometry: the observed y is a point, the columns of X span a flat surface, and the fitted ŷ is the shadow of y on that surface. The residual y − ŷ is perpendicular to the surface, which is exactly why residuals are uncorrelated with every feature.
Minimising squared errors is not arbitrary. Under the classical assumptions (linear truth, independent errors with constant variance) OLS is the best linear unbiased estimator (Gauss-Markov), and with normal errors it coincides with maximum likelihood.
Read a coefficient as a comparison, not a cause: β₁ is how much ŷ moves when x₁ rises by one unit while the other features stay fixed. Whether that is causal depends on the design (experiment, instrument, natural experiment), never on the fit.
Formula / theory
y = β₀ + β₁x + ε ; β̂ = (XᵀX)⁻¹Xᵀy minimises Σ(yᵢ − ŷᵢ)²
In Python
from sklearn.linear_model import LinearRegression model = LinearRegression().fit(X, y) model.coef_, model.intercept_, model.score(X, y) # slope(s), intercept, R²
In SQL
-- simple OLS slope of y on x, from the moments SELECT (AVG(x*y) - AVG(x)*AVG(y)) / (AVG(x*x) - AVG(x)*AVG(x)) AS beta_1 FROM observations;
Fit a line through three points by hand
- Data: x = (1, 2, 3), y = (2, 4, 5). Means: x̄ = 2, ȳ = 11/3 ≈ 3.667. Deviations of y from its mean: (−5/3, 1/3, 4/3).
- Slope: β̂₁ = Σ(xᵢ − x̄)(yᵢ − ȳ) / Σ(xᵢ − x̄)² = [(−1)(−5/3) + (0)(1/3) + (1)(4/3)] / (1 + 0 + 1) = (5/3 + 4/3) / 2 = 3/2 = 1.5.
- Intercept: β̂₀ = ȳ − β̂₁ x̄ = 11/3 − 1.5 × 2 = 11/3 − 3 = 2/3 ≈ 0.667.
- Fitted values ŷ = 2/3 + 1.5x = (13/6, 11/3, 31/6) ≈ (2.167, 3.667, 5.167). Residuals y − ŷ = (−1/6, 1/3, −1/6): they sum to zero, as OLS guarantees whenever there is an intercept.
- SSE = (1/6)² + (1/3)² + (1/6)² = 1/36 + 4/36 + 1/36 = 6/36 = 1/6 ≈ 0.167. SST = Σ(yᵢ − ȳ)² = 25/9 + 1/9 + 16/9 = 42/9 = 14/3 ≈ 4.667. R² = 1 − SSE/SST = 1 − (1/6) / (14/3) = 1 − 1/28 = 27/28 ≈ 0.964.
Two means, one ratio, one subtraction: the line explains 96.4 % of the spread in y. On the same three points, sklearn's LinearRegression returns coef_ = 1.5 and intercept_ ≈ 0.667.
Common pitfalls
- Reading R² as proof of a good model: a high R² with a curved residual plot means the line has the wrong shape, not the right answer on average.
- Extrapolating outside the range of x. The fit knows nothing about x = 10 when the data stop at 3.
- Multicollinearity: two near-identical features make XᵀX nearly singular, so coefficients swing wildly from sample to sample while predictions barely change.
- Heteroskedastic or autocorrelated errors (typical in finance) do not bias the coefficients but make the default standard errors and p-values wrong. Use robust (HC) or Newey-West standard errors.
- Dropping the intercept: it absorbs the mean, so without it residuals no longer sum to zero and R² can even turn negative.
Frequently asked questions
What does 'ordinary' mean in ordinary least squares?
That every observation gets the same weight and the errors are assumed to have constant variance. Weighted and generalised least squares relax exactly that.
Is linear regression statistics or machine learning?
Both. Statistics asks what the coefficients mean and how uncertain they are; machine learning asks how well the line predicts new data. Same fit, different questions, so report a coefficient table and an out-of-sample error.
Why minimise squared errors rather than absolute errors?
Squares give a closed-form solution, penalise large misses heavily and match maximum likelihood under normal errors. Absolute errors (median regression) resist outliers better but need an iterative optimiser.
How is OLS related to logistic regression?
Both are linear in the parameters. OLS predicts a continuous y directly; logistic regression passes the same linear score through a sigmoid to get a probability and is fitted by maximum likelihood instead of least squares.
Test yourself
Further reading
Where to go deeper. Free means a full, legal copy is online.
Chapter 3: the cleanest introduction to linear regression and its diagnostics.
Chapters 2-4 for the econometric reading of coefficients, standard errors and Gauss-Markov.
Chapter 3 for the geometry of least squares and shrinkage (ridge, lasso).