Overfitting — The Silent Portfolio Killer 💀

Monday, 9:30am. Your fund goes live with a strategy the research team polished for six months. Backtested over 10 years of history it returned 31% a year with barely a losing month — an equity curve like a staircase to heaven. It loses money on day one. And day two. And the rest of the quarter. The market did not change. The strategy never worked: it had not learned how markets behave — it had memorised what already happened, noise included. That failure mode is called overfitting, and it is the single most important concept in financial machine learning.

🧠 The mental model: memorising past papers vs learning the subject

Two students prepare for the same economics exam.

• ›Student A learns the subject: supply and demand, elasticity, how to reason through a new problem.
• ›Student B memorises the answer keys of ten years of past papers, word for word.

Quiz them on those past papers and B wins — a perfect score. But the real exam asks new questions. A performs about as well as ever; B collapses, because memorised answers only match questions that already happened.

Fitting the past is easy. Predicting is hard. Only exam day counts.

The flexibility knob: polynomial degree

In this lesson the knob is the degree of a polynomial regression: PolynomialFeatures(degree=d) expands one signal x into the columns x, x^2, …, x^d, and LinearRegression — literally the OLS you know from econometrics — fits a coefficient to each. Degree 1 is a straight line (2 coefficients). Degree 12 is a snake with 13 coefficients that can twist through almost any 30 points you give it.

Econometrics already taught you the trap: adding regressors never lowers in-sample R². More flexibility always looks better on the estimation sample — that is an algebraic fact, not evidence of skill. The only question that matters is whether the extra bends captured signal or noise. Here is the pattern you will see over and over:

MSE (mean squared error) is the score: the average of squared prediction mistakes — lower is better. Judged on the training set alone, the flexible model wins every single time. That is precisely why in-sample performance is worthless as evidence.

Fix #1: the hold-out set

Never grade a model on the data it trained on. Split before fitting, train on one part, and keep the rest locked in a drawer until judgment day. The test set is a dress rehearsal for going live — the closest thing you have to the future.

Fix #2: cross-validation — every point takes a turn at the exam

A single split can get lucky. k-fold cross-validation slices the data into k folds and rotates: train on k−1 folds, score on the one held out, repeat k times, average. Every observation sits the exam exactly once, and the averaged score has far less luck in it.

⚠️ The finance twist: time only flows one way

Standard k-fold shuffles the data before slicing — and on market data that quietly trains your model on 2023 to "predict" 2019. That is look-ahead bias, and it makes almost any strategy look brilliant. In finance the folds must respect time order:

 Python
from sklearn.model_selection import TimeSeriesSplit, cross_val_score
cv = TimeSeriesSplit(n_splits=5)   # every fold trains on the past, tests on the future
scores = cross_val_score(model, X, y, cv=cv, scoring='neg_mean_squared_error')

This walk-forward discipline is the backbone of honest strategy evaluation — the backtesting lesson builds a full walk-forward engine on top of it.

When overfitting needed a bailout

LTCM, 1998 — institutional-scale overfitting. Two Nobel laureates on the payroll, models fitted to years of spread-convergence history, leverage around 25:1, a near-perfect in-sample record. When Russia defaulted in August 1998, live data stopped resembling the training data: the fund lost $4.6 billion in a few months and the Federal Reserve had to organise a bailout to protect the financial system. Their backtest looked magnificent to the very end.

Your Task

The code simulates 40 days of a (signal, return) relationship whose true form is a straight line drowned in noise. The first 30 days are your backtest history; the last 10 days arrive after you go live. Two models are already fitted on the history — degree 1 (simple) and degree 12 (flexible) — and their train MSEs are computed.

👉 Compute test_mse_1 and test_mse_12: each model's error on the 10 live days, using the same mean_squared_error(...) pattern as the train lines just above them.

Predict before you run: which model posts the lower train MSE — and which one wins on the test set, and roughly by how much? Write both guesses down, then run and check yourself.