Predict Asset Returns with Linear Regression

advanced Python 20 minLinearRegressiontemporal train/test splitlook-ahead bias

What you'll learn

LinearRegression
temporal train/test split
look-ahead bias
R²
information coefficient
StandardScaler
OLS

Predicting Asset Returns - Your First Model That Must Not Cheat 📉

Week one on the quant desk. The intern next to you bursts out: "My model predicts next-day returns with R² = 0.60!" The portfolio manager does not even look up: "Check your split." Sure enough - train_test_split with its default shuffle. The model trained on Wednesday and Friday, then "predicted" Thursday. It was not forecasting the future - it was interpolating between days it had already seen. After fixing the split, R² collapsed to 0.02. The most expensive bugs in quant finance do not crash - they quietly cheat.

🧠 The mental model: the exam and the leaked questions

A model is a student.

Training set = the past exam papers it revises from.
Test set = this year's exam, sealed until exam day.
Temporal split = revise on 2021–2023, sit the 2024 exam. Honest.
Shuffled split = half of this year's questions were mixed into the revision pack. The student aces the exam - and has learned nothing about the real world.

Markets move through time in one direction only. Your evaluation must too.

From econometrics to ML - same OLS, new question

Good news: the estimator is the OLS you already know from econometrics. What changes is the question you ask of it:

	Econometrics (what you learned)	ML prediction (this lesson)
Question	Is β significant? (t-stats, p-values)	Does it predict out-of-sample?
Data	Fit on the whole sample	Train on the past, test on the future
Success	In-sample R², robust SEs	Test R², IC
Failure mode	Omitted variable bias	Look-ahead bias (leakage)

The model itself is the familiar one:

code

r̂[t+1] = β0 + β1·mom_1d + β2·mom_5d + β3·mom_20d + β4·vol_20 + β5·vol_ratio + ε

⚠️ Critical: no look-ahead bias!

Always split train/test by time, never by random shuffle:

Python

# ❌ Wrong - shuffles time, leaks future into past
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# ✅ Correct - strict temporal ordering
split = int(len(df) * 0.8)
X_train, X_test = X[:split], X[split:]

The same rule governs preprocessing: StandardScaler is fitted on the training set only, then applied to both sets. Fit it on all the data and the test set's mean and variance leak into training - a subtler cousin of the same bias.

Evaluation metrics - recalibrate your expectations

Metric	Meaning	Typical finance value
R²	% of return variance explained	0.01 – 0.05 is good
MAE	Mean absolute prediction error	About the size of daily vol
IC	Correlation of predicted vs actual	0.02 – 0.06 is solid

Career insight: an R² of 0.03 on daily returns is genuinely excellent. Markets are nearly efficient - most of tomorrow's move is news that does not exist yet. A model that captures even 3% of the variance, applied across thousands of positions, is a business. Do not expect the 0.9 from ML textbooks.

Your Task

The pipeline is fully written - features, scaling, model, metrics - wrapped in run_pipeline(split). One thing is missing: the temporal split index is still None, so the pipeline politely refuses to run.

Set split so the first 80% of rows train the model and the last 20% test it.
Run it - you get the out-of-sample metrics (R², MAE, IC) and the fitted coefficient table.

Predict before you run: 278 rows survive the feature-window dropna(). What number will Test size: print? And will the test R² land closer to 0.9 or to 0.0? If you catch yourself hoping for 0.9, re-read the metrics table.

Related terms in the glossary

Linear Regression