- feature engineering
- lag features
- momentum
- rolling windows
- look-ahead bias
- information coefficient
Feature Engineering — Building Predictive Signals ⚗️
Monday, 6:40 am, at a stat-arb desk. Overnight, the research cluster retrained the desk's return-prediction model — and it never saw a single price chart. Models cannot read charts, news or vibes. They see exactly one thing: the feature matrix you hand them — rows = days, columns = signals. If a signal is not in a column, it does not exist for the model. Your job today: turn 200 days of raw closes and volumes into that matrix. This step — feature engineering — is where quant funds actually compete.
📜 Sixty years of alpha machines, in brief
1961 — Ed Thorp publishes the first proof that a casino game can be systematically beaten (card counting, building on Baldwin et al.'s 1956 basic-strategy analysis), then points the same probabilistic thinking at markets: his Princeton-Newport Partners compounded roughly 20% a year for two decades. 1978 — Jim Simons, a former Cold War codebreaker at the IDA and co-discoverer of the Chern–Simons invariants (1974, with Chern), founds the firm that becomes Renaissance Technologies. 1988 — Medallion launches; its ML-driven signals return about 66% a year before fees over three decades — the greatest track record in history. 2000s — D.E. Shaw, Two Sigma, Citadel, AQR and WorldQuant industrialise the craft. Every one of them competes on exactly what you build today: features. The blow-ups were feature failures too — LTCM in 1998 (the overfitting lesson tells that story) and the 2007 Quant Quake (see the tips): features that stopped working when the regime changed, or that everyone else had also built.
The features you build determine your edge. But they also determine your correlation to every other quant who reads the same papers.
🧠 The mental model: the analyst behind a locked door
Your model is a brilliant analyst locked in a windowless room. Once a day you slide a single spreadsheet under the door; they slide back a forecast. They never see Bloomberg, never hear the news — the spreadsheet is their entire universe. Feature engineering is deciding what goes in the columns. Slide in raw prices and you get a shrug; slide in "how has the stock trended this week? this month? how choppy has it been? is volume unusual?" and you get forecasts.
Econometrics students: you already know this game. Features are the regressors in your X matrix. ML did not abolish OLS — it renamed the design matrix and industrialised the search for good columns.
The ML pipeline
Common Financial Features
| Feature | Formula | Intuition |
|---|---|---|
| Momentum 1d | r[t-1] | Yesterday's trend |
| Momentum 5d | sum(r, last 5 days) | Weekly trend |
| Momentum 20d | sum(r, last 20 days) | Monthly trend |
| Realised vol | std(r, last 20 days) | Volatility clustering |
| Volume ratio | V[t] / mean(V, last 20 days) | Volume confirmation |
Target Variable
y[t] = log(P[t+1] / P[t]) — we predict the next day's log return. Tomorrow, not today.
Two rules that keep you honest
| Rule | Why |
|---|---|
| Features must be older than the target | Features and target must never overlap in time. Here the target is tomorrow's return, and every feature is additionally lagged one day with .shift(1) — an extra-safe house convention that costs one day of information. (The direction-prediction lesson uses the tighter, equally leak-free convention — unshifted features with a next-day target — while the belt-and-braces .shift(1) here trades one day of information for extra safety. Both are leak-free; the rule that matters is that a feature is never newer than its target.) |
.dropna() before modelling | Rolling windows need warm-up: a 20-day window is NaN for the first rows, and the final row has no next-day target. Those rows are honest casualties — drop them. |
Career insight: Feature quality matters far more than model choice. A strong feature with a linear model usually beats a weak feature with a neural network.
Your Task
The pipeline below is fully built — except its two workhorse signals. mom_5d (weekly trend) and mom_20d (monthly trend) are placeholder None columns, so dropna() wipes out every row and the shape line prints (0, 6).
- ›Replace the two
Noneplaceholders with real momentum features: the rolling sum of log returns over 5 and 20 days, lagged one day. The neighbouring columns (mom_1d,vol_20) show the exact pattern. - ›Re-run: the feature→target correlation table (the IC of each feature) and the pairwise correlation matrix only appear once the matrix has rows.
Predict before you run: out of 200 trading days, how many rows survive dropna()? Remember — mom_20d needs a 20-day warm-up plus a 1-day lag, and the target needs tomorrow's return. Write your number down, then check the shape line.