Random Forest & Feature Importance
- RandomForestClassifier
- ensemble learning
- bagging
- out-of-bag score
- feature_importances_
- variance reduction
- overfitting prevention
Random Forest & Feature Importance 🌲
Monday, 8:47 AM, the quant desk. Last week you fit a single decision tree on momentum features. Training accuracy: 91%. Your PM was thrilled. Then it went live and hit 48% out-of-sample - worse than a coin flip. Digging into the tree, you find the guilty branch: "if 20-day volatility is between 1.31% and 1.34%, the market goes UP." That is not a signal. That is a coincidence with a confidence problem.
Today you fix it the way finance fixes everything: diversification.
🧠 The mental model: 100 analysts voting
One star analyst is dangerous - brilliant on the data they have seen, overconfident everywhere else. A Random Forest fires the star and hires a committee of 100 average analysts, then rigs the hiring so no two think alike:
- Bagging (bootstrap aggregating): each analyst studies a random resample of history - n days drawn with replacement, so some days repeat and ~37% are never seen at all.
- Feature subsampling: at every decision point, each analyst may only consult a random subset of the indicators - nobody is allowed to copy the one loud feature everyone else uses.
Individually, each tree is mediocre and still overconfident. But each is overconfident about different noise - and when you average 100 different errors, the errors largely cancel while the shared signal survives. You already know this theorem from portfolio theory: idiosyncratic risk diversifies away. Bagging is diversification applied to models.
Why the ensemble beats a single tree
| Single Decision Tree | Random Forest |
|---|---|
| High variance - memorises noise | Low variance - averaging cancels noise |
| One outlier can reshape the whole tree | Robust: 100 trees, 100 opinions |
| No honest self-assessment | OOB score = free validation |
The two dials of randomness that make it work:
| Randomness | scikit-learn dial | What it buys you |
|---|---|---|
| Resampled rows per tree | bootstrap=True (default) | Each tree trains on different history; ~37% of rows left out per tree |
| Random features per split | max_features (default 'sqrt') | Decorrelates the trees, so their errors actually cancel when averaged |
A perk for econometricians: trees split on thresholds (mom_5d > 0.004?), so features need no scaling - no StandardScaler today - and nonlinearities and interactions are captured automatically, without hand-building the interaction terms you would add to an OLS regression.
Out-of-Bag (OOB): the exam the forest grades itself
Sampling n rows with replacement means each tree sees only ~63% of the distinct rows (in the limit, 1 − 1/e ≈ 0.632). The other ~37% are that tree's out-of-bag rows. scikit-learn can score every training row using only the trees that never saw it - an honest, nearly-unbiased validation estimate, without sacrificing a single row to a hold-out set.
All it takes is oob_score=True in the constructor; after .fit() the result lives in rf.oob_score_.
⚠️ One honest caveat for finance: the bootstrap shuffles rows ignoring time order. OOB is a great free diagnostic, but for a tradeable strategy the final word still belongs to a walk-forward (temporal) test - which is why we keep our 80/20 time split too.
Feature importances - with a health warning
rf.feature_importances_ # array of shape (n_features,), sums to 1
Each feature's importance is the mean decrease in impurity it produced across all splits in all trees. Higher = the forest leaned on it more. But read it like a footnote in an annual report, not a headline: importances are computed on training data, favour features with many possible split points, and split the credit between correlated features - and our momentum features overlap by construction. The robust cross-check is permutation importance on held-out data.
The text bar chart - a choice, not a limitation
Charts are great in notebooks - but today we deliberately print a terminal-style text bar chart: real quant infrastructure lives in terminals - cron logs, SSH sessions, alerts piped into a chat channel - and a chart built from █ blocks works anywhere Python can print.
mom_5d |████████████████░░░░ 42.3%
vol_20 |███████████░░░░░░░░░ 28.1%
📜 Where this comes from
Random forests were published in 2001 by Leo Breiman (1928–2005), the UC Berkeley statistician who had already given the world CART decision trees (1984) and bagging (1996). That same year he wrote the famous essay "Statistical Modeling: The Two Cultures", urging statisticians to judge models by predictive accuracy rather than by assumptions. He developed random forests with Adele Cutler - and a quarter-century later it is still one of the most-used algorithms on tabular data.
Your Task
The starter builds a 100-tree forest and prints a full report - but the forest never grades itself: the configuration lacks oob_score=True, so the report prints OOB Score: not computed. Fix the forest configuration (keep n_estimators=100, max_depth=4, random_state=42).
Predict before you run: these features come from a near-random walk. Will the OOB score land closer to 90% or to 50%? And will it be higher or lower than the test accuracy? Write your two guesses down, then run.