Will the Market Go UP or DOWN Tomorrow? 🎯

Monday morning at a quant fund. An intern bursts into the research meeting: "My logistic regression predicts tomorrow's market direction with 96% accuracy!" The portfolio manager does not even look up. "Show me your target definition." She knows something the intern does not: Renaissance Technologies' Medallion fund — the most profitable fund in history — is right on only about 50.75% of its trades (Gregory Zuckerman, The Man Who Solved the Market). Nobody gets 96% on daily direction honestly. A score like that almost always means the model was shown the answer. The mistake has a name — data leakage — and in this lesson you will commit it, diagnose it, and fix it.

🧠 The mental model: trading with tomorrow's newspaper

Imagine "predicting" today's market moves while holding today's Financial Times. You would be right every time — and it would be worthless, because in live trading that newspaper has not been printed yet. Leakage is exactly this: information from the future (or from the answer itself) sneaks into training. The backtest looks god-like; the live strategy bleeds money. One golden rule prevents it:

Features may only use information available at prediction time. The target must lie strictly in the future.

From "how much" to "which way"

So far you predicted how much the market moves (regression). A trader often only needs which way: label each day 1 if tomorrow is an UP day, 0 if DOWN, and you have a binary classification problem.

Good news if you have taken econometrics: sklearn's LogisticRegression is the logit model you already know — the same one used for default prediction and labor-market studies:

P(UP | X) = 1 / (1 + e^(-Xb))

It outputs a probability of the UP class, and .predict() applies a 0.5 cutoff. One difference from the econometrics classroom: sklearn adds L2 regularization by default — think of it as built-in shrinkage on the betas.

The timeline audit — who knows what, when?

Before training any financial model, audit every column: when does this information exist?

The trap waiting in today's starter code: the target is built from log_ret — today's return, the very same information as the feature ret_1d. The model does not forecast anything; it just reads the answer off its own input. That is why it scores ~96%.

Temporal split — never shuffle time

train_test_split(shuffle=True) on time series is leakage too: rows from 2021 end up training a model that is tested on 2020. Always cut by time:

 Python
split = int(len(df) * 0.8)
X_train, X_test = X[:split], X[split:]   # the past trains, the future tests

Reading the scoreboard

The 60% smell test

On liquid markets, daily direction is nearly a coin flip. Serious published results and practitioner folklore agree on the ballpark: 52–56% is genuinely good, and anything above ~60% on daily direction means you should hunt for leakage before you celebrate. This one heuristic will save you more embarrassment than any algorithm.

Your Task

1. ›Run the starter code exactly as it is. Watch it score ~96% — and read the diagnostic line that calls it suspicious.
2. ›Predict before you fix it: once the target is rebuilt from next_ret (tomorrow's return), will accuracy be (a) still ~96%, (b) ~75%, or (c) barely above a coin flip?
3. ›Fix the 👉 YOUR TURN line so the target is tomorrow's direction, run again, and check your prediction against reality.