Newey-West (HAC) Standard Errors in Python

Newey-West Standard Errors: When OLS Is Too Confident

Scene: the return-predictability chapter of a master's thesis. You regress the stock market's return over the next 12 months on today's valuation ratio, using 24 years of monthly data. The output is thrilling: t = -2.77, p = 0.006. A high ratio predicts low returns; the chapter writes itself. Your second reader sends it back with one line: "Overlapping observations. Newey-West, please." In this lesson you see why that note is right, and you watch the "discovery" evaporate.

What it is: The standard error of a coefficient measures how much it would vary across samples. The classic OLS formula assumes the errors are uncorrelated with each other and have constant variance. Robust (White, HC) standard errors drop the constant-variance assumption. Newey-West (HAC: heteroskedasticity and autocorrelation consistent) standard errors also drop the no-correlation assumption. The coefficients do not change. Only the standard errors, and so the t-statistics and p-values, do.

Why overlapping returns break OLS

The 12-month return from January to December and the one from February to January share 11 of their 12 months. Neighbouring observations are therefore almost the same number, and so are their errors. Here the residuals have a lag-1 autocorrelation of 0.90 (a Durbin-Watson of about 0.2, far below 2). OLS counts 288 observations, but much of that is the same information repeated, so it reports a standard error that is far too small.

The data in this lesson are built so the truth is known: monthly returns are pure noise, the true slope is zero, and any "predictability" is an artefact.

The formulas

Classic OLS:  Var(β̂) = s² (X'X)⁻¹
HAC:          Var(β̂) = (X'X)⁻¹ · S · (X'X)⁻¹

S = Σ_t u_t² x_t x_t'           (White's part)
  + Σ_l w_l Σ_t u_t u_(t-l)
      × (x_t x_(t-l)' + x_(t-l) x_t')
                                (l = 1 ... L)

Bartlett weights:  w_l = 1 - l / (L + 1)

The "bread" (X'X)⁻¹ is the same as in OLS. The "meat" S adds up the products of residuals that are l periods apart, for l up to L lags, with weights that fade linearly. The Bartlett weights are Newey and West's trick: they guarantee the variance can never come out negative.

With L = 11 the weights run 11/12, 10/12, ..., 1/12. Lag 1 counts at 0.917, lag 11 at only 0.083.

Choosing the number of lags

• Overlapping h-period returns: the overlap mechanically creates correlation up to h - 1 lags, so use at least L = h - 1. For 12-month returns, L = 11.
• No overlap, general case: a common rule of thumb is L = floor(4 × (T/100)^(2/9)). For T = 288 that gives 5.

In statsmodels: one argument

# classic
fit = sm.OLS(y, X).fit()
# robust to unequal variance (White)
fit = sm.OLS(y, X).fit(cov_type="HC1")
# Newey-West: also robust to autocorrelation
fit = sm.OLS(y, X).fit(cov_type="HAC",
                       cov_kwds={"maxlags": 11})

On this lesson's data:

The coefficient is identical in every row. White's correction alone does not help, because the problem is correlation, not unequal variance. Newey-West more than doubles the standard error, and the "discovery" is no longer significant: the right answer, since the true slope is zero.

Two details in statsmodels' HAC output: the coefficient table switches from t to z (it uses the normal distribution unless you pass use_t=True), and by default it applies no small-sample correction, which is why the hand-rolled formula above matches it exactly.

Your Task

The starter fits the regression, then rebuilds Newey-West by hand as a cross-check. YOUR TURN marks the fit: it uses the classic standard errors, so the cross-check prints MISMATCH and the verdict says "significant".

Predict before you run: once you switch to HAC with 11 lags, which numbers change (coef, std err, t, p) and which stay put? Then change the .fit() call and confirm the cross-check prints MATCH.